# Class Tokenizer

java.lang.Object
com.codename1.ui.editor.Tokenizer

All Implemented Interfaces:
  `SyntaxHighlighter`

public class Tokenizer extends Object implements SyntaxHighlighter

A small, allocation light, stateful lexer for the pure code editor. It tokenizes a single line at a time given the lexer state carried over from the previous line, and returns the state to carry into the next line. This makes rehighlighting incremental: after an edit only the changed line and any following lines whose entry state changed need to be retokenized.

Only the spans that carry color (keyword, string, comment, number) are emitted; the gaps between them are drawn in the default text color.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final int`
  `COMMENT`
  Token kind: a comment.
  `static final int`
  `KEYWORD`
  Token kind: a language keyword.
  `static final int`
  `NUMBER`
  Token kind: a numeric literal.
  `static final int`
  `PROPERTY`
  Token kind: an attribute name or object property.
  `static final int`
  `STATE_BLOCK_COMMENT`
  Lexer state: inside a block comment that opened on a previous line.
  `static final int`
  `STATE_CSS_COMMENT_DECLARATION`
  Lexer state: inside a CSS comment opened from a declaration block.
  `static final int`
  `STATE_CSS_DECLARATION`
  Lexer state: inside a CSS declaration block.
  `static final int`
  `STATE_NORMAL`
  Lexer state: normal code.
  `static final int`
  `STATE_TEMPLATE`
  Lexer state: inside a backtick template string that opened on a previous line.
  `static final int`
  `STATE_TRIPLE_DOUBLE`
  Lexer state: inside a Python triple-double-quoted string.
  `static final int`
  `STATE_TRIPLE_SINGLE`
  Lexer state: inside a Python triple-single-quoted string.
  `static final int`
  `STATE_XML_COMMENT`
  Lexer state: inside an XML/HTML comment.
  `static final int`
  `STRING`
  Token kind: a string or character literal.
  `static final int`
  `TYPE`
  Token kind: an XML/HTML tag or a declared type.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `Tokenizer(LanguageDef def)`
  Creates a tokenizer for the given language.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `SyntaxHighlightResult`
  `tokenize(String line, int startState)`
  Tokenizes one line.

  ### Methods inherited from class Object

  `clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### KEYWORD

    public static final int KEYWORD
    Token kind: a language keyword.
    See Also:
      - Constant Field Values

  - ### STRING

    public static final int STRING
    Token kind: a string or character literal.
    See Also:
      - Constant Field Values

  - ### COMMENT

    public static final int COMMENT
    Token kind: a comment.
    See Also:
      - Constant Field Values

  - ### NUMBER

    public static final int NUMBER
    Token kind: a numeric literal.
    See Also:
      - Constant Field Values

  - ### TYPE

    public static final int TYPE
    Token kind: an XML/HTML tag or a declared type.
    See Also:
      - Constant Field Values

  - ### PROPERTY

    public static final int PROPERTY
    Token kind: an attribute name or object property.
    See Also:
      - Constant Field Values

  - ### STATE_NORMAL

    public static final int STATE_NORMAL
    Lexer state: normal code.
    See Also:
      - Constant Field Values

  - ### STATE_BLOCK_COMMENT

    public static final int STATE_BLOCK_COMMENT
    Lexer state: inside a block comment that opened on a previous line.
    See Also:
      - Constant Field Values

  - ### STATE_TEMPLATE

    public static final int STATE_TEMPLATE
    Lexer state: inside a backtick template string that opened on a previous line.
    See Also:
      - Constant Field Values

  - ### STATE_XML_COMMENT

    public static final int STATE_XML_COMMENT
    Lexer state: inside an XML/HTML comment.
    See Also:
      - Constant Field Values

  - ### STATE_TRIPLE_SINGLE

    public static final int STATE_TRIPLE_SINGLE
    Lexer state: inside a Python triple-single-quoted string.
    See Also:
      - Constant Field Values

  - ### STATE_TRIPLE_DOUBLE

    public static final int STATE_TRIPLE_DOUBLE
    Lexer state: inside a Python triple-double-quoted string.
    See Also:
      - Constant Field Values

  - ### STATE_CSS_DECLARATION

    public static final int STATE_CSS_DECLARATION
    Lexer state: inside a CSS declaration block.
    See Also:
      - Constant Field Values

  - ### STATE_CSS_COMMENT_DECLARATION

    public static final int STATE_CSS_COMMENT_DECLARATION
    Lexer state: inside a CSS comment opened from a declaration block.
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### Tokenizer

    public Tokenizer(LanguageDef def)
    Creates a tokenizer for the given language.

- ## Method Details

  - ### tokenize

    public SyntaxHighlightResult tokenize(String line, int startState)

    Tokenizes one line.

    ###### Parameters

    - `line`: the line text (without its trailing newline)

    - `startState`: the lexer state carried from the previous line

    ###### Returns

    the colored spans and exit state

    Specified by:
      `tokenize` in interface `SyntaxHighlighter`