tokenizeCode()

Defined in: tokenizer.ts:986

Tokenizes entire code block string into a flat token list. Adheres to fundamental compiler lexical precedence:

1. Whitespace & universal newlines (\r\n, \n, standalone \r)
2. Comments (line & block)
3. Strings & prefixed literals (e.g. f"...", r"...", ...)
4. Context-sensitive RegExp literals (JavaScript)
5. Numbers (hex, binary, float, bigint, complex, CSS hex colors)
6. Identifiers + O(1) Set classifications (keywords, builtins, booleans, functions)
7. Operators (longest match first) & Punctuation
8. Fallback character preservation

ParametersCopy Anchor Link

sourceCopy Anchor Link

string

language?Copy Anchor Link

string

ReturnsCopy Anchor Link

Token[]