Class Summary |
BeginToken |
The Begin token is special in that it has no width, and
doesn't match anything in the input. |
RegexToken |
Matches a token specified by a regular expression. |
Scanner |
A Scanner lets you read a file as a set of input tokens. |
Scanner.Test |
|
SimpleToken |
Matches a simple symbol that doesn't have to be on a "word"
boundary; punctuation, for example. |
TokenSet |
A token set is a collection of tokens that define all possible
lexical units of some language. |
WordToken |
Recognize a token that looks like a word. |