|
Class Summary |
| BeginToken |
The Begin token is special in that it has no width, and
doesn't match anything in the input. |
| RegexToken |
Matches a token specified by a regular expression. |
| Scanner |
A Scanner lets you read a file as a set of input tokens. |
| Scanner.Test |
|
| SimpleToken |
Matches a simple symbol that doesn't have to be on a "word"
boundary; punctuation, for example. |
| TokenSet |
A token set is a collection of tokens that define all possible
lexical units of some language. |
| WordToken |
Recognize a token that looks like a word. |