|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectcom.holub.text.TokenSet
public class TokenSet
A token set is a collection of tokens that define all possible
lexical units of some language. TokenSet objects serve as
token factories, and all tokens created by a particular
TokenSet are in that set. (see create(java.lang.String)
).
Scanner
ojbects use TokenSet
s to
recognize input tokens. Each Token
is responsible for
deciding whether it comes next in the input, and the
tokens examine the input in the order that they were created.
See the source code for Database
(in the distribution jar) for an example of how a token set
is used in conjunction with a Scanner.
This code may be used freely by yourself with the following restrictions:
|
Constructor Summary | |
---|---|
TokenSet()
|
Method Summary | |
---|---|
Token |
create(String spec)
Create a Token based on a specification and add it to the current set. |
Iterator |
iterator()
Return an iterator across the Token pool. |
Methods inherited from class java.lang.Object |
---|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public TokenSet()
Method Detail |
---|
public Iterator iterator()
create(java.lang.String)
was called. You can use this iterator
to list all the tokens in a given set.
public Token create(String spec)
An appropriate token type is chosen by examining the input
specification. In particular, a RegexToken
is
created unless the input string contains no regular-expression
metacharacters ({i \\[]{}()$^*+?|}) or starts with a single-quote
mark ('). In this case, a
WordToken
is created if the specification ends
in any character that could occur in a Java identifier;
otherwise a SimpleToken
is created.
If a string that starts with a single-quote mark also
ends with a single-quote mark, the end-quote mark
is discarded. The end-quote mark is optional.
Tokens are always extracted from the beginning of a String, so the characters that precede the token are irrelevant.
WordToken
,
RegexToken
,
SimpleToken
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |