com.holub.text
Class TokenSet

java.lang.Object
  extended by com.holub.text.TokenSet

public class TokenSet
extends Object

A token set is a collection of tokens that define all possible lexical units of some language. TokenSet objects serve as token factories, and all tokens created by a particular TokenSet are in that set. (see create(java.lang.String)). Scanner ojbects use TokenSets to recognize input tokens. Each Token is responsible for deciding whether it comes next in the input, and the tokens examine the input in the order that they were created.

See the source code for Database (in the distribution jar) for an example of how a token set is used in conjunction with a Scanner.

©2004 Allen I. Holub. All rights reserved.

This code may be used freely by yourself with the following restrictions:

  1. Your splash screen, about box, or equivalent, must include Allen Holub's name, copyright, and URL. For example:

    This program contains Allen Holub's SQL package.
    (c) 2005 Allen I. Holub. All Rights Reserved.
    http://www.holub.com


    If your program does not run interactively, then the foregoing notice must appear in your documentation.
  2. You may not redistribute (or mirror) the source code.
  3. You must report any bugs that you find to me. Use the form at http://www.holub.com/company/contact.html or send email.
  4. The software is supplied as is. Neither Allen Holub nor Holub Associates are responsible for any bugs (or any problems caused by bugs, including lost productivity or data) in any of this code.

Constructor Summary
TokenSet()
           
 
Method Summary
 Token create(String spec)
          Create a Token based on a specification and add it to the current set.
 Iterator iterator()
          Return an iterator across the Token pool.
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

TokenSet

public TokenSet()
Method Detail

iterator

public Iterator iterator()
Return an iterator across the Token pool. This iterator is guaranteed to return the tokens in the order that create(java.lang.String) was called. You can use this iterator to list all the tokens in a given set.


create

public Token create(String spec)
Create a Token based on a specification and add it to the current set.

An appropriate token type is chosen by examining the input specification. In particular, a RegexToken is created unless the input string contains no regular-expression metacharacters ({i \\[]{}()$^*+?|}) or starts with a single-quote mark ('). In this case, a WordToken is created if the specification ends in any character that could occur in a Java identifier; otherwise a SimpleToken is created. If a string that starts with a single-quote mark also ends with a single-quote mark, the end-quote mark is discarded. The end-quote mark is optional.

Tokens are always extracted from the beginning of a String, so the characters that precede the token are irrelevant.

See Also:
WordToken, RegexToken, SimpleToken