TokenizerProperties

All Known Implementing Classes:

AbstractTokenizerProperties, StandardTokenizerProperties
```
public interface TokenizerProperties
```
The interface TokenizerProperties declares constants and methods to maintain the characteristics of a Tokenizer, e. g. comments, keywords and special sequences. A TokenizerProperties implementation instance can be used by one or more Tokenizer instances.
A TokenizerProperties object that is used to parse Java or C code including line and column information, would be setup like this:
```
    TokenizerProperties props = new StandardTokenizerProperties();

    props.setParseFlags(Flags.F_COUNT_LINES);
    props.addLineComment("//");
    props.addBlockComment(TokenizerProperties.DEFAULT_BLOCK_COMMENT_START,
                          TokenizerProperties.DEFAULT_BLOCK_COMMENT_END);
    props.addString("\"", "\"", "\\");
    props.addString("'", "'", "\\");
    props.addSpecialSequence(">>");
    props.addSpecialSequence("<<");
    props.addSpecialSequence("++");
    props.addSpecialSequence("--");
    ...
    props.addKeyword("class");
    props.addKeyword("if");
    props.addKeyword("then");
    props.addKeyword("while");
    props.addKeyword("do");
    ...
```
Beside the dedicated method groups for comments, strings, keywords and special sequences (e.g. addKeyword(java.lang.String), removeKeyword(java.lang.String) and keywordExists(java.lang.String)) there is a set of generic methods working with TokenizerProperty objects:
When adding a property through one of the add... calls without the flags parameter, the currently active flags of the TokenizerProperties instance will be used for the property.
This interface is separated from the Tokenizer interface mainly to distinguish between the more static information, the actual source of data and the tokenizing process. Especially in multithreaded environments where multible instances of "equal" tokenizers are run at the same time, it saves both memory resources and setup effort to have one TokenizerProperties instance for all Tokenizer instances.
Beside its function as a store for lexical element descriptions, this interface also provides an event mechanism to notify all interested objects about changes (additions, modifications and removals) of such lexical element descriptions.
This interface partly replaces the older de.susebox.java.util.Tokenizer interface which is deprecated.
Author:

Heiko Blau

See Also:

Token, Tokenizer, TokenizerProperty

Field Summary

Fields
Modifier and Type	Field and Description
`static java.lang.String`	`DEFAULT_BLOCK_COMMENT_END` Default end sequence of a block comment (Java, C/C++).
`static java.lang.String`	`DEFAULT_BLOCK_COMMENT_START` Default starting sequence of a block comment (Java, C/C++).
`static java.lang.String`	`DEFAULT_CHAR_END` The well-known character ending sequence of C/C++, Java and other languages.
`static java.lang.String`	`DEFAULT_CHAR_ESCAPE` The well-known escape sequence for character literals in C/C++, Java and other languages.
`static java.lang.String`	`DEFAULT_CHAR_START` The well-known character starting sequence of C/C++, Java and other languages.
`static java.lang.String`	`DEFAULT_LINE_COMMENT` Default line comment seqence (Java, C++)
`static java.lang.String`	`DEFAULT_SEPARATORS` Separators are otherwise not remarkable characters.
`static java.lang.String`	`DEFAULT_STRING_END` The well-known string ending sequence of C/C++, Java and other languages.
`static java.lang.String`	`DEFAULT_STRING_ESCAPE` The well-known escape sequence for strings in C/C++, Java and other languages.
`static java.lang.String`	`DEFAULT_STRING_START` The well-known string starting sequence " of C/C++, Java and other languages.
`static java.lang.String`	`DEFAULT_WHITESPACES` Whitespaces are portions of the text, that contain one or more characters that separate the significant parts of the text.

Method Summary

All Methods Instance Methods Abstract Methods
Modifier and Type	Method and Description
`void`	`addBlockComment(java.lang.String start, java.lang.String end)` Registering a block comment with the parser.
`void`	`addBlockComment(java.lang.String start, java.lang.String end, java.lang.Object companion)` Registering a block comment with the parser.
`void`	`addBlockComment(java.lang.String start, java.lang.String end, java.lang.Object companion, int flags)` Registering a block comment.
`void`	`addBlockComment(java.lang.String start, java.lang.String end, java.lang.Object companion, int flags, int flagMask)` Registering a block comment with a set of flags and an associated flag mask.
`void`	`addKeyword(java.lang.String keyword)` Registering a keyword.
`void`	`addKeyword(java.lang.String keyword, java.lang.Object companion)` Registering a keyword.
`void`	`addKeyword(java.lang.String keyword, java.lang.Object companion, int flags)` Registering a keyword.
`void`	`addKeyword(java.lang.String keyword, java.lang.Object companion, int flags, int flagMask)` Registering a keyword with a set of flags and an associated flag mask..
`void`	`addLineComment(java.lang.String lineComment)` Registering a the starting sequence of a line comment.
`void`	`addLineComment(java.lang.String lineComment, java.lang.Object companion)` Registering a the starting sequence of a line comment.
`void`	`addLineComment(java.lang.String lineComment, java.lang.Object companion, int flags)` Registering a the starting sequence of a line comment.
`void`	`addLineComment(java.lang.String lineComment, java.lang.Object companion, int flags, int flagMask)` Registering a line comment with a set of flags and an associated flag mask.
`void`	`addPattern(java.lang.String pattern)` Registering a pattern.
`void`	`addPattern(java.lang.String pattern, java.lang.Object companion)` Registering a pattern with an associated object.
`void`	`addPattern(java.lang.String pattern, java.lang.Object companion, int flags)` Registering a pattern with an associated object.
`void`	`addPattern(java.lang.String pattern, java.lang.Object companion, int flags, int flagMask)` Registering a pattern with an associated object and explicitely given flags.
`void`	`addProperty(TokenizerProperty property)` Registering a `TokenizerProperty`.
`void`	`addSeparators(java.lang.String separators)` Adding new separators to the existing set.
`void`	`addSpecialSequence(java.lang.String specSeq)` Registering a special sequence of characters.
`void`	`addSpecialSequence(java.lang.String specSeq, java.lang.Object companion)` Registering a special sequence of characters.
`void`	`addSpecialSequence(java.lang.String specSeq, java.lang.Object companion, int flags)` Registering a special sequence of characters.
`void`	`addSpecialSequence(java.lang.String specSeq, java.lang.Object companion, int flags, int flagMask)` Registering a special sequence with a set of flags and an associated flag mask.
`void`	`addString(java.lang.String start, java.lang.String end, java.lang.String escape)` Registering a string description.
`void`	`addString(java.lang.String start, java.lang.String end, java.lang.String escape, java.lang.Object companion)` Registering a the sequences that are used for string-like text parts.
`void`	`addString(java.lang.String start, java.lang.String end, java.lang.String escape, java.lang.Object companion, int flags)` Registering a the sequences that are used for string-like text parts.
`void`	`addString(java.lang.String start, java.lang.String end, java.lang.String escape, java.lang.Object companion, int flags, int flagMask)` Registering a string with a set of flags and an associated flag mask.
`void`	`addTokenizerPropertyListener(TokenizerPropertyListener listener)` Registering a new `TokenizerPropertyListener`.
`void`	`addWhitespaces(java.lang.String whitespaces)` Adding new whitespaces to the existing set.
`boolean`	`blockCommentExists(java.lang.String start)` Checks if the give block comment is known.
`TokenizerProperty`	`getBlockComment(java.lang.String start)` Get the full description of a block comment property starting with the given prefix.
`java.lang.Object`	`getBlockCommentCompanion(java.lang.String start)` Retrieving a certain block comment.
`java.util.Iterator`	`getBlockComments()` This method returns an `Iterator` of `TokenizerProperty` objects.
`TokenizerProperty`	`getKeyword(java.lang.String keyword)` Get the full description of a keyword property.
`java.lang.Object`	`getKeywordCompanion(java.lang.String keyword)` Retrieving the companion of the given special sequence.
`java.util.Iterator`	`getKeywords()` This method returns an `Iterator` of `TokenizerProperty` objects.
`TokenizerProperty`	`getLineComment(java.lang.String lineComment)` Get the full description of a line comment property starting with the given prefix.
`java.lang.Object`	`getLineCommentCompanion(java.lang.String lineComment)` Retrieving the associated object of a certain line comment.
`java.util.Iterator`	`getLineComments()` This method returns an `Iterator` of `TokenizerProperty` objects.
`int`	`getParseFlags()` Retrieving the parser control flags.
`TokenizerProperty`	`getPattern(java.lang.String pattern)` Get the full description of a pattern property.
`java.lang.Object`	`getPatternCompanion(java.lang.String pattern)` Retrieving the information associated with a given pattern.
`java.util.Iterator`	`getPatterns()` This method returns an `Iterator` of `TokenizerProperty` objects.
`java.util.Iterator`	`getProperties()` This method returns an `Iterator` of `TokenizerProperty` objects.
`java.lang.String`	`getSeparators()` Obtaining the separator set of the `TokenizerProperties`.
`TokenizerProperty`	`getSpecialSequence(java.lang.String specSeq)` Get the full description of a special sequence property.
`java.lang.Object`	`getSpecialSequenceCompanion(java.lang.String specSeq)` Retrieving the companion of the given special sequence.
`java.util.Iterator`	`getSpecialSequences()` This method returns an `Iterator` of `TokenizerProperty` objects.
`TokenizerProperty`	`getString(java.lang.String start)` Get the full description of a string property starting with the given prefix.
`java.lang.Object`	`getStringCompanion(java.lang.String start)` Retrieving the information associated with a certain string.
`java.util.Iterator`	`getStrings()` This method returns an `Iterator` of `TokenizerProperty` objects.
`java.lang.String`	`getWhitespaces()` Obtaining the whitespace character set.
`boolean`	`isFlagSet(int flag)` Returns `true` if a given flag is set in the current parse flags.
`boolean`	`isFlagSet(TokenizerProperty prop, int flag)` Checks if a given flag (see `Flags`) is set for the given `TokenizerProperty` in the context of this `TokenizerProperties` instance.
`boolean`	`keywordExists(java.lang.String keyword)` Checks if the given keyword is known to the `TokenizerProperties`.
`boolean`	`lineCommentExists(java.lang.String lineComment)` Checks if the give line comment is known.
`boolean`	`patternExists(java.lang.String pattern)` Checks if the given pattern is known to the parser.
`boolean`	`propertyExists(TokenizerProperty property)` Checks if the given `TokenizerProperty` is known to this `TokenizerProperties` instance.
`void`	`removeBlockComment(java.lang.String start)` Removing a certain block comment.
`void`	`removeKeyword(java.lang.String keyword)` Removing a keyword property from the store.
`void`	`removeLineComment(java.lang.String lineComment)` Removing a certain line comment.
`void`	`removePattern(java.lang.String pattern)` Removing a pattern.
`void`	`removeProperty(TokenizerProperty property)` Deregistering a `TokenizerProperty` from the store.
`void`	`removeSeparators(java.lang.String separators)` Removing separators from the existing set.
`void`	`removeSpecialSequence(java.lang.String specSeq)` Removing a special sequence property from the store.
`void`	`removeString(java.lang.String start)` Removing a string description.
`void`	`removeTokenizerPropertyListener(TokenizerPropertyListener listener)` Removing a listener from the list of registered `TokenizerPropertyListener` instances.
`void`	`removeWhitespaces(java.lang.String whitespaces)` Removing whitespaces from the existing set.
`void`	`setParseFlags(int flags)` Setting the control flags of the `TokenizerProperties`.
`void`	`setSeparators(java.lang.String separators)` Setting the separator set.
`void`	`setWhitespaces(java.lang.String whitespaces)` Setting the whitespace character set of the tokenizer.
`boolean`	`specialSequenceExists(java.lang.String specSeq)` Checks if the given special sequence is known to the `TokenizerProperties`.
`boolean`	`stringExists(java.lang.String start)` Checks if the given starting sequence of the string is known to the parser.

- Field Detail
  - DEFAULT_WHITESPACES
```
static final java.lang.String DEFAULT_WHITESPACES
```
    Whitespaces are portions of the text, that contain one or more characters that separate the significant parts of the text. Generally, a sequence of whitespaces is equally represented by one single whitespace character. That is the difference to separators.
    The value of this constant contains the ASCII characters space, tab, carriage return and linefeed.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_SEPARATORS
```
static final java.lang.String DEFAULT_SEPARATORS
```
    Separators are otherwise not remarkable characters. An opening parenthesis might be nessecary for a syntactically correct text, but without any special meaning to the compiler, interpreter etc. after it has been detected.
    The value of this constant includes all printable characters that are not alphanumeric and not whitespaces. Note, that adding one of the separator characters as a special sequence (addSpecialSequence(java.lang.String)) takes precedence.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_BLOCK_COMMENT_START
```
static final java.lang.String DEFAULT_BLOCK_COMMENT_START
```
    Default starting sequence of a block comment (Java, C/C++).
    
    See Also:
    
    Constant Field Values
  - DEFAULT_BLOCK_COMMENT_END
```
static final java.lang.String DEFAULT_BLOCK_COMMENT_END
```
    Default end sequence of a block comment (Java, C/C++).
    
    See Also:
    
    Constant Field Values
  - DEFAULT_LINE_COMMENT
```
static final java.lang.String DEFAULT_LINE_COMMENT
```
    Default line comment seqence (Java, C++)
    
    See Also:
    
    Constant Field Values
  - DEFAULT_STRING_START
```
static final java.lang.String DEFAULT_STRING_START
```
    The well-known string starting sequence " of C/C++, Java and other languages.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_STRING_END
```
static final java.lang.String DEFAULT_STRING_END
```
    The well-known string ending sequence of C/C++, Java and other languages.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_STRING_ESCAPE
```
static final java.lang.String DEFAULT_STRING_ESCAPE
```
    The well-known escape sequence for strings in C/C++, Java and other languages.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_CHAR_START
```
static final java.lang.String DEFAULT_CHAR_START
```
    The well-known character starting sequence of C/C++, Java and other languages.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_CHAR_END
```
static final java.lang.String DEFAULT_CHAR_END
```
    The well-known character ending sequence of C/C++, Java and other languages.
    
    See Also:
    
    Constant Field Values
  - DEFAULT_CHAR_ESCAPE
```
static final java.lang.String DEFAULT_CHAR_ESCAPE
```
    The well-known escape sequence for character literals in C/C++, Java and other languages.
    
    See Also:
    
    Constant Field Values
- Method Detail
  - setParseFlags
```
void setParseFlags(int flags)
```
    Setting the control flags of the TokenizerProperties. Use a combination of the Flags for the parameter.
    This method sets the parse flags globally for all Tokenizer objects sharing this TokenizerProperties instance. Some flags can be set for single tokenizers separately.
    When adding a property like a keyword without explicitely specifying flags, these properties are handled with the flags that were effective at the time of adding.
    The method fires a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_MODIFIED each time it is called with different flag masks. The passed TokenizerProperty has the type TokenizerProperty.PARSE_FLAG_MASK.
    
    Parameters:
    
    flags - the parser control flags
  - getParseFlags
```
int getParseFlags()
```
    Retrieving the parser control flags. A bitmask containing the Flags constants is returned. These flags are the ones set for all Tokenizer instances sharing this TokenizerProperties object and may be partially overridden in these tokenizers.
    
    Returns:
    
    the globally set parser control flags
    
    See Also:
    
    setParseFlags(int)
  - isFlagSet
```
boolean isFlagSet(int flag)
```
    Returns true if a given flag is set in the current parse flags. If the parameter contains more than one bit the method returns only true if all bits are set.
    
    Parameters:
    
    flag - the flag to test
    
    Returns:
    
    true if all bits in flag are set.
    
    See Also:
    
    setParseFlags(int), getParseFlags(), isFlagSet(TokenizerProperty, int)
  - isFlagSet
```
boolean isFlagSet(TokenizerProperty prop,
                  int flag)
           throws java.lang.NullPointerException
```
    Checks if a given flag (see Flags) is set for the given TokenizerProperty in the context of this TokenizerProperties instance.
    The method uses the flag state of the TokenizerProperty object if it is contained in the flag mask of the property (see TokenizerProperty.containsFlag(int)). Otherwise it falls back to the flag state set globally for this TokenizerProperties instance.
    
    Parameters:
    
    prop - the TokenizerProperty concerned
    
    flag - the flag to check (may contain more than one bit)
    
    Returns:
    
    true if the flag is set either explicit in the property or globally for this TokenizerProperties object, false otherwise
    
    Throws:
    
    java.lang.NullPointerException - if no property is given
  - setWhitespaces
```
void setWhitespaces(java.lang.String whitespaces)
             throws java.lang.IllegalArgumentException
```
    Setting the whitespace character set of the tokenizer. Implementations should be able to operate on ranges like "a-z" when more than two whitespace characters are neighbours in the UNICODE character set.
    Whitespaces are sequences that have the same syntactical meaning as one single whitespace character would have. That means " " (many spaces) is the same as " " (one space).
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED or TokenizerPropertyEvent.PROPERTY_MODIFIED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)).
    The method accepts an empty string to describe "no whitespaces".
    
    Parameters:
    
    whitespaces - the whitespace set
    
    Throws:
    
    java.lang.IllegalArgumentException - when null is passed or incomplete ranges are specified (e.g. "a-")
    
    See Also:
    
    getWhitespaces(), addWhitespaces(java.lang.String), removeWhitespaces(java.lang.String)
  - addWhitespaces
```
void addWhitespaces(java.lang.String whitespaces)
             throws java.lang.IllegalArgumentException
```
    Adding new whitespaces to the existing set. This is a convenience method to complete the whitespace set without having to include the already known whitespaces (for instance the DEFAULT_WHITESPACES set).
    Whitespaces that are already known, are ignored (the new whitespaces are "merged" into the known set).
    For more information see setWhitespaces(java.lang.String).
    
    Parameters:
    
    whitespaces - additional whitespaces for the whitespace set
    
    Throws:
    
    java.lang.IllegalArgumentException - when null is passed or incomplete ranges are specified (e.g. "a-")
    
    See Also:
    
    getWhitespaces(), setWhitespaces(java.lang.String), removeWhitespaces(java.lang.String)
  - removeWhitespaces
```
void removeWhitespaces(java.lang.String whitespaces)
                throws java.lang.IllegalArgumentException
```
    Removing whitespaces from the existing set. This is a convenience method to modify the whitespace set without having to reconstruct the probably major part of the whitespaces (for instance the DEFAULT_WHITESPACES set). Especially, this method can be used to remove a known single whitespace without bothering about the other whitespaces
    Whitespaces that are not known, are ignored.
    For more information see setWhitespaces(java.lang.String).
    
    Parameters:
    
    whitespaces - whitespaces to remove from the whitespace set
    
    Throws:
    
    java.lang.IllegalArgumentException - when null is passed or incomplete ranges are specified (e.g. "a-")
    
    See Also:
    
    getWhitespaces(), setWhitespaces(java.lang.String), addWhitespaces(java.lang.String)
  - getWhitespaces
```
java.lang.String getWhitespaces()
```
    Obtaining the whitespace character set. The set may contain ranges. The method may return an emtpy string, if no whitespaces are known.
    
    Returns:
    
    the currently active whitespace set
    
    See Also:
    
    setWhitespaces(java.lang.String)
  - setSeparators
```
void setSeparators(java.lang.String separators)
            throws java.lang.IllegalArgumentException
```
    Setting the separator set. This set may contain ranges. A range is a character (lower limit) followed by a '-' (minus) followed by a second character (upper limit). A range of "a-z" means: all characters in the UNICODE character set between and including 'a' and 'z'. The character '-' itself should be preceded by an escape character. Ranges should be used whenever possible since they speed up the parsing process.
    Separators are characters that are significant for the syntax. A sequence of separators is NOT equal to one single separator. Thats the difference to whitespaces.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED or TokenizerPropertyEvent.PROPERTY_MODIFIED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)).
    The method accepts an empty string to describe "no separators".
    
    Parameters:
    
    separators - the set of separating characters
    
    Throws:
    
    java.lang.IllegalArgumentException - when null is passed or incomplete ranges are specified (e.g. "[-")
    
    See Also:
    
    getSeparators()
  - addSeparators
```
void addSeparators(java.lang.String separators)
            throws java.lang.IllegalArgumentException
```
    Adding new separators to the existing set. This is a convenience method to complete the separator set without having to include the already known separators (for instance the DEFAULT_SEPARATORS set).
    Separators that are already known, are ignored (the new separators are "merged" into the known set).
    For more information see setSeparators(java.lang.String).
    
    Parameters:
    
    separators - additional set of separating characters
    
    Throws:
    
    java.lang.IllegalArgumentException - when null is passed or incomplete ranges are specified (e.g. "a-")
    
    See Also:
    
    getSeparators(), setSeparators(java.lang.String), removeSeparators(java.lang.String)
  - removeSeparators
```
void removeSeparators(java.lang.String separators)
               throws java.lang.IllegalArgumentException
```
    Removing separators from the existing set. This is a convenience method to modify the separator set without having to reconstruct the probably major part of the separators (for instance the DEFAULT_SEPARATORS set). Especially, this method can be used to remove a known single separator without bothering about the other separators.
    Separators that are not known, are ignored.
    For more information see setSeparators(java.lang.String).
    
    Parameters:
    
    separators - separating characters to remove from the separator set
    
    Throws:
    
    java.lang.IllegalArgumentException - when null is passed or incomplete ranges are specified (e.g. "a-")
    
    See Also:
    
    getSeparators(), setSeparators(java.lang.String), addSeparators(java.lang.String)
  - getSeparators
```
java.lang.String getSeparators()
```
    Obtaining the separator set of the TokenizerProperties. The set may contain ranges or may be empty.
    
    Returns:
    
    the currently used set of separating characters
    
    See Also:
    
    setSeparators(java.lang.String)
  - addString
```
void addString(java.lang.String start,
               java.lang.String end,
               java.lang.String escape)
        throws java.lang.IllegalArgumentException,
               java.lang.UnsupportedOperationException
```
    Registering a string description. Strings are things like the primitive string literals in C/C++, SQL varchar literals, but also the character literals of C/C++ and Java.
    If the given string starting sequence is already known to the parser, it will simply be re-registered. Using this method on a known string with an associated companion will remove that companion.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new string TokenizerProperty if the string is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the string is re-registered.
    Passing null or an empty string for the escape sequence means that no escapes are used in the described string element.
    
    Parameters:
    
    start - the starting sequence of a string
    
    end - the finishing sequence of a string
    
    escape - the escape sequence inside the string
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for start or end
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addString(String, String, String, Object), addString(String, String, String, Object, int), removeString(java.lang.String)
  - addString
```
void addString(java.lang.String start,
               java.lang.String end,
               java.lang.String escape,
               java.lang.Object companion)
        throws java.lang.IllegalArgumentException
```
    Registering a the sequences that are used for string-like text parts. This method supports also an information associated with the string, called the companion.
    If the given string starting sequence is already known to the parser, it will simply be re-registered. Using this method on a known string with an associated companion will replace that companion against the given one.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new string TokenizerProperty if the string is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the string is re-registered.
    Passing null or an empty string for the escape sequence means that no escapes are used in the described string element.
    
    Parameters:
    
    start - the starting sequence of a string
    
    end - the finishing sequence of a string
    
    escape - the escape sequence inside the string
    
    companion - the associated information
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for start or end
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addString(String, String, String), addString(String, String, String, Object, int), removeString(java.lang.String)
  - addString
```
void addString(java.lang.String start,
               java.lang.String end,
               java.lang.String escape,
               java.lang.Object companion,
               int flags)
        throws java.lang.IllegalArgumentException,
               java.lang.UnsupportedOperationException
```
    Registering a the sequences that are used for string-like text parts. This method supports also an information associated with the string, called the companion.
    If the given string starting sequence is already known to the parser, it will simply be re-registered. Using this method on a known string with an associated companion will replace that companion against the given one.
    This version of addString supports a bitmask of the Flags to modify the general tokenizer settings (see setParseFlags(int) for this special element.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new string TokenizerProperty if the string is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the string is re-registered.
    Passing null or an empty string for the escape sequence means that no escapes are used in the described string element.
    A call to this method is equivalent to addString(start, end, escape, flags, flags).
    
    Parameters:
    
    start - the starting sequence of a string
    
    end - the finishing sequence of a string
    
    escape - the escape sequence inside the string
    
    companion - the associated information
    
    flags - modification flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for start or end
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addString(String, String, String), addString(String, String, String, Object), addString(String, String, String, Object, int, int), removeString(java.lang.String)
  - addString
```
void addString(java.lang.String start,
               java.lang.String end,
               java.lang.String escape,
               java.lang.Object companion,
               int flags,
               int flagMask)
        throws java.lang.IllegalArgumentException,
               java.lang.UnsupportedOperationException
```
    Registering a string with a set of flags and an associated flag mask.
    The method is an extension to #addString(String, Object, int) having a bitmask for the flags that are explicitely specified for the block comment property. All other flag values (states) should be taken from the TokenizerProperty instance or from the Tokenizer.
    
    Parameters:
    
    start - the starting sequence of a string
    
    end - the finishing sequence of a string
    
    escape - the escape sequence inside the string
    
    companion - the associated information
    
    flags - modification flags
    
    flagMask - flags that have valid values in the parameter flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for keyword
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addString(String, String, String), addString(String, String, String, Object), addString(String, String, String, Object, int), removeString(java.lang.String)
  - removeString
```
void removeString(java.lang.String start)
           throws java.lang.IllegalArgumentException
```
    Removing a string description. The method does nothing if the string description identified by the given start sequence is not known to this TokenizerProperties instance. The method may throw an IllegalArgumentException if the given string start is null or empty.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_REMOVED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) if a string is actually removed.
    
    Parameters:
    
    start - the starting sequence of a string
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
  - getStringCompanion
```
java.lang.Object getStringCompanion(java.lang.String start)
                             throws java.lang.IllegalArgumentException,
                                    java.lang.UnsupportedOperationException
```
    Retrieving the information associated with a certain string. Only the starting sequence is nessecary to identify the string. If the string is not known to the parser, null will be returned.
    If one needs to know if a string exists without a companion or if the string is unknown so far, use also the method stringExists(java.lang.String).
    
    Parameters:
    
    start - the starting sequence of a string
    
    Returns:
    
    the associated information or null
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an emtpy string is passed
    
    java.lang.UnsupportedOperationException
  - stringExists
```
boolean stringExists(java.lang.String start)
```
    Checks if the given starting sequence of the string is known to the parser. The method accepts both empty and null strings for start by returning false.
    Note that there is no choice of parsing flags (different to the addString(String, String, String, Object, int) method), since it makes generally no sense to "overload" properties. A string might be introduced by the starting case-insensitive sequence STR, but there shouldn't be a case-sensitive start str.
    
    Parameters:
    
    start - the starting sequence of a string
    
    Returns:
    
    true if the string is registered, false otherwise
  - getString
```
TokenizerProperty getString(java.lang.String start)
                     throws java.lang.IllegalArgumentException
```
    Get the full description of a string property starting with the given prefix. The method returns null if the passed start parameter cannot be mapped to a known string description (stringExists(java.lang.String) would return false).
    
    Parameters:
    
    start - the starting sequence of a string
    
    Returns:
    
    the full string description or null
    
    Throws:
    
    java.lang.IllegalArgumentException - when null is passed
  - getStrings
```
java.util.Iterator getStrings()
```
    This method returns an Iterator of TokenizerProperty objects. Each TokenizerProperty object contains the starting, finishing and escaping sequence of a string description and the companion if it exists.
    
    Returns:
    
    enumeration of TokenizerProperty objects
  - addLineComment
```
void addLineComment(java.lang.String lineComment)
             throws java.lang.IllegalArgumentException,
                    java.lang.UnsupportedOperationException
```
    Registering a the starting sequence of a line comment. The line comment is a special type of whitespace. It starts with the given character sequence and contains all characters up to and including the next end-of-line character(s).
    Although most languages have only one line comment sequence, it is possible to use more than one.
    If the given line comment starting sequence is already known to the parser, it will simply be re-registered. Using this method on a known line comment with an associated companion will effectively remove the companion.
    
    Parameters:
    
    lineComment - the starting sequence of the line comment
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addLineComment(String, Object), addLineComment(String, Object, int), removeLineComment(java.lang.String)
  - addLineComment
```
void addLineComment(java.lang.String lineComment,
                    java.lang.Object companion)
             throws java.lang.IllegalArgumentException,
                    java.lang.UnsupportedOperationException
```
    Registering a the starting sequence of a line comment. The line comment is a special type of whitespace. It starts with the given character sequence and contains all characters up to and including the next end-of-line character(s).
    Although most languages have only one line comment sequence, it is possible to use more than one.
    This method supports also an information associated with the line comment, called the companion.
    If the given line comment starting sequence is already known to the parser, it will simply be re-registered. Using this method on a known line comment with an associated companion will replace that companion against the given one.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new line comment TokenizerProperty if the line comment is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the line comment is re-registered.
    
    Parameters:
    
    lineComment - the starting sequence of a line comment
    
    companion - the associated information
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for lineComment
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addLineComment(String), addLineComment(String, Object, int), removeLineComment(java.lang.String)
  - addLineComment
```
void addLineComment(java.lang.String lineComment,
                    java.lang.Object companion,
                    int flags)
             throws java.lang.IllegalArgumentException,
                    java.lang.UnsupportedOperationException
```
    Registering a the starting sequence of a line comment. The line comment is a special type of whitespace. It starts with the given character sequence and contains all characters up to and including the next end-of-line character(s).
    Although most languages have only one line comment sequence, it is possible to use more than one.
    This method supports also an information associated with the line comment, called the companion.
    If the given line comment starting sequence is already known to the parser, it will simply be re-registered. Using this method on a known line comment with an associated companion will replace that companion against the given one.
    This version of addLineComment supports a bitmask of the Flags to modify the general tokenizer settings (see setParseFlags(int)) for this special element.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new line comment TokenizerProperty if the line comment is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the line comment is re-registered.
    A call to this method is equivalent to addLineComment(lineComment, companion, flags, flags).
    
    Parameters:
    
    lineComment - the starting sequence of a line comment
    
    companion - the associated information
    
    flags - modification flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for lineComment
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addLineComment(String), addLineComment(String, Object), addLineComment(String, Object, int, int), removeLineComment(java.lang.String)
  - addLineComment
```
void addLineComment(java.lang.String lineComment,
                    java.lang.Object companion,
                    int flags,
                    int flagMask)
             throws java.lang.IllegalArgumentException,
                    java.lang.UnsupportedOperationException
```
    Registering a line comment with a set of flags and an associated flag mask.
    The method is an extension to addLineComment(String, Object, int) having a bitmask for the flags that are explicitely specified for the block comment property. All other flag values (states) should be taken from the TokenizerProperty instance or from the Tokenizer.
    
    Parameters:
    
    lineComment - the starting sequence of a line comment
    
    companion - the associated information
    
    flags - modification flags
    
    flagMask - flags that have valid values in the parameter flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for keyword
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addLineComment(String), addLineComment(String, Object), addLineComment(String, Object, int), removeLineComment(java.lang.String)
  - removeLineComment
```
void removeLineComment(java.lang.String lineComment)
                throws java.lang.IllegalArgumentException
```
    Removing a certain line comment. If the given comment is not known to this TokenizerProperties instance, the method does nothing.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_REMOVED to all registered listeners if a line comment property is actually removed.
    
    Parameters:
    
    lineComment - the starting sequence of the line comment
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
    
    See Also:
    
    addLineComment(String), addLineComment(String, Object), addLineComment(String, Object, int)
  - getLineCommentCompanion
```
java.lang.Object getLineCommentCompanion(java.lang.String lineComment)
                                  throws java.lang.IllegalArgumentException
```
    Retrieving the associated object of a certain line comment. If the given starting sequence of a line comment is not known to the parser, then the method returns null.
    To distinguish between an unknown line comment and companion-less line comment, use the method lineCommentExists(java.lang.String).
    
    Parameters:
    
    lineComment - the starting sequence of the line comment
    
    Returns:
    
    the object associated with the line comment
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for lineComment
    
    See Also:
    
    lineCommentExists(java.lang.String)
  - lineCommentExists
```
boolean lineCommentExists(java.lang.String lineComment)
```
    Checks if the give line comment is known. The method accepts both empty and null strings for lineComment by returning false.
    
    Parameters:
    
    lineComment - the starting sequence of the line comment
    
    Returns:
    
    true if the line comment is known, false otherwise
  - getLineComment
```
TokenizerProperty getLineComment(java.lang.String lineComment)
                          throws java.lang.IllegalArgumentException
```
    Get the full description of a line comment property starting with the given prefix. The method returns null if the passed lineComment parameter cannot be mapped to a known line comment description (lineCommentExists(java.lang.String) would return false).
    
    Parameters:
    
    lineComment - the starting sequence of the line comment
    
    Returns:
    
    the full line comment description or null
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
    
    See Also:
    
    lineCommentExists(java.lang.String)
  - getLineComments
```
java.util.Iterator getLineComments()
```
    This method returns an Iterator of TokenizerProperty objects. Each TokenizerProperty object contains one starting sequence of a line comment and its companion if it exists.
    
    Returns:
    
    enumeration of TokenizerProperty objects
  - addBlockComment
```
void addBlockComment(java.lang.String start,
                     java.lang.String end)
              throws java.lang.IllegalArgumentException,
                     java.lang.UnsupportedOperationException
```
    Registering a block comment with the parser. This version takes only the starting and finishing sequence of the block comment.
    If the given starting sequence is already known to the parser, the block comment is simply re-registered. Using this method on a known block comment with an associated companion will remove that companion.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new block comment TokenizerProperty if the comment is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the comment is re-registered.
    
    Parameters:
    
    start - the starting sequence of the block comment
    
    end - the finishing sequence of the block comment
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for start or end
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addBlockComment(String, String, Object), addBlockComment(String, String, Object, int), removeBlockComment(java.lang.String)
  - addBlockComment
```
void addBlockComment(java.lang.String start,
                     java.lang.String end,
                     java.lang.Object companion)
              throws java.lang.IllegalArgumentException,
                     java.lang.UnsupportedOperationException
```
    Registering a block comment with the parser. Beside the obviously nessecary starting and finishing sequence of the block comment, it takes an object that is associated with the block comment, called the companion.
    If the given starting sequence is already known to the parser, the block comment is simply re-registered. Using this method on a known block comment with an associated companion will replace that companion against the given one.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new block comment TokenizerProperty if the comment is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the comment is re-registered.
    
    Parameters:
    
    start - the starting sequence of the block comment
    
    end - the finishing sequence of the block comment
    
    companion - information object associated with this block comment
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for start or end
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addBlockComment(String, String), addBlockComment(String, String, Object, int), removeBlockComment(java.lang.String)
  - addBlockComment
```
void addBlockComment(java.lang.String start,
                     java.lang.String end,
                     java.lang.Object companion,
                     int flags)
              throws java.lang.IllegalArgumentException,
                     java.lang.UnsupportedOperationException
```
    Registering a block comment. Beside the obviously nessecary starting and finishing sequence of the block comment, it takes an object that is associated with the block comment, called the companion.
    If the given starting sequence is already known to the parser, the block comment is simply re-registered. Using this method on a known block comment with an associated companion will replace that companion against the given one.
    This version of addBlockComment supports a bitmask of the Flags to modify the general tokenizer settings (see setParseFlags(int)) for this special element.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new block comment TokenizerProperty if the comment is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the comment is re-registered.
    A call to this method is equivalent to addBlockComment(start, end, companion, flags, flags).
    
    Parameters:
    
    start - the starting sequence of the block comment
    
    end - the finishing sequence of the block comment
    
    companion - information object associated with this block comment
    
    flags - modification flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for start or end
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addBlockComment(String, String), addBlockComment(String, String, Object), addBlockComment(String, String, Object, int, int), removeBlockComment(java.lang.String)
  - addBlockComment
```
void addBlockComment(java.lang.String start,
                     java.lang.String end,
                     java.lang.Object companion,
                     int flags,
                     int flagMask)
              throws java.lang.IllegalArgumentException,
                     java.lang.UnsupportedOperationException
```
    Registering a block comment with a set of flags and an associated flag mask.
    The method is an extension to addBlockComment(String, String, Object, int) having a bitmask for the flags that are explicitely specified for the block comment property. All other flag values (states) should be taken from the TokenizerProperty instance or from the Tokenizer.
    
    Parameters:
    
    start - the starting sequence of the block comment
    
    end - the finishing sequence of the block comment
    
    companion - information object associated with this block comment
    
    flags - modification flags
    
    flagMask - flags that have valid values in the parameter flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for keyword
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addBlockComment(String, String), addBlockComment(String, String, Object), addBlockComment(String, String, Object, int), removeBlockComment(java.lang.String)
  - removeBlockComment
```
void removeBlockComment(java.lang.String start)
                 throws java.lang.IllegalArgumentException
```
    Removing a certain block comment. Only the starting sequence is nessecary to identify the block comment.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_REMOVED to all registered listeners if a block comment property is actually removed.
    
    Parameters:
    
    start - the starting sequence of the block comment
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
  - getBlockCommentCompanion
```
java.lang.Object getBlockCommentCompanion(java.lang.String start)
                                   throws java.lang.IllegalArgumentException
```
    Retrieving a certain block comment. Only the starting sequence is nessecary to identify the block comment. If the block comment is not known to the parser, then null is returned.
    To distinguish between an unknown line comment and companion-less line comment, use the method lineCommentExists(java.lang.String).
    
    Parameters:
    
    start - the starting sequence of the block comment
    
    Returns:
    
    the associated object of the block comment
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
  - blockCommentExists
```
boolean blockCommentExists(java.lang.String start)
```
    Checks if the give block comment is known. Only the starting sequence is nessecary to identify the block comment. The method accepts both empty and null strings for start by returning false.
    
    Parameters:
    
    start - the starting sequence of the block comment
    
    Returns:
    
    true if the block comment is known, false otherwise
  - getBlockComment
```
TokenizerProperty getBlockComment(java.lang.String start)
                           throws java.lang.IllegalArgumentException
```
    Get the full description of a block comment property starting with the given prefix. The method returns null if the passed start parameter cannot be mapped to a known block comment description (blockCommentExists(java.lang.String) would return false).
    
    Parameters:
    
    start - the starting sequence of the block comment
    
    Returns:
    
    the full block comment description or null
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
  - getBlockComments
```
java.util.Iterator getBlockComments()
```
    This method returns an Iterator of TokenizerProperty objects. Each TokenizerProperty object contains the starting and finishing sequence of a block comment and the companion if it exists.
    
    Returns:
    
    enumeration of TokenizerProperty objects
  - addSpecialSequence
```
void addSpecialSequence(java.lang.String specSeq)
                 throws java.lang.IllegalArgumentException,
                        java.lang.UnsupportedOperationException
```
    Registering a special sequence of characters. Such sequences may be multicharacter operators like the shift operators in Java.
    Unlike keywords, special sequences act also as separators between other tokens. If one special sequence is the prefix of other special sequences (in Java the shift operator >> is the prefix of the shift operator >>>), always the longest possible match is returned. Testing on special sequences takes place after whitespaces and comments are ruled out, but before ordinary separators are tested.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new special sequence TokenizerProperty if the sequence is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the sequence is re-registered.
    
    Parameters:
    
    specSeq - special sequence to register
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addSpecialSequence(String, Object), addSpecialSequence(String, Object, int), removeSpecialSequence(java.lang.String)
  - addSpecialSequence
```
void addSpecialSequence(java.lang.String specSeq,
                        java.lang.Object companion)
                 throws java.lang.IllegalArgumentException,
                        java.lang.UnsupportedOperationException
```
    Registering a special sequence of characters. Such sequences may be multicharacter operators like the shift operators in Java.
    Unlike keywords, special sequences act also as separators between other tokens. If one special sequence is the prefix of other special sequences (in Java the shift operator >> is the prefix of the shift operator >>>), always the longest possible match is returned. Testing on special sequences takes place after whitespaces and comments are ruled out, but before ordinary separators are tested. This form of addSpecialSequence also takes an object associated with the special sequence, called the companion.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new special sequence TokenizerProperty if the sequence is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the sequence is re-registered.
    
    Parameters:
    
    specSeq - special sequence to register
    
    companion - information object associated with this special sequence
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for specSeq
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addSpecialSequence(String), addSpecialSequence(String, Object, int), removeSpecialSequence(java.lang.String)
  - addSpecialSequence
```
void addSpecialSequence(java.lang.String specSeq,
                        java.lang.Object companion,
                        int flags)
                 throws java.lang.IllegalArgumentException,
                        java.lang.UnsupportedOperationException
```
    Registering a special sequence of characters. Such sequences may be multicharacter operators like the shift operators in Java.
    Unlike keywords, special sequences act also as separators between other tokens. If one special sequence is the prefix of other special sequences (in Java the shift operator >> is the prefix of the shift operator >>>), always the longest possible match is returned. Testing on special sequences takes place after whitespaces and comments are ruled out, but before ordinary separators are tested. This form of addSpecialSequence also takes an object associated with the special sequence, called the companion.
    This version of addSpecialSequence supports a bitmask of the Flags to modify the general tokenizer settings (see setParseFlags(int)) for this special element.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new special sequence TokenizerProperty if the sequence is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the sequence is re-registered.
    A call to this method is equivalent to addSpecialSequence(keyword, companion, flags, flags).
    
    Parameters:
    
    specSeq - special sequence to register
    
    companion - information object associated with this special sequence
    
    flags - modification flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for specSeq
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addSpecialSequence(String), addSpecialSequence(String, Object), addSpecialSequence(String, Object, int, int), removeSpecialSequence(java.lang.String)
  - addSpecialSequence
```
void addSpecialSequence(java.lang.String specSeq,
                        java.lang.Object companion,
                        int flags,
                        int flagMask)
                 throws java.lang.IllegalArgumentException,
                        java.lang.UnsupportedOperationException
```
    Registering a special sequence with a set of flags and an associated flag mask.
    The method is an extension to addSpecialSequence(String, Object, int) having a bitmask for the flags that are explicitely specified for the special sequence property. All other flag values (states) should be taken from the TokenizerProperty instance or from the Tokenizer.
    
    Parameters:
    
    specSeq - special sequence to register
    
    companion - information object associated with this special sequence
    
    flags - modification flags
    
    flagMask - flags that have valid values in the parameter flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for keyword
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addSpecialSequence(String), addSpecialSequence(String, Object), addSpecialSequence(String, Object, int), removeSpecialSequence(java.lang.String)
  - removeSpecialSequence
```
void removeSpecialSequence(java.lang.String specSeq)
                    throws java.lang.IllegalArgumentException
```
    Removing a special sequence property from the store. If the special sequence is not known, the method does nothing.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_REMOVED to all registered listeners if a special sequence property is actually removed.
    
    Parameters:
    
    specSeq - sequence to remove
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
    
    See Also:
    
    addSpecialSequence(String), addSpecialSequence(String, Object), addSpecialSequence(String, Object, int)
  - getSpecialSequenceCompanion
```
java.lang.Object getSpecialSequenceCompanion(java.lang.String specSeq)
                                      throws java.lang.IllegalArgumentException
```
    Retrieving the companion of the given special sequence. If the special sequence doesn't exist the method returns null.
    
    Parameters:
    
    specSeq - sequence to remove
    
    Returns:
    
    the object associated with the special sequence
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
  - getSpecialSequences
```
java.util.Iterator getSpecialSequences()
```
    This method returns an Iterator of TokenizerProperty objects. Each TokenizerProperty object contains a special sequence and the companion if it exists.
    
    Returns:
    
    enumeration of TokenizerProperty objects
  - specialSequenceExists
```
boolean specialSequenceExists(java.lang.String specSeq)
```
    Checks if the given special sequence is known to the TokenizerProperties. The method accepts both empty and null strings for specSeq by returning false.
    
    Parameters:
    
    specSeq - sequence to check
    
    Returns:
    
    true if the block comment is known, false otherwise
  - getSpecialSequence
```
TokenizerProperty getSpecialSequence(java.lang.String specSeq)
                              throws java.lang.IllegalArgumentException
```
    Get the full description of a special sequence property. The method returns null if the passed specSeq image is unknown (specialSequenceExists(java.lang.String) would return false).
    
    Parameters:
    
    specSeq - sequence to find
    
    Returns:
    
    the full sequence description or null
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
  - addKeyword
```
void addKeyword(java.lang.String keyword)
         throws java.lang.IllegalArgumentException,
                java.lang.UnsupportedOperationException
```
    Registering a keyword. If the keyword is already known to the TokenizerProperties then it is simply re-registered. If the known keyword has an associated companion it will be removed.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new keyword TokenizerProperty if the keyword is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the keyword is re-registered.
    
    Parameters:
    
    keyword - keyword to register
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addKeyword(String, Object), addKeyword(String, Object, int), removeKeyword(java.lang.String)
  - addKeyword
```
void addKeyword(java.lang.String keyword,
                java.lang.Object companion)
         throws java.lang.IllegalArgumentException,
                java.lang.UnsupportedOperationException
```
    Registering a keyword. If the keyword is already known to the TokenizerProperties then it is simply re-registered. If the known keyword has an associated companion it will be replaced against the given one.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new keyword TokenizerProperty if the keyword is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the keyword is re-registered.
    
    Parameters:
    
    keyword - keyword to register
    
    companion - information object associated with this keyword
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for keyword
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addKeyword(String), addKeyword(String, Object, int), removeKeyword(java.lang.String)
  - addKeyword
```
void addKeyword(java.lang.String keyword,
                java.lang.Object companion,
                int flags)
         throws java.lang.IllegalArgumentException,
                java.lang.UnsupportedOperationException
```
    Registering a keyword. If the keyword is already known to the TokenizerProperties then it is simply re-registered. If the known keyword has an associated companion it will be replaced against the given one.
    This version of addKeyword supports a bitmask of the Flags to modify the general tokenizer settings (see setParseFlags(int)) for this special element.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new keyword TokenizerProperty if the keyword is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the keyword is re-registered.
    A call to this method is equivalent to addKeyword(keyword, companion, flags, flags).
    
    Parameters:
    
    keyword - keyword to register
    
    companion - information object associated with this keyword
    
    flags - modification flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for keyword
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addKeyword(String), addKeyword(String, Object), removeKeyword(java.lang.String)
  - addKeyword
```
void addKeyword(java.lang.String keyword,
                java.lang.Object companion,
                int flags,
                int flagMask)
         throws java.lang.IllegalArgumentException,
                java.lang.UnsupportedOperationException
```
    Registering a keyword with a set of flags and an associated flag mask..
    The method is an extension to addKeyword(String, Object, int) having a bitmask for the flags that are explicitely specified for the pattern property. All other flag values (states) should be taken from the TokenizerProperty instance or from the Tokenizer.
    
    Parameters:
    
    keyword - keyword to register
    
    companion - information object associated with this keyword
    
    flags - modification flags
    
    flagMask - flags that have valid values in the parameter flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed for keyword
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    addKeyword(String), addKeyword(String, Object), removeKeyword(java.lang.String)
  - removeKeyword
```
void removeKeyword(java.lang.String keyword)
            throws java.lang.IllegalArgumentException
```
    Removing a keyword property from the store. If the keyword is not known then the method does nothing.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_REMOVED to all registered listeners if a keyword property is actually removed.
    
    Parameters:
    
    keyword - keyword to remove
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
    
    See Also:
    
    addKeyword(String), addKeyword(String, Object), addKeyword(String, Object, int)
  - getKeywordCompanion
```
java.lang.Object getKeywordCompanion(java.lang.String keyword)
                              throws java.lang.IllegalArgumentException
```
    Retrieving the companion of the given special sequence. If the special sequence doesn't exist the method returns null.
    
    Parameters:
    
    keyword - keyword thats companion is sought
    
    Returns:
    
    the object associated with the keyword
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
  - getKeywords
```
java.util.Iterator getKeywords()
```
    This method returns an Iterator of TokenizerProperty objects. Each TokenizerProperty object contains a keyword and the companion if it exists.
    
    Returns:
    
    enumeration of TokenizerProperty objects
  - keywordExists
```
boolean keywordExists(java.lang.String keyword)
```
    Checks if the given keyword is known to the TokenizerProperties. The method accepts both empty and null strings for keyword by returning false.
    
    Parameters:
    
    keyword - keyword to search
    
    Returns:
    
    true if the keyword is known, false otherwise
  - getKeyword
```
TokenizerProperty getKeyword(java.lang.String keyword)
                      throws java.lang.IllegalArgumentException
```
    Get the full description of a keyword property. The method returns null if the passed keyword image is unknown (keywordExists(java.lang.String) would return false).
    
    Parameters:
    
    keyword - keyword to search
    
    Returns:
    
    the full sequence description or null
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
  - addPattern
```
void addPattern(java.lang.String pattern)
         throws java.lang.IllegalArgumentException,
                java.lang.UnsupportedOperationException
```
    Registering a pattern. Pattern can describe identifiers, numbers etc. They provide a way to deal with token that cannot be enumerated like keywords.
    A pattern is usually a regular expression that is used by Pattern. But implementations of PatternHandler may use other pattern syntaxes, for example the simpler syntax used for file path matching.
    Pattern are applied to input data in the order of their registration. The first matching pattern stops the iteration.
    Pattern matching is a rather complex operation that may have a significant impact on the speed of a Tokenizer. On the other hand, pattern may be used instead of string or comment descriptions (see addString(java.lang.String, java.lang.String, java.lang.String) etc.) or if a pattern matching would be performed after the return of a token of type Token.NORMAL.
    If the given pattern is already known to the parser, it will simply be re-registered. Using this method on a known pattern with an associated companion will remove that companion.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new pattern TokenizerProperty if the pattern is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the pattern is re-registered.
    
    Parameters:
    
    pattern - the regular expression to be added
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty pattern is passed
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    removePattern(java.lang.String), addPattern(String, Object), addPattern(String, Object, int)
  - addPattern
```
void addPattern(java.lang.String pattern,
                java.lang.Object companion)
         throws java.lang.IllegalArgumentException,
                java.lang.UnsupportedOperationException
```
    Registering a pattern with an associated object. See the description of the addPattern(String) for details on pattern.
    If the given pattern is already known to the parser, it will simply be re-registered. The associated companion will be replaced against the new one.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new pattern TokenizerProperty if the pattern is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the pattern is re-registered.
    
    Parameters:
    
    pattern - the regular expression to be added
    
    companion - information object associated with this pattern
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty pattern is passed
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    removePattern(java.lang.String), addPattern(String), addPattern(String, Object, int)
  - addPattern
```
void addPattern(java.lang.String pattern,
                java.lang.Object companion,
                int flags)
         throws java.lang.IllegalArgumentException,
                java.lang.UnsupportedOperationException
```
    Registering a pattern with an associated object. See the description of the addPattern(String) for details on pattern.
    If the given pattern is already known to the parser, it will simply be re-registered. The associated companion will be replaced against the new one.
    This version of addPattern supports a bitmask of the Flags to modify the general tokenizer settings (see setParseFlags(int)) for this special pattern.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) with the new pattern TokenizerProperty if the pattern is a new one, or of type TokenizerPropertyEvent.PROPERTY_MODIFIED if the pattern is re-registered.
    A call to this method is equivalent to addPattern(pattern, companion, flags, flags).
    
    Parameters:
    
    pattern - the regular expression to be added
    
    companion - information object associated with this keyword
    
    flags - modification flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty pattern is passed
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    removePattern(java.lang.String), addPattern(String), addPattern(String, Object)
  - addPattern
```
void addPattern(java.lang.String pattern,
                java.lang.Object companion,
                int flags,
                int flagMask)
         throws java.lang.IllegalArgumentException,
                java.lang.UnsupportedOperationException
```
    Registering a pattern with an associated object and explicitely given flags. See the description of the addPattern(String) for details on pattern.
    The method is an extension to addPattern(String, Object, int) having a bitmask for the flags that are specified for the pattern property. All other flag values (states) should be taken from the TokenizerProperty instance or from the Tokenizer.
    
    Parameters:
    
    pattern - the regular expression to be added
    
    companion - information object associated with this keyword
    
    flags - values for modification flags
    
    flagMask - flags that have valid values in the parameter flags
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty pattern is passed
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
    
    See Also:
    
    removePattern(java.lang.String), addPattern(String), addPattern(String, Object), addPattern(String, Object, int)
  - removePattern
```
void removePattern(java.lang.String pattern)
            throws java.lang.IllegalArgumentException
```
    Removing a pattern. The method does nothing if the given pattern is not known to this TokenizerProperties instance. The method may throw an IllegalArgumentException if the given pattern is null or empty.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_REMOVED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)) if a pattern is actually removed.
    
    Parameters:
    
    pattern - the regular expression to be removed
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an empty string is passed
  - getPatternCompanion
```
java.lang.Object getPatternCompanion(java.lang.String pattern)
                              throws java.lang.IllegalArgumentException
```
    Retrieving the information associated with a given pattern. If the pattern is not known to the parser, null will be returned.
    If You need to know if a pattern is known to this TokenizerProperties instance with or without a companion, use the method patternExists(java.lang.String) instead.
    
    Parameters:
    
    pattern - the regular expression to be removed
    
    Returns:
    
    the associated information or null
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an emtpy pattern is passed
  - patternExists
```
boolean patternExists(java.lang.String pattern)
```
    Checks if the given pattern is known to the parser. The method accepts both empty and null strings for pattern by returning false.
    Note that there is no choice of parsing flags (different to the addString(String, String, String, Object, int) method), since it makes generally no sense to "overload" properties.
    
    Parameters:
    
    pattern - the regular expression to be looked for
    
    Returns:
    
    true if the pattern is registered, false otherwise
  - getPattern
```
TokenizerProperty getPattern(java.lang.String pattern)
                      throws java.lang.IllegalArgumentException
```
    Get the full description of a pattern property. The method returns null if the passed pattern parameter cannot be mapped to a known pattern description (patternExists(java.lang.String) would return false).
    
    Parameters:
    
    pattern - the regular expression to be looked for
    
    Returns:
    
    the full pattern description or null
    
    Throws:
    
    java.lang.IllegalArgumentException - when null or an emtpy pattern is passed
  - getPatterns
```
java.util.Iterator getPatterns()
```
    This method returns an Iterator of TokenizerProperty objects. Each TokenizerProperty object contains a pattern and its companion if such an associated object exists.
    
    Returns:
    
    enumeration of TokenizerProperty objects
  - addProperty
```
void addProperty(TokenizerProperty property)
          throws java.lang.IllegalArgumentException,
                 java.lang.UnsupportedOperationException
```
    Registering a TokenizerProperty. This can be a keyword, comment etc., even whitespaces, separators and property types not defined in Token.
    If the property is already known to this TokenizerProperties instance then it is simply re-registered.
    An implementation of this interface should fire a TokenizerPropertyEvent of type TokenizerPropertyEvent.PROPERTY_ADDED or TokenizerPropertyEvent.PROPERTY_MODIFIED to all registered listeners (see addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)).
    An implementation of this interface may or may not support adding of whitespaces and separators.
    
    Parameters:
    
    property - property to register
    
    Throws:
    
    java.lang.IllegalArgumentException - when null, an incomplete or otherwise unusable property is passed
    
    java.lang.UnsupportedOperationException - if the method is not available for an implementation of the TokenizerProperties interface
  - removeProperty
```
void removeProperty(TokenizerProperty property)
             throws java.lang.IllegalArgumentException
```
    Deregistering a TokenizerProperty from the store. If the property is not known the method does nothing.
    
    Parameters:
    
    property - property to register
    
    Throws:
    
    java.lang.IllegalArgumentException - when null, an incomplete or otherwise unusable property is passed
  - getProperties
```
java.util.Iterator getProperties()
```
    This method returns an Iterator of TokenizerProperty objects.
    
    Returns:
    
    enumeration of TokenizerProperty objects
  - propertyExists
```
boolean propertyExists(TokenizerProperty property)
```
    Checks if the given TokenizerProperty is known to this TokenizerProperties instance. The method compares the characteristics given in property against all known properties.
    The method accepts null for property by returning false.
    
    Parameters:
    
    property - the property to search
    
    Returns:
    
    true if the property is known, false otherwise
  - addTokenizerPropertyListener
```
void addTokenizerPropertyListener(TokenizerPropertyListener listener)
```
    Registering a new TokenizerPropertyListener. An implementation of the TokenizerProperties interface should call the approbriate methods in the TokenizerPropertyListener interface for all registered listeners whenever a TokenizerProperty is added, removed or modified.
    Adding is done by one of the add... calls like addKeyword(java.lang.String) or addString(java.lang.String, java.lang.String, java.lang.String).
    Modifications are re-registering of keywords, comments etc. regardless if companions or parse flags actually changed. Also, setWhitespaces(java.lang.String) and setSeparators(java.lang.String) are modifications.
    Removals are performed by the remove... calls, for instance removeKeyword(java.lang.String).
    Similar to the policy of event listener registration in the JDK, passing null does nothing.
    
    Parameters:
    
    listener - the new TokenizerPropertyListener
    
    See Also:
    
    removeTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)
  - removeTokenizerPropertyListener
```
void removeTokenizerPropertyListener(TokenizerPropertyListener listener)
```
    Removing a listener from the list of registered TokenizerPropertyListener instances. If the given listener is null or unknown, nothing is done.
    
    Parameters:
    
    listener - the TokenizerPropertyListener to deregister
    
    See Also:
    
    addTokenizerPropertyListener(de.susebox.jtopas.TokenizerPropertyListener)

Interface TokenizerProperties

Field Summary

Method Summary

Field Detail

DEFAULT_WHITESPACES

DEFAULT_SEPARATORS

DEFAULT_BLOCK_COMMENT_START

DEFAULT_BLOCK_COMMENT_END

DEFAULT_LINE_COMMENT

DEFAULT_STRING_START

DEFAULT_STRING_END

DEFAULT_STRING_ESCAPE

DEFAULT_CHAR_START

DEFAULT_CHAR_END

DEFAULT_CHAR_ESCAPE

Method Detail

setParseFlags

getParseFlags

isFlagSet

isFlagSet

setWhitespaces

addWhitespaces

removeWhitespaces

getWhitespaces

setSeparators

addSeparators

removeSeparators

getSeparators

addString

addString

addString

addString

removeString

getStringCompanion

stringExists

getString

getStrings

addLineComment

addLineComment

addLineComment

addLineComment

removeLineComment

getLineCommentCompanion

lineCommentExists

getLineComment

getLineComments

addBlockComment

addBlockComment

addBlockComment

addBlockComment

removeBlockComment

getBlockCommentCompanion

blockCommentExists

getBlockComment

getBlockComments

addSpecialSequence

addSpecialSequence

addSpecialSequence

addSpecialSequence

removeSpecialSequence

getSpecialSequenceCompanion

getSpecialSequences

specialSequenceExists

getSpecialSequence

addKeyword

addKeyword

addKeyword

addKeyword

removeKeyword

getKeywordCompanion

getKeywords

keywordExists

getKeyword

addPattern

addPattern

addPattern

addPattern

removePattern

getPatternCompanion

patternExists