de.susebox.jtopas

Class StandardTokenizer

java.lang.Object

de.susebox.jtopas.AbstractTokenizer

de.susebox.jtopas.StandardTokenizer

All Implemented Interfaces:

Tokenizer, TokenizerPropertyListener

public class StandardTokenizer
extends AbstractTokenizer
implements Tokenizer, TokenizerPropertyListener

This is the mainstream Tokenizer. It implements the Tokenizer interface in a straightforward approach without too specialized parse optimizations.

Beside the Tokenizer interface, the class StandardTokenizer provides some basic features for cascading (nested) tokenizers. Consider the usual HTML pages found today in the WWW. Most of them are a mixture of regular HTML, cascading style sheets (CSS) and embedded JavaScript. These different languages use different syntaxes, so one needs varous tokenizers on the same input stream.

This Tokenizer implementation is not synchronized. Take care when using with multible threads.

Author:

Heiko Blau

See Also:

Tokenizer, TokenizerProperties

Constructor Summary

Constructors

Constructor and Description
StandardTokenizer() Default constructor that sets the tokenizer control flags as it would be approbriate for C/C++ and Java.
StandardTokenizer(TokenizerProperties properties) Contructing a StandardTokenizer with a backing TokenizerProperties instance.

Method Summary

Modifier and Type	Method and Description
void	close() Closing this tokenizer frees resources.
int	getRangeStart() This method returns the absolute offset in characters to the start of the parsed stream.
void	setSource(TokenizerSource source) Additionally to the common behaviour implemented in #de.susebox.jtopas.AbstractTokenizer#setSource, this method ajusts the state speicific to the StandardTokenizer class.

Methods inherited from class de.susebox.jtopas.AbstractTokenizer

addTokenizer, changeParseFlags, currentImage, currentlyAvailable, currentToken, getChar, getColumnNumber, getCurrentColumn, getCurrentLine, getKeywordHandler, getLineNumber, getParseFlags, getPatternHandler, getReadPosition, getSeparatorHandler, getSequenceHandler, getSource, getText, getTokenizerProperties, getWhitespaceHandler, hasMoreToken, nextImage, nextToken, propertyChanged, readMore, setKeywordHandler, setPatternHandler, setReadPositionAbsolute, setReadPositionRelative, setSeparatorHandler, setSequenceHandler, setSource, setTokenizerProperties, setWhitespaceHandler, switchTo

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface de.susebox.jtopas.Tokenizer

changeParseFlags, currentImage, currentlyAvailable, currentToken, getChar, getColumnNumber, getKeywordHandler, getLineNumber, getParseFlags, getPatternHandler, getReadPosition, getSeparatorHandler, getSequenceHandler, getSource, getText, getTokenizerProperties, getWhitespaceHandler, hasMoreToken, nextImage, nextToken, readMore, setKeywordHandler, setPatternHandler, setReadPositionAbsolute, setReadPositionRelative, setSeparatorHandler, setSequenceHandler, setTokenizerProperties, setWhitespaceHandler

Methods inherited from interface de.susebox.jtopas.TokenizerPropertyListener

propertyChanged

Constructor Detail

StandardTokenizer

public StandardTokenizer()

Default constructor that sets the tokenizer control flags as it would be approbriate for C/C++ and Java. Found token images are copied. No line nor column informations are provided. Nested comments are not allowed.
The tokenizer will use the TokenizerProperties.DEFAULT_WHITESPACES and TokenizerProperties.DEFAULT_SEPARATORS for whitespace and separator handling.

StandardTokenizer

public StandardTokenizer(TokenizerProperties properties)

Contructing a StandardTokenizer with a backing TokenizerProperties instance.

Parameters:

properties - an TokenizerProperties object containing the settings for the tokenizing process

Method Detail

getRangeStart

public int getRangeStart()

This method returns the absolute offset in characters to the start of the parsed stream. See the method description in Tokenizer.

Specified by:

getRangeStart in interface Tokenizer

Returns:

the absolute offset of the current text window in characters from the start of the data source of the Tokenizer

See Also:

AbstractTokenizer.getReadPosition()

setSource

public void setSource(TokenizerSource source)

Additionally to the common behaviour implemented in #de.susebox.jtopas.AbstractTokenizer#setSource, this method ajusts the state speicific to the StandardTokenizer class.

Specified by:

setSource in interface Tokenizer

Overrides:

setSource in class AbstractTokenizer

Parameters:

source - a TokenizerSource to read data from

See Also:

AbstractTokenizer.getSource()

close

public void close()

Closing this tokenizer frees resources.

Specified by:

close in interface Tokenizer

Overrides:

close in class AbstractTokenizer