Flags

```
public interface Flags
```
The interface defines flags that are used by various classes during tokenizing. A flag can be set in three ways:
- Globally for a TokenizerProperties object: The setting affects all Tokenizer instances that share this TokenizerProperties object as well as the TokenizerProperty objects registered in this TokenizerProperties that do haven't set the flag locally.
- Separately for a Tokenizer (see Tokenizer.changeParseFlags(int, int): A single Tokenizer will behave differently to the setting in the used TokenizerProperties object, but still follow the setting for a single TokenizerProperty object. Only a limited number of flags can be set for a Tokenizer, especially the flags that are "dynamic", applicable more for the tokenizing process than describing an attribute of a TokenizerProperty, e. g. F_COUNT_LINES and F_KEEP_DATA.
- Specifically for a single TokenizerProperty: This setting affects only the handling of the property and overrules both settings for the TokenizerProperties that contains the property, and settings for a Tokenizer using the TokenizerProperties object. Only a limited number of flags can be set for a singel property including the descriptive flags like F_NO_CASE, F_ALLOW_NESTED_COMMENTS and F_SINGLE_LINE_STRING.
Author:

Heiko Blau

See Also:

TokenizerProperties

Field Summary

Fields
Modifier and Type	Field and Description
`static short`	`F_ALLOW_NESTED_COMMENTS` Nested block comments are normally not allowed.
`static short`	`F_CASE` Deprecated. for properties with a case handling different to the global settings of a `TokenizerProperties` instance use the constructor `TokenizerProperty(int, java.lang.String[], java.lang.Object, int, int)`
`static short`	`F_COUNT_LINES` Tells a `Tokenizer` to count lines and columns.
`static short`	`F_FREE_PATTERN` Treat pattern the same way as whitespaces, separators or special sequences.
`static short`	`F_KEEP_DATA` Set this flag to let a `Tokenizer` buffer all data.
`static short`	`F_NO_CASE` When this flag is set globally for a `TokenizerProperties` instance (see `#setParseFlags`, input data is generally treated case-insensitive.
`static short`	`F_RETURN_BLOCK_COMMENTS` Return block comments.
`static short`	`F_RETURN_IMAGE_PARTS` By setting this flag for a `TokenizerProperties` instance, a `Tokenizer` or for a single property, a tokenizer returns not only the token images but also image parts (see `Token.getImageParts()`).
`static short`	`F_RETURN_LINE_COMMENTS` Return line comments.
`static short`	`F_RETURN_SIMPLE_WHITESPACES` Return simple whitespaces.
`static short`	`F_RETURN_WHITESPACES` In many cases, parsers are not interested in whitespaces.
`static short`	`F_SINGLE_LINE_STRING` Per default, strings are all characters between and including a pair of string start and end sequences, regardless if there are line separators in between.
`static short`	`F_TOKEN_POS_ONLY` For performance and memory reasons, this flag is used to avoid copy operations for every token.

- Field Detail
  - F_NO_CASE
```
static final short F_NO_CASE
```
    When this flag is set globally for a TokenizerProperties instance (see #setParseFlags, input data is generally treated case-insensitive. Specific properties may still be treated case-sensitive. Set this flag set in the flag mask and cleared in the corresponding flags).
    Implementation note: The flag should be applicable for both TokenizerProperties and TokenizerProperty instances. It should not to be used dynamically (Tokenizer.changeParseFlags(int, int)).
    
    See Also:
    
    Constant Field Values
  - F_CASE
```
static final short F_CASE
```
    Deprecated. for properties with a case handling different to the global settings of a TokenizerProperties instance use the constructor TokenizerProperty(int, java.lang.String[], java.lang.Object, int, int)
    
    General compare operations are case-sensitive, that means 'A' equals 'A' but not 'a'. It is not nessecary to set this flag, since case-sensitive comparison is the default.
    The flag was mainly used in conjunction with F_NO_CASE. If F_NO_CASE is set via TokenizerProperties.setParseFlags(int), F_CASE can be used for single properties where case-sensitivity is nessecary inspite of the global case-insensitivity.
    If neither F_CASE nor F_NO_CASE is set, F_CASE is assumed. If both flags are set, F_CASE takes preceedence.
    Implementation note: The flag should be applicable for both TokenizerProperties and TokenizerProperty instances. It should not to be used dynamically (Tokenizer.changeParseFlags(int, int)).
    
    See Also:
    
    Constant Field Values
  - F_TOKEN_POS_ONLY
```
static final short F_TOKEN_POS_ONLY
```
    For performance and memory reasons, this flag is used to avoid copy operations for every token. The token image itself is not returned in a Token instance, only its position and length in the input stream.
    Implementation note: The flag should be applicable for TokenizerProperties, and TokenizerProperty instances. It should also be a dynamic flag that can be switched on and off during runtime using Tokenizer.changeParseFlags(int, int).
    
    See Also:
    
    Constant Field Values
  - F_KEEP_DATA
```
static final short F_KEEP_DATA
```
    Set this flag to let a Tokenizer buffer all data. Usually, a tokenizer will apply a strategie to allocate only a reasonable amount of memory.
    Implementation note: The flag should be applicable for TokenizerProperties and Tokenizer objects, but not for single TokenizerProperty instances. It could also be a dynamic flag that can be switched on and off during runtime of a tokenizer (Tokenizer.changeParseFlags(int, int)), although it is generally set before parsing starts.
    
    See Also:
    
    Constant Field Values
  - F_COUNT_LINES
```
static final short F_COUNT_LINES
```
    Tells a Tokenizer to count lines and columns. The tokenizer may use java.lang.System.getProperty("line.separator") to obtain the end-of-line sequence or accept different line separator sequences for a better portability: single carriage return (Mac OS), single line feed (Unix), combination of carriage return and line feed (Windows OS).
    Usually, the end-of-line characters '\r' and '\n' are whitespaces. If they are also part of one or more special sequences or pattern, it is NOT guaranteed that the line counting mechanism of a Tokenizer implementation finds these occurences. This is in order to maintain a good performance, since otherwise there would be a potential huge amount of unsuccessfull newline scans in these tokens. Consider defining special sequences for '\r', '\n' and '\r\n' alone and remove them from the whitespace set, if You cannot live with the described limitation.
    Implementation note: The flag should be applicable for TokenizerProperties and Tokenizer objects, but not for single TokenizerProperty instances. It could also be a dynamic flag that can be switched on and off during runtime of a tokenizer, although it is generally set before parsing starts.
    
    See Also:
    
    Constant Field Values
  - F_ALLOW_NESTED_COMMENTS
```
static final short F_ALLOW_NESTED_COMMENTS
```
    Nested block comments are normally not allowed. This flag changes the default behaviour.
    Implementation note: The flag should be applicable for both TokenizerProperties and TokenizerProperty instances. It should not to be used dynamically (as in versions of JTopas prior to 0.8).
    
    See Also:
    
    Constant Field Values
  - F_FREE_PATTERN
```
static final short F_FREE_PATTERN
```
    Treat pattern the same way as whitespaces, separators or special sequences. Pattern of this type are recognized anywhere outside comments and strings. They terminate normal token. In fact, strings and comments could be described as free pattern.
    Without this flag, pattern are treated in the same way as normal token. They are preceeded and followed by whitespaces, separators or special sequences.
    Implementation note: The flag should be applicable for both TokenizerProperties and TokenizerProperty instances. It should not to be used dynamically.
    
    See Also:
    
    Constant Field Values
  - F_RETURN_SIMPLE_WHITESPACES
```
static final short F_RETURN_SIMPLE_WHITESPACES
```
    Return simple whitespaces. These whitespaces are the ones set by #setWhitespaces. The flag is part of the composite mask F_RETURN_WHITESPACES.
    Implementation note: The flag should be applicable for TokenizerProperties and Tokenizer, but not for single TokenizerProperty instances. It is also a dynamic flag that can be switched on and off during runtime of a tokenizer (Note:: Flags for a single TokenizerProperty take precedence over other settings).
    
    See Also:
    
    Constant Field Values
  - F_RETURN_BLOCK_COMMENTS
```
static final short F_RETURN_BLOCK_COMMENTS
```
    Return block comments. The flag is part of the composite mask F_RETURN_WHITESPACES.
    Implementation note: The flag should be applicable for TokenizerProperties, Tokenizer and for single TokenizerProperty instances. It is also a dynamic flag that can be switched on and off during runtime of a tokenizer (Note:: Flags for a single TokenizerProperty take precedence over other settings).
    
    See Also:
    
    Constant Field Values
  - F_RETURN_LINE_COMMENTS
```
static final short F_RETURN_LINE_COMMENTS
```
    Return line comments. The flag is part of the composite mask F_RETURN_WHITESPACES.
    Implementation note: The flag should be applicable for TokenizerProperties, Tokenizer and for single TokenizerProperty instances. It is also a dynamic flag that can be switched on and off during runtime of a tokenizer (Note:: Flags for a single TokenizerProperty take precedence over other settings).
    
    See Also:
    
    Constant Field Values
  - F_RETURN_WHITESPACES
```
static final short F_RETURN_WHITESPACES
```
    In many cases, parsers are not interested in whitespaces. If You are, use this value to force the tokenizer to return whitespace sequences and comments as a token. Per default, the flag is not set.
    You can control the whitespace policy with finer granularity by using the flags F_RETURN_SIMPLE_WHITESPACES, F_RETURN_BLOCK_COMMENTS and F_RETURN_LINE_COMMENTS either by setting it generally for a TokenizerProperties or a single Tokenizer object or even more specific for a single TokenizerProperties.
    
    See Also:
    
    Constant Field Values
  - F_SINGLE_LINE_STRING
```
static final short F_SINGLE_LINE_STRING
```
    Per default, strings are all characters between and including a pair of string start and end sequences, regardless if there are line separators in between. This flag changes that behaviour for the TokenizerProperties instance in general or for a single string property.
    Implementation note: The flag should be applicable for both TokenizerProperties and TokenizerProperty instances. It should not to be used dynamically.
    
    See Also:
    
    Constant Field Values
  - F_RETURN_IMAGE_PARTS
```
static final short F_RETURN_IMAGE_PARTS
```
    By setting this flag for a TokenizerProperties instance, a Tokenizer or for a single property, a tokenizer returns not only the token images but also image parts (see Token.getImageParts()).
    Implementation note: The flag should be applicable for TokenizerProperties, Tokenizer and for single TokenizerProperty instances.
    
    See Also:
    
    Constant Field Values

Interface Flags

Field Summary

Field Detail

F_NO_CASE

F_CASE

F_TOKEN_POS_ONLY

F_KEEP_DATA

F_COUNT_LINES

F_ALLOW_NESTED_COMMENTS

F_FREE_PATTERN

F_RETURN_SIMPLE_WHITESPACES

F_RETURN_BLOCK_COMMENTS

F_RETURN_LINE_COMMENTS

F_RETURN_WHITESPACES

F_SINGLE_LINE_STRING

F_RETURN_IMAGE_PARTS