StandardTokenizerInterface (The Adobe Experience Manager SDK 2022.11.9850.20221116T162329Z-220900)

All Known Implementing Classes:

StandardTokenizerImpl, StandardTokenizerImpl31, StandardTokenizerImpl34, StandardTokenizerImpl40, UAX29URLEmailTokenizerImpl, UAX29URLEmailTokenizerImpl31, UAX29URLEmailTokenizerImpl34, UAX29URLEmailTokenizerImpl36, UAX29URLEmailTokenizerImpl40
```
public interface StandardTokenizerInterface
```
Internal interface for supporting versioned grammars.

Field Summary

Fields
Modifier and Type Field Description

static int YYEOF
This character denotes the end of file

Method Summary

All Methods Instance Methods Abstract Methods
Modifier and Type	Method	Description
`int`	`getNextToken()`	Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
`void`	`getText(CharTermAttribute t)`	Copies the matched text into the CharTermAttribute
`int`	`yychar()`	Returns the current position.
`int`	`yylength()`	Returns the length of the matched text region.
`void`	`yyreset(java.io.Reader reader)`	Resets the scanner to read from a new input stream.

- Field Detail
  - YYEOF
```
static final int YYEOF
```
    This character denotes the end of file
    
    See Also:
    
    Constant Field Values
- Method Detail
  - getText
```
void getText(CharTermAttribute t)
```
    Copies the matched text into the CharTermAttribute
  - yychar
```
int yychar()
```
    Returns the current position.
  - yyreset
```
void yyreset(java.io.Reader reader)
```
    Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL.
    
    Parameters:
    
    reader - the new input stream
  - yylength
```
int yylength()
```
    Returns the length of the matched text region.
  - getNextToken
```
int getNextToken()
          throws java.io.IOException
```
    Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
    
    Returns:
    
    the next token, YYEOF on end of stream
    
    Throws:
    
    java.io.IOException - if any I/O-Error occurs