org.apache.lucene.analysis.standard
Interface StandardTokenizerInterface

All Known Implementing Classes:
StandardTokenizerImpl, StandardTokenizerImpl31, UAX29URLEmailTokenizerImpl, UAX29URLEmailTokenizerImpl31

public interface StandardTokenizerInterface

NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

Field Summary
static int YYEOF
          This character denotes the end of file
 
Method Summary
 int getNextToken()
          Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
 void getText(CharTermAttribute t)
          Copies the matched text into the CharTermAttribute
 int yychar()
          Returns the current position.
 int yylength()
          Returns the length of the matched text region.
 void yyreset(Reader reader)
          Resets the scanner to read from a new input stream.
 

Field Detail

YYEOF

static final int YYEOF
This character denotes the end of file

See Also:
Constant Field Values
Method Detail

getText

void getText(CharTermAttribute t)
Copies the matched text into the CharTermAttribute


yychar

int yychar()
Returns the current position.


yyreset

void yyreset(Reader reader)
Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL.

Parameters:
reader - the new input stream

yylength

int yylength()
Returns the length of the matched text region.


getNextToken

int getNextToken()
                 throws IOException
Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.

Returns:
the next token, YYEOF on end of stream
Throws:
IOException - if any I/O-Error occurs


Copyright © 2000-2011 Apache Software Foundation. All Rights Reserved.