Uses of Class
org.apache.lucene.analysis.Tokenizer

Packages that use Tokenizer

Package	Description
org.apache.lucene.analysis	Text analysis.
org.apache.lucene.analysis.standard	Fast, general-purpose grammar-based tokenizer StandardTokenizer implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

Uses of Tokenizer in org.apache.lucene.analysis

Fields in org.apache.lucene.analysis declared as Tokenizer

Modifier and Type	Field and Description
protected Tokenizer	Analyzer.TokenStreamComponents.source Original source of the tokens.

Methods in org.apache.lucene.analysis that return Tokenizer

Modifier and Type	Method and Description
Tokenizer	Analyzer.TokenStreamComponents.getTokenizer() Returns the component's Tokenizer

Constructors in org.apache.lucene.analysis with parameters of type Tokenizer

Constructor and Description
TokenStreamComponents(Tokenizer source) Creates a new Analyzer.TokenStreamComponents instance.
TokenStreamComponents(Tokenizer source, TokenStream result) Creates a new Analyzer.TokenStreamComponents instance.

Uses of Tokenizer in org.apache.lucene.analysis.standard

Subclasses of Tokenizer in org.apache.lucene.analysis.standard

Modifier and Type	Class and Description
class	StandardTokenizer A grammar-based tokenizer constructed with JFlex.