Uses of Class
org.apache.lucene.analysis.Tokenizer
Package
Description
Text analysis.
Fast, general-purpose grammar-based tokenizer
StandardTokenizer
implements the Word Break rules from the
Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.-
Uses of Tokenizer in org.apache.lucene.analysis
Modifier and TypeMethodDescriptionfinal Tokenizer
TokenizerFactory.create()
Creates a TokenStream of the specified input using the default attribute factory.abstract Tokenizer
TokenizerFactory.create
(AttributeFactory factory) Creates a TokenStream of the specified input using the given AttributeFactoryModifierConstructorDescriptionTokenStreamComponents
(Tokenizer tokenizer) Creates a newAnalyzer.TokenStreamComponents
from a TokenizerTokenStreamComponents
(Tokenizer tokenizer, TokenStream result) Creates a newAnalyzer.TokenStreamComponents
instance -
Uses of Tokenizer in org.apache.lucene.analysis.standard
Modifier and TypeClassDescriptionfinal class
A grammar-based tokenizer constructed with JFlex.