Fast, general-purpose grammar-based tokenizer
StandardTokenizer implements the Word Break rules from the
Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29. Unlike
UAX29URLEmailTokenizer from the analysis module, URLs and email addresses are not
tokenized as single tokens, but are instead split up into tokens according to the UAX#29 word
Class Summary Class Description StandardAnalyzer StandardTokenizerA grammar-based tokenizer constructed with JFlex. StandardTokenizerFactoryFactory for
StandardTokenizerImplThis class implements Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.