Uses of Class
org.apache.lucene.analysis.TokenStream
Package
Description
Text analysis.
Fast, general-purpose grammar-based tokenizer
StandardTokenizer
implements the Word Break rules from the
Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.Codecs API: API for customization of the encoding and structure of the index.
The logical representation of a
Document
for indexing and
searching.Code to maintain and access indices.
Some utility classes.
Utility classes for working with token streams as graphs.
-
Uses of TokenStream in org.apache.lucene.analysis
Modifier and TypeClassDescriptionfinal class
This class can be used if the token attributes of a TokenStream are intended to be consumed more than once.class
Abstract base class for TokenFilters that may remove tokens.class
An abstract TokenFilter that exposes its input stream as a graphclass
Normalizes token text to lower case.class
Removes stop words from a token stream.class
A TokenFilter is a TokenStream whose input is another TokenStream.class
A Tokenizer is a TokenStream whose input is a Reader.Modifier and TypeFieldDescriptionprotected final TokenStream
TokenFilter.input
The source of tokens for this filter.protected final TokenStream
Analyzer.TokenStreamComponents.sink
Sink tokenstream, such as the outer tokenfilter decorating the chain.Modifier and TypeMethodDescriptionabstract TokenStream
TokenFilterFactory.create
(TokenStream input) Transform the specified input TokenStreamAnalyzer.TokenStreamComponents.getTokenStream()
Returns the sinkTokenStream
protected TokenStream
Analyzer.normalize
(String fieldName, TokenStream in) Wrap the givenTokenStream
in order to apply normalization filters.protected final TokenStream
AnalyzerWrapper.normalize
(String fieldName, TokenStream in) TokenFilterFactory.normalize
(TokenStream input) Normalize the specified input TokenStream While the default implementation returns input unchanged, filters that should be applied at normalization time can delegate tocreate
method.final TokenStream
Analyzer.tokenStream
(String fieldName, Reader reader) Returns a TokenStream suitable forfieldName
, tokenizing the contents ofreader
.final TokenStream
Analyzer.tokenStream
(String fieldName, String text) Returns a TokenStream suitable forfieldName
, tokenizing the contents oftext
.static TokenStream
AutomatonToTokenStream.toTokenStream
(Automaton automaton) converts an automaton into a TokenStream.TokenFilter.unwrap()
protected TokenStream
AnalyzerWrapper.wrapTokenStreamForNormalization
(String fieldName, TokenStream in) Wraps / alters the given TokenStream for normalization purposes, taken from the wrapped Analyzer, to form new components.protected final TokenStream
DelegatingAnalyzerWrapper.wrapTokenStreamForNormalization
(String fieldName, TokenStream in) Modifier and TypeMethodDescriptionabstract TokenStream
TokenFilterFactory.create
(TokenStream input) Transform the specified input TokenStreamprotected TokenStream
Analyzer.normalize
(String fieldName, TokenStream in) Wrap the givenTokenStream
in order to apply normalization filters.protected final TokenStream
AnalyzerWrapper.normalize
(String fieldName, TokenStream in) TokenFilterFactory.normalize
(TokenStream input) Normalize the specified input TokenStream While the default implementation returns input unchanged, filters that should be applied at normalization time can delegate tocreate
method.TokenStreamToAutomaton.toAutomaton
(TokenStream in) Pulls the graph (includingPositionLengthAttribute
) from the providedTokenStream
, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.protected TokenStream
AnalyzerWrapper.wrapTokenStreamForNormalization
(String fieldName, TokenStream in) Wraps / alters the given TokenStream for normalization purposes, taken from the wrapped Analyzer, to form new components.protected final TokenStream
DelegatingAnalyzerWrapper.wrapTokenStreamForNormalization
(String fieldName, TokenStream in) ModifierConstructorDescriptionCachingTokenFilter
(TokenStream input) Create a new CachingTokenFilter aroundinput
.Create a newFilteringTokenFilter
.GraphTokenFilter
(TokenStream input) Create a new GraphTokenFilterCreate a new LowerCaseFilter, that normalizes token text to lower case.StopFilter
(TokenStream in, CharArraySet stopWords) Constructs a filter which removes words from the input TokenStream that are named in the Set.protected
TokenFilter
(TokenStream input) Construct a token stream filtering the given input.TokenStreamComponents
(Consumer<Reader> source, TokenStream result) Creates a newAnalyzer.TokenStreamComponents
instance.TokenStreamComponents
(Tokenizer tokenizer, TokenStream result) Creates a newAnalyzer.TokenStreamComponents
instance -
Uses of TokenStream in org.apache.lucene.analysis.standard
Modifier and TypeClassDescriptionfinal class
A grammar-based tokenizer constructed with JFlex.Modifier and TypeMethodDescriptionprotected TokenStream
StandardAnalyzer.normalize
(String fieldName, TokenStream in) Modifier and TypeMethodDescriptionprotected TokenStream
StandardAnalyzer.normalize
(String fieldName, TokenStream in) -
Uses of TokenStream in org.apache.lucene.codecs
Modifier and TypeMethodDescriptionStoredFieldsWriter.MergeVisitor.tokenStream
(Analyzer analyzer, TokenStream reuse) Modifier and TypeMethodDescriptionStoredFieldsWriter.MergeVisitor.tokenStream
(Analyzer analyzer, TokenStream reuse) -
Uses of TokenStream in org.apache.lucene.document
Modifier and TypeFieldDescriptionprotected TokenStream
Field.tokenStream
Pre-analyzed tokenStream for indexed fields; this is separate from fieldsData because you are allowed to have both; eg maybe field has a String value but you customize how it's tokenizedModifier and TypeMethodDescriptionFeatureField.tokenStream
(Analyzer analyzer, TokenStream reuse) Field.tokenStream
(Analyzer analyzer, TokenStream reuse) Field.tokenStreamValue()
The TokenStream for this field to be used when indexing, or null.Modifier and TypeMethodDescriptionvoid
Field.setTokenStream
(TokenStream tokenStream) Expert: sets the token stream to be used for indexing and causes isIndexed() and isTokenized() to return true.FeatureField.tokenStream
(Analyzer analyzer, TokenStream reuse) Field.tokenStream
(Analyzer analyzer, TokenStream reuse) ModifierConstructorDescriptionField
(String name, TokenStream tokenStream, IndexableFieldType type) Create field with TokenStream value.TextField
(String name, TokenStream stream) Creates a new un-stored TextField with TokenStream value. -
Uses of TokenStream in org.apache.lucene.index
Modifier and TypeMethodDescriptionIndexableField.tokenStream
(Analyzer analyzer, TokenStream reuse) Creates the TokenStream used for indexing this field.Modifier and TypeMethodDescriptionIndexableField.tokenStream
(Analyzer analyzer, TokenStream reuse) Creates the TokenStream used for indexing this field. -
Uses of TokenStream in org.apache.lucene.util
Modifier and TypeMethodDescriptionprotected Query
QueryBuilder.analyzeBoolean
(String field, TokenStream stream) Creates simple boolean query from the cached tokenstream contentsprotected Query
QueryBuilder.analyzeGraphBoolean
(String field, TokenStream source, BooleanClause.Occur operator) Creates a boolean query from a graph token stream.protected Query
QueryBuilder.analyzeGraphPhrase
(TokenStream source, String field, int phraseSlop) Creates graph phrase query from the tokenstream contentsprotected Query
QueryBuilder.analyzeMultiBoolean
(String field, TokenStream stream, BooleanClause.Occur operator) Creates complex boolean query from the cached tokenstream contentsprotected Query
QueryBuilder.analyzeMultiPhrase
(String field, TokenStream stream, int slop) Creates complex phrase query from the cached tokenstream contentsprotected Query
QueryBuilder.analyzePhrase
(String field, TokenStream stream, int slop) Creates simple phrase query from the cached tokenstream contentsprotected Query
QueryBuilder.analyzeTerm
(String field, TokenStream stream) Creates simple term query from the cached tokenstream contentsprotected Query
QueryBuilder.createFieldQuery
(TokenStream source, BooleanClause.Occur operator, String field, boolean quoted, int phraseSlop) Creates a query from a token stream. -
Uses of TokenStream in org.apache.lucene.util.graph
Modifier and TypeMethodDescriptionGraphTokenStreamFiniteStrings.getFiniteStrings()
Get all finite strings from the automaton.GraphTokenStreamFiniteStrings.getFiniteStrings
(int startState, int endState) Get all finite strings that start atstartState
and end atendState
.