Uses of Class
org.apache.lucene.analysis.TokenStream

Packages that use TokenStream

org.apache.lucene.analysis	API and code to convert text into indexable/searchable tokens.
org.apache.lucene.analysis.ar	Analyzer for Arabic.
org.apache.lucene.analysis.br	Analyzer for Brazilian Portuguese.
org.apache.lucene.analysis.cjk	Analyzer for Chinese, Japanese, and Korean, which indexes bigrams (overlapping groups of two adjacent Han characters).
org.apache.lucene.analysis.cn	Analyzer for Chinese, which indexes unigrams (individual chinese characters).
org.apache.lucene.analysis.cn.smart	Analyzer for Simplified Chinese, which indexes words.
org.apache.lucene.analysis.compound	A filter that decomposes compound words you find in many Germanic languages into the word parts.
org.apache.lucene.analysis.cz	Analyzer for Czech.
org.apache.lucene.analysis.de	Analyzer for German.
org.apache.lucene.analysis.el	Analyzer for Greek.
org.apache.lucene.analysis.fa	Analyzer for Persian.
org.apache.lucene.analysis.fr	Analyzer for French.
org.apache.lucene.analysis.miscellaneous	Miscellaneous TokenStreams
org.apache.lucene.analysis.ngram	Character n-gram tokenizers and filters.
org.apache.lucene.analysis.nl	Analyzer for Dutch.
org.apache.lucene.analysis.payloads	Provides various convenience classes for creating payloads on Tokens.
org.apache.lucene.analysis.position	Filter for assigning position increments.
org.apache.lucene.analysis.query	Automatically filter high-frequency stopwords.
org.apache.lucene.analysis.reverse	Filter to reverse token text.
org.apache.lucene.analysis.ru	Analyzer for Russian.
org.apache.lucene.analysis.shingle	Word n-gram filters
org.apache.lucene.analysis.snowball	TokenFilter and Analyzer implementations that use Snowball stemmers.
org.apache.lucene.analysis.standard	A fast grammar-based tokenizer constructed with JFlex.
org.apache.lucene.analysis.th	Analyzer for Thai.
org.apache.lucene.collation	CollationKeyFilter and ICUCollationKeyFilter convert each token into its binary CollationKey using the provided Collator, and then encode the CollationKey as a String using IndexableBinaryStringTools, to allow it to be stored as an index term.
org.apache.lucene.document	The logical representation of a Document for indexing and searching.
org.apache.lucene.index.memory	High-performance single-document main memory Apache Lucene fulltext search index.
org.apache.lucene.search.highlight	The highlight package contains classes to provide "keyword in context" features typically used to highlight search terms in the text of results pages.
org.apache.lucene.wikipedia.analysis	Tokenizer that is aware of Wikipedia syntax.
org.apache.lucene.wordnet	This package uses synonyms defined by WordNet.

Uses of TokenStream in org.apache.lucene.analysis

Subclasses of TokenStream in org.apache.lucene.analysis

class	ASCIIFoldingFilter This class converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into their ASCII equivalents, if one exists.
class	CachingTokenFilter This class can be used if the token attributes of a TokenStream are intended to be consumed more than once.
class	CharTokenizer An abstract base class for simple, character-oriented tokenizers.
class	ISOLatin1AccentFilter Deprecated. If you build a new index, use ASCIIFoldingFilter which covers a superset of Latin 1. This class is included for use with existing indexes and will be removed in a future release (possibly Lucene 4.0).
class	KeywordTokenizer Emits the entire input as a single token.
class	LengthFilter Removes words that are too long or too short from the stream.
class	LetterTokenizer A LetterTokenizer is a tokenizer that divides text at non-letters.
class	LowerCaseFilter Normalizes token text to lower case.
class	LowerCaseTokenizer LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together.
class	NumericTokenStream Expert: This class provides a TokenStream for indexing numeric values that can be used by NumericRangeQuery or NumericRangeFilter.
class	PorterStemFilter Transforms the token stream as per the Porter stemming algorithm.
class	StopFilter Removes stop words from a token stream.
class	TeeSinkTokenFilter This TokenFilter provides the ability to set aside attribute states that have already been analyzed.
static class	TeeSinkTokenFilter.SinkTokenStream
class	TokenFilter A TokenFilter is a TokenStream whose input is another TokenStream.
class	Tokenizer A Tokenizer is a TokenStream whose input is a Reader.
class	WhitespaceTokenizer A WhitespaceTokenizer is a tokenizer that divides text at whitespace.

Fields in org.apache.lucene.analysis declared as TokenStream

protected TokenStream

TokenFilter.input The source of tokens for this filter.

Methods in org.apache.lucene.analysis that return TokenStream

TokenStream	WhitespaceAnalyzer.reusableTokenStream(String fieldName, Reader reader)
TokenStream	StopAnalyzer.reusableTokenStream(String fieldName, Reader reader)
TokenStream	SimpleAnalyzer.reusableTokenStream(String fieldName, Reader reader)
TokenStream	PerFieldAnalyzerWrapper.reusableTokenStream(String fieldName, Reader reader)
TokenStream	KeywordAnalyzer.reusableTokenStream(String fieldName, Reader reader)
TokenStream	Analyzer.reusableTokenStream(String fieldName, Reader reader) Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method.
TokenStream	WhitespaceAnalyzer.tokenStream(String fieldName, Reader reader)
TokenStream	StopAnalyzer.tokenStream(String fieldName, Reader reader) Filters LowerCaseTokenizer with StopFilter.
TokenStream	SimpleAnalyzer.tokenStream(String fieldName, Reader reader)
TokenStream	PerFieldAnalyzerWrapper.tokenStream(String fieldName, Reader reader)
TokenStream	KeywordAnalyzer.tokenStream(String fieldName, Reader reader)
abstract TokenStream	Analyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis with parameters of type TokenStream

ASCIIFoldingFilter(TokenStream input)

CachingTokenFilter(TokenStream input)

ISOLatin1AccentFilter(TokenStream input)
Deprecated.

LengthFilter(TokenStream in, int min, int max)
Build a filter that removes words that are too long or too short from the text.

LowerCaseFilter(TokenStream in)

PorterStemFilter(TokenStream in)

StopFilter(boolean enablePositionIncrements, TokenStream in, Set<?> stopWords)
Constructs a filter which removes words from the input TokenStream that are named in the Set.

StopFilter(boolean enablePositionIncrements, TokenStream input, Set<?> stopWords, boolean ignoreCase)
Construct a token stream filtering the given input.

TeeSinkTokenFilter(TokenStream input)
Instantiates a new TeeSinkTokenFilter.

TokenFilter(TokenStream input)
Construct a token stream filtering the given input.

Uses of TokenStream in org.apache.lucene.analysis.ar

Subclasses of TokenStream in org.apache.lucene.analysis.ar

class	ArabicLetterTokenizer Tokenizer that breaks text into runs of letters and diacritics.
class	ArabicNormalizationFilter A TokenFilter that applies ArabicNormalizer to normalize the orthography.
class	ArabicStemFilter A TokenFilter that applies ArabicStemmer to stem Arabic words..

Methods in org.apache.lucene.analysis.ar that return TokenStream

TokenStream	ArabicAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	ArabicAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.ar with parameters of type TokenStream

ArabicNormalizationFilter(TokenStream input)

ArabicStemFilter(TokenStream input)

Uses of TokenStream in org.apache.lucene.analysis.br

Subclasses of TokenStream in org.apache.lucene.analysis.br

class

BrazilianStemFilter A TokenFilter that applies BrazilianStemmer.

Methods in org.apache.lucene.analysis.br that return TokenStream

TokenStream	BrazilianAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	BrazilianAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.br with parameters of type TokenStream

BrazilianStemFilter(TokenStream in)

BrazilianStemFilter(TokenStream in, Set exclusiontable)

Uses of TokenStream in org.apache.lucene.analysis.cjk

Subclasses of TokenStream in org.apache.lucene.analysis.cjk

class

CJKTokenizer CJKTokenizer is designed for Chinese, Japanese, and Korean languages.

Methods in org.apache.lucene.analysis.cjk that return TokenStream

TokenStream	CJKAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	CJKAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Uses of TokenStream in org.apache.lucene.analysis.cn

Subclasses of TokenStream in org.apache.lucene.analysis.cn

class	ChineseFilter A TokenFilter with a stop word table.
class	ChineseTokenizer Tokenize Chinese text as individual chinese characters.

Methods in org.apache.lucene.analysis.cn that return TokenStream

TokenStream	ChineseAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	ChineseAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.cn with parameters of type TokenStream

ChineseFilter(TokenStream in)

Uses of TokenStream in org.apache.lucene.analysis.cn.smart

Subclasses of TokenStream in org.apache.lucene.analysis.cn.smart

class	SentenceTokenizer Tokenizes input text into sentences.
class	WordTokenFilter A TokenFilter that breaks sentences into words.

Methods in org.apache.lucene.analysis.cn.smart that return TokenStream

TokenStream	SmartChineseAnalyzer.reusableTokenStream(String fieldName, Reader reader)
TokenStream	SmartChineseAnalyzer.tokenStream(String fieldName, Reader reader)

Constructors in org.apache.lucene.analysis.cn.smart with parameters of type TokenStream

WordTokenFilter(TokenStream in)
Construct a new WordTokenizer.

Uses of TokenStream in org.apache.lucene.analysis.compound

Subclasses of TokenStream in org.apache.lucene.analysis.compound

class	CompoundWordTokenFilterBase Base class for decomposition token filters.
class	DictionaryCompoundWordTokenFilter A TokenFilter that decomposes compound words found in many Germanic languages.
class	HyphenationCompoundWordTokenFilter A TokenFilter that decomposes compound words found in many Germanic languages.

Constructors in org.apache.lucene.analysis.compound with parameters of type TokenStream

CompoundWordTokenFilterBase(TokenStream input, Set dictionary)

CompoundWordTokenFilterBase(TokenStream input, Set dictionary, boolean onlyLongestMatch)

CompoundWordTokenFilterBase(TokenStream input, Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)

CompoundWordTokenFilterBase(TokenStream input, String[] dictionary)

CompoundWordTokenFilterBase(TokenStream input, String[] dictionary, boolean onlyLongestMatch)

CompoundWordTokenFilterBase(TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)

DictionaryCompoundWordTokenFilter(TokenStream input, Set dictionary)

DictionaryCompoundWordTokenFilter(TokenStream input, Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)

DictionaryCompoundWordTokenFilter(TokenStream input, String[] dictionary)

DictionaryCompoundWordTokenFilter(TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)

HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, Set dictionary)

HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)

HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, String[] dictionary)

HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)

Uses of TokenStream in org.apache.lucene.analysis.cz

Methods in org.apache.lucene.analysis.cz that return TokenStream

TokenStream	CzechAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	CzechAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Uses of TokenStream in org.apache.lucene.analysis.de

Subclasses of TokenStream in org.apache.lucene.analysis.de

class

GermanStemFilter A TokenFilter that stems German words.

Methods in org.apache.lucene.analysis.de that return TokenStream

TokenStream	GermanAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	GermanAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.de with parameters of type TokenStream

GermanStemFilter(TokenStream in)

GermanStemFilter(TokenStream in, Set exclusionSet)
Builds a GermanStemFilter that uses an exclusion table.

Uses of TokenStream in org.apache.lucene.analysis.el

Subclasses of TokenStream in org.apache.lucene.analysis.el

class

GreekLowerCaseFilter Normalizes token text to lower case, removes some Greek diacritics, and standardizes final sigma to sigma.

Methods in org.apache.lucene.analysis.el that return TokenStream

TokenStream	GreekAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	GreekAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.el with parameters of type TokenStream

GreekLowerCaseFilter(TokenStream in)

Uses of TokenStream in org.apache.lucene.analysis.fa

Subclasses of TokenStream in org.apache.lucene.analysis.fa

class

PersianNormalizationFilter A TokenFilter that applies PersianNormalizer to normalize the orthography.

Methods in org.apache.lucene.analysis.fa that return TokenStream

TokenStream	PersianAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	PersianAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.fa with parameters of type TokenStream

PersianNormalizationFilter(TokenStream input)

Uses of TokenStream in org.apache.lucene.analysis.fr

Subclasses of TokenStream in org.apache.lucene.analysis.fr

class	ElisionFilter Removes elisions from a TokenStream.
class	FrenchStemFilter A TokenFilter that stems french words.

Methods in org.apache.lucene.analysis.fr that return TokenStream

TokenStream	FrenchAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	FrenchAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.fr with parameters of type TokenStream

ElisionFilter(TokenStream input)
Constructs an elision filter with standard stop words

ElisionFilter(TokenStream input, Set<?> articles)
Constructs an elision filter with a Set of stop words

ElisionFilter(TokenStream input, String[] articles)
Constructs an elision filter with an array of stop words

FrenchStemFilter(TokenStream in)

FrenchStemFilter(TokenStream in, Set exclusiontable)

Uses of TokenStream in org.apache.lucene.analysis.miscellaneous

Subclasses of TokenStream in org.apache.lucene.analysis.miscellaneous

class	EmptyTokenStream An always exhausted token stream.
class	PrefixAndSuffixAwareTokenFilter Links two PrefixAwareTokenFilter.
class	PrefixAwareTokenFilter Joins two token streams and leaves the last token of the first stream available to be used when updating the token values in the second stream based on that token.
class	SingleTokenTokenStream A TokenStream containing a single token.

Methods in org.apache.lucene.analysis.miscellaneous that return TokenStream

TokenStream	PrefixAwareTokenFilter.getPrefix()
TokenStream	PrefixAwareTokenFilter.getSuffix()
TokenStream	PatternAnalyzer.tokenStream(String fieldName, Reader reader) Creates a token stream that tokenizes all the text in the given Reader; This implementation forwards to tokenStream(String, String) and is less efficient than tokenStream(String, String).
TokenStream	PatternAnalyzer.tokenStream(String fieldName, String text) Creates a token stream that tokenizes the given string into token terms (aka words).

Methods in org.apache.lucene.analysis.miscellaneous with parameters of type TokenStream

void	PrefixAwareTokenFilter.setPrefix(TokenStream prefix)
void	PrefixAwareTokenFilter.setSuffix(TokenStream suffix)

Constructors in org.apache.lucene.analysis.miscellaneous with parameters of type TokenStream

PrefixAndSuffixAwareTokenFilter(TokenStream prefix, TokenStream input, TokenStream suffix)

PrefixAwareTokenFilter(TokenStream prefix, TokenStream suffix)

Uses of TokenStream in org.apache.lucene.analysis.ngram

Subclasses of TokenStream in org.apache.lucene.analysis.ngram

class	EdgeNGramTokenFilter Tokenizes the given token into n-grams of given size(s).
class	EdgeNGramTokenizer Tokenizes the input from an edge into n-grams of given size(s).
class	NGramTokenFilter Tokenizes the input into n-grams of the given size(s).
class	NGramTokenizer Tokenizes the input into n-grams of the given size(s).

Constructors in org.apache.lucene.analysis.ngram with parameters of type TokenStream

EdgeNGramTokenFilter(TokenStream input)

EdgeNGramTokenFilter(TokenStream input, EdgeNGramTokenFilter.Side side, int minGram, int maxGram)
Creates EdgeNGramTokenFilter that can generate n-grams in the sizes of the given range

EdgeNGramTokenFilter(TokenStream input, String sideLabel, int minGram, int maxGram)
Creates EdgeNGramTokenFilter that can generate n-grams in the sizes of the given range

NGramTokenFilter(TokenStream input)
Creates NGramTokenFilter with default min and max n-grams.

NGramTokenFilter(TokenStream input, int minGram, int maxGram)
Creates NGramTokenFilter with given min and max n-grams.

Uses of TokenStream in org.apache.lucene.analysis.nl

Subclasses of TokenStream in org.apache.lucene.analysis.nl

class

DutchStemFilter A TokenFilter that stems Dutch words.

Methods in org.apache.lucene.analysis.nl that return TokenStream

TokenStream	DutchAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	DutchAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.nl with parameters of type TokenStream

DutchStemFilter(TokenStream _in)

DutchStemFilter(TokenStream _in, Set exclusiontable)
Builds a DutchStemFilter that uses an exclusion table.

DutchStemFilter(TokenStream _in, Set exclusiontable, Map stemdictionary)

Uses of TokenStream in org.apache.lucene.analysis.payloads

Subclasses of TokenStream in org.apache.lucene.analysis.payloads

class	DelimitedPayloadTokenFilter Characters before the delimiter are the "token", those after are the payload.
class	NumericPayloadTokenFilter Assigns a payload to a token based on the Token.type()
class	TokenOffsetPayloadTokenFilter Adds the Token.setStartOffset(int) and Token.setEndOffset(int) First 4 bytes are the start
class	TypeAsPayloadTokenFilter Makes the Token.type() a payload.

Constructors in org.apache.lucene.analysis.payloads with parameters of type TokenStream

DelimitedPayloadTokenFilter(TokenStream input)
Construct a token stream filtering the given input.

DelimitedPayloadTokenFilter(TokenStream input, char delimiter, PayloadEncoder encoder)

NumericPayloadTokenFilter(TokenStream input, float payload, String typeMatch)

TokenOffsetPayloadTokenFilter(TokenStream input)

TypeAsPayloadTokenFilter(TokenStream input)

Uses of TokenStream in org.apache.lucene.analysis.position

Subclasses of TokenStream in org.apache.lucene.analysis.position

class

PositionFilter Set the positionIncrement of all tokens to the "positionIncrement", except the first return token which retains its original positionIncrement value.

Constructors in org.apache.lucene.analysis.position with parameters of type TokenStream

PositionFilter(TokenStream input)
Constructs a PositionFilter that assigns a position increment of zero to all but the first token from the given input stream.

PositionFilter(TokenStream input, int positionIncrement)
Constructs a PositionFilter that assigns the given position increment to all but the first token from the given input stream.

Uses of TokenStream in org.apache.lucene.analysis.query

Methods in org.apache.lucene.analysis.query that return TokenStream

TokenStream	QueryAutoStopWordAnalyzer.reusableTokenStream(String fieldName, Reader reader)
TokenStream	QueryAutoStopWordAnalyzer.tokenStream(String fieldName, Reader reader)

Uses of TokenStream in org.apache.lucene.analysis.reverse

Subclasses of TokenStream in org.apache.lucene.analysis.reverse

class

ReverseStringFilter Reverse token string, for example "country" => "yrtnuoc".

Constructors in org.apache.lucene.analysis.reverse with parameters of type TokenStream

ReverseStringFilter(TokenStream in)
Create a new ReverseStringFilter that reverses all tokens in the supplied TokenStream.

ReverseStringFilter(TokenStream in, char marker)
Create a new ReverseStringFilter that reverses and marks all tokens in the supplied TokenStream.

Uses of TokenStream in org.apache.lucene.analysis.ru

Subclasses of TokenStream in org.apache.lucene.analysis.ru

class	RussianLetterTokenizer A RussianLetterTokenizer is a Tokenizer that extends LetterTokenizer by also allowing the basic latin digits 0-9.
class	RussianLowerCaseFilter Deprecated. Use LowerCaseFilter instead, which has the same functionality. This filter will be removed in Lucene 4.0
class	RussianStemFilter A TokenFilter that stems Russian words.

Methods in org.apache.lucene.analysis.ru that return TokenStream

TokenStream	RussianAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) TokenStream which tokenizes all the text in the provided Reader.
TokenStream	RussianAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.ru with parameters of type TokenStream

RussianLowerCaseFilter(TokenStream in)
Deprecated.

RussianStemFilter(TokenStream in)

Uses of TokenStream in org.apache.lucene.analysis.shingle

Subclasses of TokenStream in org.apache.lucene.analysis.shingle

class	ShingleFilter A ShingleFilter constructs shingles (token n-grams) from a token stream.
class	ShingleMatrixFilter A ShingleMatrixFilter constructs shingles (token n-grams) from a token stream.

Methods in org.apache.lucene.analysis.shingle that return TokenStream

TokenStream	ShingleAnalyzerWrapper.reusableTokenStream(String fieldName, Reader reader)
TokenStream	ShingleAnalyzerWrapper.tokenStream(String fieldName, Reader reader)

Constructors in org.apache.lucene.analysis.shingle with parameters of type TokenStream

ShingleFilter(TokenStream input)
Construct a ShingleFilter with default shingle size.

ShingleFilter(TokenStream input, int maxShingleSize)
Constructs a ShingleFilter with the specified single size from the TokenStream input

ShingleFilter(TokenStream input, String tokenType)
Construct a ShingleFilter with the specified token type for shingle tokens.

ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize)
Creates a shingle filter using default settings.

ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize, Character spacerCharacter)
Creates a shingle filter using default settings.

ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize, Character spacerCharacter, boolean ignoringSinglePrefixOrSuffixShingle)
Creates a shingle filter using the default ShingleMatrixFilter.TokenSettingsCodec.

ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize, Character spacerCharacter, boolean ignoringSinglePrefixOrSuffixShingle, ShingleMatrixFilter.TokenSettingsCodec settingsCodec)
Creates a shingle filter with ad hoc parameter settings.

Uses of TokenStream in org.apache.lucene.analysis.snowball

Subclasses of TokenStream in org.apache.lucene.analysis.snowball

class

SnowballFilter A filter that stems words using a Snowball-generated stemmer.

Methods in org.apache.lucene.analysis.snowball that return TokenStream

TokenStream	SnowballAnalyzer.reusableTokenStream(String fieldName, Reader reader) Returns a (possibly reused) StandardTokenizer filtered by a StandardFilter, a LowerCaseFilter, a StopFilter, and a SnowballFilter
TokenStream	SnowballAnalyzer.tokenStream(String fieldName, Reader reader) Constructs a StandardTokenizer filtered by a StandardFilter, a LowerCaseFilter, a StopFilter, and a SnowballFilter

Constructors in org.apache.lucene.analysis.snowball with parameters of type TokenStream

SnowballFilter(TokenStream input, SnowballProgram stemmer)

SnowballFilter(TokenStream in, String name)
Construct the named stemming filter.

Uses of TokenStream in org.apache.lucene.analysis.standard

Subclasses of TokenStream in org.apache.lucene.analysis.standard

class	StandardFilter Normalizes tokens extracted with StandardTokenizer.
class	StandardTokenizer A grammar-based tokenizer constructed with JFlex

Methods in org.apache.lucene.analysis.standard that return TokenStream

TokenStream	StandardAnalyzer.reusableTokenStream(String fieldName, Reader reader)
TokenStream	StandardAnalyzer.tokenStream(String fieldName, Reader reader) Constructs a StandardTokenizer filtered by a StandardFilter, a LowerCaseFilter and a StopFilter.

Constructors in org.apache.lucene.analysis.standard with parameters of type TokenStream

StandardFilter(TokenStream in)
Construct filtering in.

Uses of TokenStream in org.apache.lucene.analysis.th

Subclasses of TokenStream in org.apache.lucene.analysis.th

class

ThaiWordFilter TokenFilter that use BreakIterator to break each Token that is Thai into separate Token(s) for each Thai word.

Methods in org.apache.lucene.analysis.th that return TokenStream

TokenStream	ThaiAnalyzer.reusableTokenStream(String fieldName, Reader reader)
TokenStream	ThaiAnalyzer.tokenStream(String fieldName, Reader reader)

Constructors in org.apache.lucene.analysis.th with parameters of type TokenStream

ThaiWordFilter(TokenStream input)

Uses of TokenStream in org.apache.lucene.collation

Subclasses of TokenStream in org.apache.lucene.collation

class	CollationKeyFilter Converts each token into its CollationKey, and then encodes the CollationKey with IndexableBinaryStringTools, to allow it to be stored as an index term.
class	ICUCollationKeyFilter Converts each token into its CollationKey, and then encodes the CollationKey with IndexableBinaryStringTools, to allow it to be stored as an index term.

Methods in org.apache.lucene.collation that return TokenStream

TokenStream	ICUCollationKeyAnalyzer.reusableTokenStream(String fieldName, Reader reader)
TokenStream	CollationKeyAnalyzer.reusableTokenStream(String fieldName, Reader reader)
TokenStream	ICUCollationKeyAnalyzer.tokenStream(String fieldName, Reader reader)
TokenStream	CollationKeyAnalyzer.tokenStream(String fieldName, Reader reader)

Constructors in org.apache.lucene.collation with parameters of type TokenStream

CollationKeyFilter(TokenStream input, Collator collator)

ICUCollationKeyFilter(TokenStream input, com.ibm.icu.text.Collator collator)

Uses of TokenStream in org.apache.lucene.document

Fields in org.apache.lucene.document declared as TokenStream

protected TokenStream

AbstractField.tokenStream

Methods in org.apache.lucene.document that return TokenStream

TokenStream	NumericField.tokenStreamValue() Returns a NumericTokenStream for indexing the numeric value.
TokenStream	Field.tokenStreamValue() The TokesStream for this field to be used when indexing, or null.
TokenStream	Fieldable.tokenStreamValue() The TokenStream for this field to be used when indexing, or null.

Methods in org.apache.lucene.document with parameters of type TokenStream

void	Field.setTokenStream(TokenStream tokenStream) Expert: sets the token stream to be used for indexing and causes isIndexed() and isTokenized() to return true.

Constructors in org.apache.lucene.document with parameters of type TokenStream

Field(String name, TokenStream tokenStream)
Create a tokenized and indexed field that is not stored.

Field(String name, TokenStream tokenStream, Field.TermVector termVector)
Create a tokenized and indexed field that is not stored, optionally with storing term vectors.

Uses of TokenStream in org.apache.lucene.index.memory

Methods in org.apache.lucene.index.memory that return TokenStream

<T> TokenStream

MemoryIndex.keywordTokenStream(Collection<T> keywords) Convenience method; Creates and returns a token stream that generates a token for each keyword in the given collection, "as is", without any transforming text analysis.

Methods in org.apache.lucene.index.memory with parameters of type TokenStream

void	MemoryIndex.addField(String fieldName, TokenStream stream) Equivalent to addField(fieldName, stream, 1.0f).
void	MemoryIndex.addField(String fieldName, TokenStream stream, float boost) Iterates over the given token stream and adds the resulting terms to the index; Equivalent to adding a tokenized, indexed, termVectorStored, unstored, Lucene Field.

Uses of TokenStream in org.apache.lucene.search.highlight

Methods in org.apache.lucene.search.highlight that return TokenStream

static TokenStream	TokenSources.getAnyTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer) A convenience method that tries a number of approaches to getting a token stream.
static TokenStream	TokenSources.getAnyTokenStream(IndexReader reader, int docId, String field, Document doc, Analyzer analyzer) A convenience method that tries to first get a TermPositionVector for the specified docId, then, falls back to using the passed in Document to retrieve the TokenStream.
TokenStream	WeightedSpanTermExtractor.getTokenStream()
static TokenStream	TokenSources.getTokenStream(Document doc, String field, Analyzer analyzer)
static TokenStream	TokenSources.getTokenStream(IndexReader reader, int docId, String field)
static TokenStream	TokenSources.getTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer)
static TokenStream	TokenSources.getTokenStream(String field, String contents, Analyzer analyzer)
static TokenStream	TokenSources.getTokenStream(TermPositionVector tpv)
static TokenStream	TokenSources.getTokenStream(TermPositionVector tpv, boolean tokenPositionsGuaranteedContiguous) Low level api.
TokenStream	Scorer.init(TokenStream tokenStream) Called to init the Scorer with a TokenStream.
TokenStream	QueryScorer.init(TokenStream tokenStream)
TokenStream	QueryTermScorer.init(TokenStream tokenStream)

Methods in org.apache.lucene.search.highlight with parameters of type TokenStream

String	Highlighter.getBestFragment(TokenStream tokenStream, String text) Highlights chosen terms in a text, extracting the most relevant section.
String[]	Highlighter.getBestFragments(TokenStream tokenStream, String text, int maxNumFragments) Highlights chosen terms in a text, extracting the most relevant sections.
String	Highlighter.getBestFragments(TokenStream tokenStream, String text, int maxNumFragments, String separator) Highlights terms in the text , extracting the most relevant sections and concatenating the chosen fragments with a separator (typically "...").
TextFragment[]	Highlighter.getBestTextFragments(TokenStream tokenStream, String text, boolean mergeContiguousFragments, int maxNumFragments) Low level api to get the most relevant (formatted) sections of the document.
Map<String,WeightedSpanTerm>	WeightedSpanTermExtractor.getWeightedSpanTerms(Query query, TokenStream tokenStream) Creates a Map of WeightedSpanTerms from the given Query and TokenStream.
Map<String,WeightedSpanTerm>	WeightedSpanTermExtractor.getWeightedSpanTerms(Query query, TokenStream tokenStream, String fieldName) Creates a Map of WeightedSpanTerms from the given Query and TokenStream.
Map<String,WeightedSpanTerm>	WeightedSpanTermExtractor.getWeightedSpanTermsWithScores(Query query, TokenStream tokenStream, String fieldName, IndexReader reader) Creates a Map of WeightedSpanTerms from the given Query and TokenStream.
TokenStream	Scorer.init(TokenStream tokenStream) Called to init the Scorer with a TokenStream.
TokenStream	QueryScorer.init(TokenStream tokenStream)
TokenStream	QueryTermScorer.init(TokenStream tokenStream)
void	Fragmenter.start(String originalText, TokenStream tokenStream) Initializes the Fragmenter.
void	SimpleSpanFragmenter.start(String originalText, TokenStream tokenStream)
void	NullFragmenter.start(String s, TokenStream tokenStream)
void	SimpleFragmenter.start(String originalText, TokenStream stream)

Constructors in org.apache.lucene.search.highlight with parameters of type TokenStream

TokenGroup(TokenStream tokenStream)

Uses of TokenStream in org.apache.lucene.wikipedia.analysis

Subclasses of TokenStream in org.apache.lucene.wikipedia.analysis

class

WikipediaTokenizer Extension of StandardTokenizer that is aware of Wikipedia syntax.

Uses of TokenStream in org.apache.lucene.wordnet

Subclasses of TokenStream in org.apache.lucene.wordnet

class

SynonymTokenFilter Injects additional tokens for synonyms of token terms fetched from the underlying child stream; the child stream must deliver lowercase tokens for synonyms to be found.

Constructors in org.apache.lucene.wordnet with parameters of type TokenStream

SynonymTokenFilter(TokenStream input, SynonymMap synonyms, int maxSynonyms)
Creates an instance for the given underlying stream and synonym table.