Uses of Class
org.apache.lucene.analysis.Token

Packages that use Token

org.apache.lucene.analysis	API and code to convert text into indexable/searchable tokens.
org.apache.lucene.analysis.compound	A filter that decomposes compound words you find in many Germanic languages into the word parts.
org.apache.lucene.analysis.miscellaneous	Miscellaneous TokenStreams
org.apache.lucene.analysis.shingle	Word n-gram filters
org.apache.lucene.search.highlight	The highlight package contains classes to provide "keyword in context" features typically used to highlight search terms in the text of results pages.

Uses of Token in org.apache.lucene.analysis

Methods in org.apache.lucene.analysis that return Token

Token	Token.clone(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset) Makes a clone, but replaces the term buffer & start/end offset in the process.
Token	Token.reinit(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset) Shorthand for calling clear(), CharTermAttributeImpl.copyBuffer(char[], int, int), setStartOffset(int), setEndOffset(int) setType(java.lang.String) on Token.DEFAULT_TYPE
Token	Token.reinit(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset, String newType) Shorthand for calling clear(), CharTermAttributeImpl.copyBuffer(char[], int, int), setStartOffset(int), setEndOffset(int), setType(java.lang.String)
Token	Token.reinit(String newTerm, int newStartOffset, int newEndOffset) Shorthand for calling clear(), CharTermAttributeImpl.append(CharSequence), setStartOffset(int), setEndOffset(int) setType(java.lang.String) on Token.DEFAULT_TYPE
Token	Token.reinit(String newTerm, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset) Shorthand for calling clear(), CharTermAttributeImpl.append(CharSequence, int, int), setStartOffset(int), setEndOffset(int) setType(java.lang.String) on Token.DEFAULT_TYPE
Token	Token.reinit(String newTerm, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset, String newType) Shorthand for calling clear(), CharTermAttributeImpl.append(CharSequence, int, int), setStartOffset(int), setEndOffset(int) setType(java.lang.String)
Token	Token.reinit(String newTerm, int newStartOffset, int newEndOffset, String newType) Shorthand for calling clear(), CharTermAttributeImpl.append(CharSequence), setStartOffset(int), setEndOffset(int) setType(java.lang.String)

Methods in org.apache.lucene.analysis with parameters of type Token

void	Token.reinit(Token prototype) Copy the prototype token's fields into this one.
void	Token.reinit(Token prototype, char[] newTermBuffer, int offset, int length) Copy the prototype token's fields into this one, with a different term.
void	Token.reinit(Token prototype, String newTerm) Copy the prototype token's fields into this one, with a different term.

Uses of Token in org.apache.lucene.analysis.compound

Fields in org.apache.lucene.analysis.compound with type parameters of type Token

protected LinkedList<Token>

CompoundWordTokenFilterBase.tokens

Methods in org.apache.lucene.analysis.compound that return Token

protected Token

CompoundWordTokenFilterBase.createToken(int offset, int length, Token prototype)

Methods in org.apache.lucene.analysis.compound with parameters of type Token

protected Token	CompoundWordTokenFilterBase.createToken(int offset, int length, Token prototype)
protected void	CompoundWordTokenFilterBase.decompose(Token token)
protected void	HyphenationCompoundWordTokenFilter.decomposeInternal(Token token)
protected void	DictionaryCompoundWordTokenFilter.decomposeInternal(Token token)
protected abstract void	CompoundWordTokenFilterBase.decomposeInternal(Token token)

Uses of Token in org.apache.lucene.analysis.miscellaneous

Methods in org.apache.lucene.analysis.miscellaneous that return Token

Token	SingleTokenTokenStream.getToken()
Token	PrefixAndSuffixAwareTokenFilter.updateInputToken(Token inputToken, Token lastPrefixToken)
Token	PrefixAndSuffixAwareTokenFilter.updateSuffixToken(Token suffixToken, Token lastInputToken)
Token	PrefixAwareTokenFilter.updateSuffixToken(Token suffixToken, Token lastPrefixToken) The default implementation adds last prefix token end offset to the suffix token start and end offsets.

Methods in org.apache.lucene.analysis.miscellaneous with parameters of type Token

void	SingleTokenTokenStream.setToken(Token token)
Token	PrefixAndSuffixAwareTokenFilter.updateInputToken(Token inputToken, Token lastPrefixToken)
Token	PrefixAndSuffixAwareTokenFilter.updateSuffixToken(Token suffixToken, Token lastInputToken)
Token	PrefixAwareTokenFilter.updateSuffixToken(Token suffixToken, Token lastPrefixToken) The default implementation adds last prefix token end offset to the suffix token start and end offsets.

Constructors in org.apache.lucene.analysis.miscellaneous with parameters of type Token

SingleTokenTokenStream(Token token)

Uses of Token in org.apache.lucene.analysis.shingle

Methods in org.apache.lucene.analysis.shingle that return types with arguments of type Token

List<Token>

ShingleMatrixFilter.Matrix.Column.Row.getTokens()

Methods in org.apache.lucene.analysis.shingle with parameters of type Token

float	ShingleMatrixFilter.calculateShingleWeight(Token shingleToken, List<Token> shingle, int currentPermutationStartOffset, List<ShingleMatrixFilter.Matrix.Column.Row> currentPermutationRows, List<Token> currentPermuationTokens) Deprecated. Evaluates the new shingle token weight.
abstract ShingleMatrixFilter.TokenPositioner	ShingleMatrixFilter.TokenSettingsCodec.getTokenPositioner(Token token) Retrieves information on how a Token is to be inserted to a ShingleMatrixFilter.Matrix.
ShingleMatrixFilter.TokenPositioner	ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec.getTokenPositioner(Token token)
ShingleMatrixFilter.TokenPositioner	ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec.getTokenPositioner(Token token)
ShingleMatrixFilter.TokenPositioner	ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec.getTokenPositioner(Token token)
abstract float	ShingleMatrixFilter.TokenSettingsCodec.getWeight(Token token) Have this method return 1f in order to 'disable' weights.
float	ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec.getWeight(Token token)
float	ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec.getWeight(Token token)
float	ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec.getWeight(Token token) Returns a 32 bit float from the payload, or 1f it null.
abstract void	ShingleMatrixFilter.TokenSettingsCodec.setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner) Sets information on how a Token is to be inserted to a ShingleMatrixFilter.Matrix.
void	ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec.setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner)
void	ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec.setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner)
void	ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec.setTokenPositioner(Token token, ShingleMatrixFilter.TokenPositioner tokenPositioner) Sets the TokenPositioner as token flags int value.
abstract void	ShingleMatrixFilter.TokenSettingsCodec.setWeight(Token token, float weight) Have this method do nothing in order to 'disable' weights.
void	ShingleMatrixFilter.OneDimensionalNonWeightedTokenSettingsCodec.setWeight(Token token, float weight)
void	ShingleMatrixFilter.TwoDimensionalNonWeightedSynonymTokenSettingsCodec.setWeight(Token token, float weight)
void	ShingleMatrixFilter.SimpleThreeDimensionalTokenSettingsCodec.setWeight(Token token, float weight) Stores a 32 bit float in the payload, or set it to null if 1f;
void	ShingleMatrixFilter.updateToken(Token token, List<Token> shingle, int currentPermutationStartOffset, List<ShingleMatrixFilter.Matrix.Column.Row> currentPermutationRows, List<Token> currentPermuationTokens) Deprecated. Final touch of a shingle token before it is passed on to the consumer from method ShingleMatrixFilter.incrementToken().

Method parameters in org.apache.lucene.analysis.shingle with type arguments of type Token

float	ShingleMatrixFilter.calculateShingleWeight(Token shingleToken, List<Token> shingle, int currentPermutationStartOffset, List<ShingleMatrixFilter.Matrix.Column.Row> currentPermutationRows, List<Token> currentPermuationTokens) Deprecated. Evaluates the new shingle token weight.
float	ShingleMatrixFilter.calculateShingleWeight(Token shingleToken, List<Token> shingle, int currentPermutationStartOffset, List<ShingleMatrixFilter.Matrix.Column.Row> currentPermutationRows, List<Token> currentPermuationTokens) Deprecated. Evaluates the new shingle token weight.
void	ShingleMatrixFilter.Matrix.Column.Row.setTokens(List<Token> tokens)
void	ShingleMatrixFilter.updateToken(Token token, List<Token> shingle, int currentPermutationStartOffset, List<ShingleMatrixFilter.Matrix.Column.Row> currentPermutationRows, List<Token> currentPermuationTokens) Deprecated. Final touch of a shingle token before it is passed on to the consumer from method ShingleMatrixFilter.incrementToken().
void	ShingleMatrixFilter.updateToken(Token token, List<Token> shingle, int currentPermutationStartOffset, List<ShingleMatrixFilter.Matrix.Column.Row> currentPermutationRows, List<Token> currentPermuationTokens) Deprecated. Final touch of a shingle token before it is passed on to the consumer from method ShingleMatrixFilter.incrementToken().

Constructors in org.apache.lucene.analysis.shingle with parameters of type Token

ShingleMatrixFilter.Matrix.Column(Token token)

Uses of Token in org.apache.lucene.search.highlight

Methods in org.apache.lucene.search.highlight that return Token

Token

TokenGroup.getToken(int index)