Package org.apache.lucene.analysis

Class Token

java.lang.Object

org.apache.lucene.util.AttributeImpl

org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl

org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl

org.apache.lucene.analysis.Token

All Implemented Interfaces:

Appendable, CharSequence, Cloneable, CharTermAttribute, FlagsAttribute, OffsetAttribute, PayloadAttribute, PositionIncrementAttribute, PositionLengthAttribute, TermFrequencyAttribute, TermToBytesRefAttribute, TypeAttribute, Attribute

public class Token
extends PackedTokenAttributeImpl
implements FlagsAttribute, PayloadAttribute

A Token is an occurrence of a term from the text of a field. It consists of a term's text, the start and end offset of the term in the text of the field, and a type string.

The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC display, etc.

The type is a string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".

A Token can optionally have metadata (a.k.a. payload) in the form of a variable length byte array. Use PostingsEnum.getPayload() to retrieve the payloads from the index. A few things to note:

• clear() initializes all of the fields to default values. This was changed in contrast to Lucene 2.4, but should affect no one.
• Because TokenStreams can be chained, one cannot assume that the Token's current type is correct.
• The startOffset and endOffset represent the start and offset in the source text, so be careful in adjusting them.
• When caching a reusable token, clone it. When injecting a cached token into a stream that can be reset, clone it again.

Field Summary

Fields

Modifier and Type	Field	Description
static AttributeFactory	TOKEN_ATTRIBUTE_FACTORY	Convenience factory that returns Token as implementation for the basic attributes and return the default impl (with "Impl" appended) for all other attributes.

Fields inherited from class org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl

builder

Fields inherited from interface org.apache.lucene.analysis.tokenattributes.TypeAttribute

DEFAULT_TYPE

Constructor Summary

Constructors

Constructor	Description
Token()	Constructs a Token will null text.
Token(CharSequence text, int start, int end)	Constructs a Token with the given term text, start and end offsets.
Token(CharSequence text, int posInc, int start, int end)	Constructs a Token with the given term text, position increment, start and end offsets

Method Summary

Modifier and Type	Method	Description
void	clear()	Resets the term text, payload, flags, positionIncrement, positionLength, startOffset, endOffset and token type to default.
Token	clone()
void	copyTo(AttributeImpl target)
boolean	equals(Object obj)
int	getFlags()
BytesRef	getPayload()
int	hashCode()
void	reflectWith(AttributeReflector reflector)
void	reinit(Token prototype)	Copy the prototype token's fields into this one.
void	setFlags(int flags)
void	setPayload(BytesRef payload)

Methods inherited from class org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl

end, endOffset, getPositionIncrement, getPositionLength, getTermFrequency, setOffset, setPositionIncrement, setPositionLength, setTermFrequency, setType, startOffset, type

Methods inherited from class org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl

append, append, append, append, append, append, buffer, charAt, copyBuffer, getBytesRef, length, resizeBuffer, setEmpty, setLength, subSequence, toString

Methods inherited from class org.apache.lucene.util.AttributeImpl

reflectAsString

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.CharSequence

chars, codePoints

Field Detail

TOKEN_ATTRIBUTE_FACTORY

public static final AttributeFactory TOKEN_ATTRIBUTE_FACTORY

Convenience factory that returns Token as implementation for the basic attributes and return the default impl (with "Impl" appended) for all other attributes.

Since:

3.0

Constructor Detail

Token

public Token()

Constructs a Token will null text.

Token

public Token(CharSequence text,
             int start,
             int end)

Constructs a Token with the given term text, start and end offsets. The type defaults to "word." NOTE: for better indexing speed you should instead use the char[] termBuffer methods to set the term text.

Parameters:

text - term text

start - start offset in the source text

end - end offset in the source text

Token

public Token(CharSequence text,
             int posInc,
             int start,
             int end)

Constructs a Token with the given term text, position increment, start and end offsets

Method Detail

getFlags

public int getFlags()

Specified by:

getFlags in interface FlagsAttribute

See Also:

FlagsAttribute

setFlags

public void setFlags(int flags)

Specified by:

setFlags in interface FlagsAttribute

See Also:

FlagsAttribute

getPayload

public BytesRef getPayload()

Specified by:

getPayload in interface PayloadAttribute

See Also:

PayloadAttribute

setPayload

public void setPayload(BytesRef payload)

Specified by:

setPayload in interface PayloadAttribute

See Also:

PayloadAttribute

clear

public void clear()

Resets the term text, payload, flags, positionIncrement, positionLength, startOffset, endOffset and token type to default.

Overrides:

clear in class PackedTokenAttributeImpl

equals

public boolean equals(Object obj)

Overrides:

equals in class PackedTokenAttributeImpl

hashCode

public int hashCode()

Overrides:

hashCode in class PackedTokenAttributeImpl

clone

public Token clone()

Overrides:

clone in class PackedTokenAttributeImpl

reinit

public void reinit(Token prototype)

Copy the prototype token's fields into this one. Note: Payloads are shared.

Parameters:

prototype - source Token to copy fields from

copyTo

public void copyTo(AttributeImpl target)

Overrides:

copyTo in class PackedTokenAttributeImpl

reflectWith

public void reflectWith(AttributeReflector reflector)

Overrides:

reflectWith in class PackedTokenAttributeImpl