Package org.apache.lucene.tests.analysis
Class Token
-
- All Implemented Interfaces:
Appendable
,CharSequence
,Cloneable
,CharTermAttribute
,FlagsAttribute
,OffsetAttribute
,PayloadAttribute
,PositionIncrementAttribute
,PositionLengthAttribute
,TermFrequencyAttribute
,TermToBytesRefAttribute
,TypeAttribute
,Attribute
public class Token extends PackedTokenAttributeImpl implements FlagsAttribute, PayloadAttribute
A Token is an occurrence of a term from the text of a field. It consists of a term's text, the start and end offset of the term in the text of the field, and a type string.The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC display, etc.
The type is a string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".
A Token can optionally have metadata (a.k.a. payload) in the form of a variable length byte array. Use
PostingsEnum.getPayload()
to retrieve the payloads from the index.A few things to note:
- clear() initializes all of the fields to default values. This was changed in contrast to Lucene 2.4, but should affect no one.
- Because
TokenStreams
can be chained, one cannot assume that theToken's
current type is correct. - The startOffset and endOffset represent the start and offset in the source text, so be careful in adjusting them.
- When caching a reusable token, clone it. When injecting a cached token into a stream that can be reset, clone it again.
-
-
Field Summary
Fields Modifier and Type Field Description static AttributeFactory
TOKEN_ATTRIBUTE_FACTORY
Convenience factory that returnsToken
as implementation for the basic attributes and return the default impl (with "Impl" appended) for all other attributes.-
Fields inherited from class org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl
builder
-
Fields inherited from interface org.apache.lucene.analysis.tokenattributes.TypeAttribute
DEFAULT_TYPE
-
-
Constructor Summary
Constructors Constructor Description Token()
Constructs a Token will null text.Token(CharSequence text, int start, int end)
Constructs a Token with the given term text, start and end offsets.Token(CharSequence text, int posInc, int start, int end)
Constructs a Token with the given term text, position increment, start and end offsetsToken(CharSequence text, int posInc, int start, int end, int posLength)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
clear()
Resets the term text, payload, flags, positionIncrement, positionLength, startOffset, endOffset and token type to default.Token
clone()
void
copyTo(AttributeImpl target)
boolean
equals(Object obj)
int
getFlags()
BytesRef
getPayload()
int
hashCode()
void
reflectWith(AttributeReflector reflector)
void
reinit(Token prototype)
Copy the prototype token's fields into this one.void
setFlags(int flags)
void
setPayload(BytesRef payload)
-
Methods inherited from class org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl
end, endOffset, getPositionIncrement, getPositionLength, getTermFrequency, setOffset, setPositionIncrement, setPositionLength, setTermFrequency, setType, startOffset, type
-
Methods inherited from class org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl
append, append, append, append, append, append, buffer, charAt, copyBuffer, getBytesRef, length, resizeBuffer, setEmpty, setLength, subSequence, toString
-
Methods inherited from class org.apache.lucene.util.AttributeImpl
reflectAsString
-
Methods inherited from class java.lang.Object
finalize, getClass, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface java.lang.CharSequence
chars, codePoints
-
-
-
-
Field Detail
-
TOKEN_ATTRIBUTE_FACTORY
public static final AttributeFactory TOKEN_ATTRIBUTE_FACTORY
Convenience factory that returnsToken
as implementation for the basic attributes and return the default impl (with "Impl" appended) for all other attributes.- Since:
- 3.0
-
-
Constructor Detail
-
Token
public Token()
Constructs a Token will null text.
-
Token
public Token(CharSequence text, int start, int end)
Constructs a Token with the given term text, start and end offsets. The type defaults to "word." NOTE: for better indexing speed you should instead use the char[] termBuffer methods to set the term text.- Parameters:
text
- term textstart
- start offset in the source textend
- end offset in the source text
-
Token
public Token(CharSequence text, int posInc, int start, int end)
Constructs a Token with the given term text, position increment, start and end offsets
-
Token
public Token(CharSequence text, int posInc, int start, int end, int posLength)
-
-
Method Detail
-
getFlags
public int getFlags()
- Specified by:
getFlags
in interfaceFlagsAttribute
- See Also:
FlagsAttribute
-
setFlags
public void setFlags(int flags)
- Specified by:
setFlags
in interfaceFlagsAttribute
- See Also:
FlagsAttribute
-
getPayload
public BytesRef getPayload()
- Specified by:
getPayload
in interfacePayloadAttribute
- See Also:
PayloadAttribute
-
setPayload
public void setPayload(BytesRef payload)
- Specified by:
setPayload
in interfacePayloadAttribute
- See Also:
PayloadAttribute
-
clear
public void clear()
Resets the term text, payload, flags, positionIncrement, positionLength, startOffset, endOffset and token type to default.- Overrides:
clear
in classPackedTokenAttributeImpl
-
equals
public boolean equals(Object obj)
- Overrides:
equals
in classPackedTokenAttributeImpl
-
hashCode
public int hashCode()
- Overrides:
hashCode
in classPackedTokenAttributeImpl
-
clone
public Token clone()
- Overrides:
clone
in classPackedTokenAttributeImpl
-
reinit
public void reinit(Token prototype)
Copy the prototype token's fields into this one. Note: Payloads are shared.- Parameters:
prototype
- source Token to copy fields from
-
copyTo
public void copyTo(AttributeImpl target)
- Overrides:
copyTo
in classPackedTokenAttributeImpl
-
reflectWith
public void reflectWith(AttributeReflector reflector)
- Overrides:
reflectWith
in classPackedTokenAttributeImpl
-
-