@Deprecated public class Token extends PackedTokenAttributeImpl implements FlagsAttribute, PayloadAttribute
The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC display, etc.
The type is a string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".
A Token can optionally have metadata (a.k.a. payload) in the form of a variable
length byte array. Use PostingsEnum.getPayload() to retrieve the
payloads from the index.
NOTE: As of 2.9, Token implements all Attribute interfaces
that are part of core Lucene and can be found in the tokenattributes subpackage.
Even though it is not necessary to use Token anymore, with the new TokenStream API it can
be used as convenience class that implements all Attributes, which is especially useful
to easily switch from the old to the new TokenStream API.
A few things to note:
TokenStreams can be chained, one cannot assume that the Token's current type is correct.
Please note: With Lucene 3.1, the toString() method had to be changed to match the
CharSequence interface introduced by the interface CharTermAttribute.
This method now only prints the term text, no additional information anymore.
| Modifier and Type | Field and Description |
|---|---|
static AttributeFactory |
TOKEN_ATTRIBUTE_FACTORY
Deprecated.
Convenience factory that returns
Token as implementation for the basic
attributes and return the default impl (with "Impl" appended) for all other
attributes. |
builderDEFAULT_TYPE| Constructor and Description |
|---|
Token()
Deprecated.
Constructs a Token will null text.
|
Token(CharSequence text,
int start,
int end)
Deprecated.
Constructs a Token with the given term text, start
and end offsets.
|
| Modifier and Type | Method and Description |
|---|---|
void |
clear()
Deprecated.
Resets the term text, payload, flags, positionIncrement, positionLength,
startOffset, endOffset and token type to default.
|
Token |
clone()
Deprecated.
In most cases the clone is, and should be, deep in order to be able to
properly capture the state of all attributes.
|
void |
copyTo(AttributeImpl target)
Deprecated.
Copies the values from this Attribute into the passed-in
target attribute.
|
boolean |
equals(Object obj)
Deprecated.
|
int |
getFlags()
Deprecated.
Get the bitset for any bits that have been set.
|
BytesRef |
getPayload()
Deprecated.
Returns this Token's payload.
|
int |
hashCode()
Deprecated.
|
void |
reflectWith(AttributeReflector reflector)
Deprecated.
This method is for introspection of attributes, it should simply
add the key/values this attribute holds to the given
AttributeReflector. |
void |
reinit(Token prototype)
Deprecated.
Copy the prototype token's fields into this one.
|
void |
setFlags(int flags)
Deprecated.
Set the flags to a new bitset.
|
void |
setPayload(BytesRef payload)
Deprecated.
Sets this Token's payload.
|
end, endOffset, getPositionIncrement, getPositionLength, setOffset, setPositionIncrement, setPositionLength, setType, startOffset, typeappend, append, append, append, append, append, buffer, charAt, copyBuffer, getBytesRef, length, resizeBuffer, setEmpty, setLength, subSequence, toStringreflectAsStringfinalize, getClass, notify, notifyAll, wait, wait, waitchars, codePointspublic static final AttributeFactory TOKEN_ATTRIBUTE_FACTORY
Token as implementation for the basic
attributes and return the default impl (with "Impl" appended) for all other
attributes.public Token()
public Token(CharSequence text, int start, int end)
text - term textstart - start offset in the source textend - end offset in the source textpublic int getFlags()
getFlags in interface FlagsAttributeFlagsAttributepublic void setFlags(int flags)
setFlags in interface FlagsAttributeFlagsAttributepublic BytesRef getPayload()
getPayload in interface PayloadAttributePayloadAttributepublic void setPayload(BytesRef payload)
setPayload in interface PayloadAttributePayloadAttributepublic void clear()
clear in class PackedTokenAttributeImplpublic boolean equals(Object obj)
equals in class PackedTokenAttributeImplpublic int hashCode()
hashCode in class PackedTokenAttributeImplpublic Token clone()
AttributeImplclone in class PackedTokenAttributeImplpublic void reinit(Token prototype)
prototype - source Token to copy fields frompublic void copyTo(AttributeImpl target)
AttributeImplcopyTo in class PackedTokenAttributeImplpublic void reflectWith(AttributeReflector reflector)
AttributeImplAttributeReflector.
Implementations look like this (e.g. for a combined attribute implementation):
public void reflectWith(AttributeReflector reflector) {
reflector.reflect(CharTermAttribute.class, "term", term());
reflector.reflect(PositionIncrementAttribute.class, "positionIncrement", getPositionIncrement());
}
If you implement this method, make sure that for each invocation, the same set of Attribute
interfaces and keys are passed to AttributeReflector.reflect(java.lang.Class<? extends org.apache.lucene.util.Attribute>, java.lang.String, java.lang.Object) in the same order, but possibly
different values. So don't automatically exclude e.g. null properties!
reflectWith in class PackedTokenAttributeImplAttributeImpl.reflectAsString(boolean)Copyright © 2000-2017 Apache Software Foundation. All Rights Reserved.