@Deprecated public class Token extends PackedTokenAttributeImpl implements FlagsAttribute, PayloadAttribute
The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC display, etc.
The type is a string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".
A Token can optionally have metadata (a.k.a. payload) in the form of a variable
length byte array. Use PostingsEnum.getPayload()
to retrieve the
payloads from the index.
NOTE: As of 2.9, Token implements all Attribute
interfaces
that are part of core Lucene and can be found in the tokenattributes
subpackage.
Even though it is not necessary to use Token anymore, with the new TokenStream API it can
be used as convenience class that implements all Attribute
s, which is especially useful
to easily switch from the old to the new TokenStream API.
A few things to note:
TokenStreams
can be chained, one cannot assume that the Token's
current type is correct.
Please note: With Lucene 3.1, the toString()
method had to be changed to match the
CharSequence
interface introduced by the interface CharTermAttribute
.
This method now only prints the term text, no additional information anymore.
Modifier and Type | Field and Description |
---|---|
static AttributeFactory |
TOKEN_ATTRIBUTE_FACTORY
Deprecated.
Convenience factory that returns
Token as implementation for the basic
attributes and return the default impl (with "Impl" appended) for all other
attributes. |
builder
DEFAULT_TYPE
Constructor and Description |
---|
Token()
Deprecated.
Constructs a Token will null text.
|
Token(CharSequence text,
int start,
int end)
Deprecated.
Constructs a Token with the given term text, start
and end offsets.
|
Modifier and Type | Method and Description |
---|---|
void |
clear()
Deprecated.
Resets the term text, payload, flags, positionIncrement, positionLength,
startOffset, endOffset and token type to default.
|
Token |
clone()
Deprecated.
In most cases the clone is, and should be, deep in order to be able to
properly capture the state of all attributes.
|
void |
copyTo(AttributeImpl target)
Deprecated.
Copies the values from this Attribute into the passed-in
target attribute.
|
boolean |
equals(Object obj)
Deprecated.
|
int |
getFlags()
Deprecated.
Get the bitset for any bits that have been set.
|
BytesRef |
getPayload()
Deprecated.
Returns this Token's payload.
|
int |
hashCode()
Deprecated.
|
void |
reflectWith(AttributeReflector reflector)
Deprecated.
This method is for introspection of attributes, it should simply
add the key/values this attribute holds to the given
AttributeReflector . |
void |
reinit(Token prototype)
Deprecated.
Copy the prototype token's fields into this one.
|
void |
setFlags(int flags)
Deprecated.
Set the flags to a new bitset.
|
void |
setPayload(BytesRef payload)
Deprecated.
Sets this Token's payload.
|
endOffset, getPositionIncrement, getPositionLength, setOffset, setPositionIncrement, setPositionLength, setType, startOffset, type
append, append, append, append, append, append, buffer, charAt, copyBuffer, getBytesRef, length, resizeBuffer, setEmpty, setLength, subSequence, toString
reflectAsString
finalize, getClass, notify, notifyAll, wait, wait, wait
chars, codePoints
public static final AttributeFactory TOKEN_ATTRIBUTE_FACTORY
Token
as implementation for the basic
attributes and return the default impl (with "Impl" appended) for all other
attributes.public Token()
public Token(CharSequence text, int start, int end)
text
- term textstart
- start offset in the source textend
- end offset in the source textpublic int getFlags()
getFlags
in interface FlagsAttribute
FlagsAttribute
public void setFlags(int flags)
setFlags
in interface FlagsAttribute
FlagsAttribute
public BytesRef getPayload()
getPayload
in interface PayloadAttribute
PayloadAttribute
public void setPayload(BytesRef payload)
setPayload
in interface PayloadAttribute
PayloadAttribute
public void clear()
clear
in class PackedTokenAttributeImpl
public boolean equals(Object obj)
equals
in class PackedTokenAttributeImpl
public int hashCode()
hashCode
in class PackedTokenAttributeImpl
public Token clone()
AttributeImpl
clone
in class PackedTokenAttributeImpl
public void reinit(Token prototype)
prototype
- source Token to copy fields frompublic void copyTo(AttributeImpl target)
AttributeImpl
copyTo
in class PackedTokenAttributeImpl
public void reflectWith(AttributeReflector reflector)
AttributeImpl
AttributeReflector
.
Implementations look like this (e.g. for a combined attribute implementation):
public void reflectWith(AttributeReflector reflector) { reflector.reflect(CharTermAttribute.class, "term", term()); reflector.reflect(PositionIncrementAttribute.class, "positionIncrement", getPositionIncrement()); }
If you implement this method, make sure that for each invocation, the same set of Attribute
interfaces and keys are passed to AttributeReflector.reflect(java.lang.Class<? extends org.apache.lucene.util.Attribute>, java.lang.String, java.lang.Object)
in the same order, but possibly
different values. So don't automatically exclude e.g. null
properties!
reflectWith
in class PackedTokenAttributeImpl
AttributeImpl.reflectAsString(boolean)
Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.