public class Token extends TermAttributeImpl implements TypeAttribute, PositionIncrementAttribute, FlagsAttribute, OffsetAttribute, PayloadAttribute, PositionLengthAttribute
The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC display, etc.
The type is a string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".
A Token can optionally have metadata (a.k.a. Payload) in the form of a variable
length byte array. Use TermPositions.getPayloadLength()
and
TermPositions.getPayload(byte[], int)
to retrieve the payloads from the index.
NOTE: As of 2.9, Token implements all Attribute
interfaces
that are part of core Lucene and can be found in the tokenattributes
subpackage.
Even though it is not necessary to use Token anymore, with the new TokenStream API it can
be used as convenience class that implements all Attribute
s, which is especially useful
to easily switch from the old to the new TokenStream API.
Tokenizers and TokenFilters should try to re-use a Token
instance when possible for best performance, by
implementing the TokenStream.incrementToken()
API.
Failing that, to create a new Token you should first use
one of the constructors that starts with null text. To load
the token from a char[] use CharTermAttributeImpl.copyBuffer(char[], int, int)
.
To load from a String use CharTermAttributeImpl.setEmpty()
followed by CharTermAttributeImpl.append(CharSequence)
or CharTermAttributeImpl.append(CharSequence, int, int)
.
Alternatively you can get the Token's termBuffer by calling either CharTermAttributeImpl.buffer()
,
if you know that your text is shorter than the capacity of the termBuffer
or CharTermAttributeImpl.resizeBuffer(int)
, if there is any possibility
that you may need to grow the buffer. Fill in the characters of your term into this
buffer, with String.getChars(int, int, char[], int)
if loading from a string,
or with System.arraycopy(Object, int, Object, int, int)
, and finally call CharTermAttributeImpl.setLength(int)
to
set the length of the term text. See LUCENE-969
for details.
Typical Token reuse patterns:
TypeAttribute.DEFAULT_TYPE
if not specified):return reusableToken.reinit(string, startOffset, endOffset[, type]);
TypeAttribute.DEFAULT_TYPE
if not specified):return reusableToken.reinit(string, 0, string.length(), startOffset, endOffset[, type]);
TypeAttribute.DEFAULT_TYPE
if not specified):return reusableToken.reinit(buffer, 0, buffer.length, startOffset, endOffset[, type]);
TypeAttribute.DEFAULT_TYPE
if not specified):return reusableToken.reinit(buffer, start, end - start, startOffset, endOffset[, type]);
TypeAttribute.DEFAULT_TYPE
if not specified):return reusableToken.reinit(source.buffer(), 0, source.length(), source.startOffset(), source.endOffset()[, source.type()]);
TokenStreams
can be chained, one cannot assume that the Token's
current type is correct.
Please note: With Lucene 3.1, the toString()
method had to be changed to match the
CharSequence
interface introduced by the interface CharTermAttribute
.
This method now only prints the term text, no additional information anymore.
Payload
,
Serialized FormModifier and Type | Class and Description |
---|---|
static class |
Token.TokenAttributeFactory
Expert: Creates a TokenAttributeFactory returning
Token as instance for the basic attributes
and for all other attributes calls the given delegate factory. |
Modifier and Type | Field and Description |
---|---|
static AttributeSource.AttributeFactory |
TOKEN_ATTRIBUTE_FACTORY
Convenience factory that returns
Token as implementation for the basic
attributes and return the default impl (with "Impl" appended) for all other
attributes. |
enableBackwards
DEFAULT_TYPE
Constructor and Description |
---|
Token()
Constructs a Token will null text.
|
Token(char[] startTermBuffer,
int termBufferOffset,
int termBufferLength,
int start,
int end)
Constructs a Token with the given term buffer (offset
& length), start and end
offsets
|
Token(int start,
int end)
Constructs a Token with null text and start & end
offsets.
|
Token(int start,
int end,
int flags)
Constructs a Token with null text and start & end
offsets plus flags.
|
Token(int start,
int end,
String typ)
Constructs a Token with null text and start & end
offsets plus the Token type.
|
Token(String text,
int start,
int end)
Constructs a Token with the given term text, and start
& end offsets.
|
Token(String text,
int start,
int end,
int flags)
Constructs a Token with the given text, start and end
offsets, & type.
|
Token(String text,
int start,
int end,
String typ)
Constructs a Token with the given text, start and end
offsets, & type.
|
Modifier and Type | Method and Description |
---|---|
void |
clear()
Resets the term text, payload, flags, and positionIncrement,
startOffset, endOffset and token type to default.
|
Object |
clone()
Shallow clone.
|
Token |
clone(char[] newTermBuffer,
int newTermOffset,
int newTermLength,
int newStartOffset,
int newEndOffset)
Makes a clone, but replaces the term buffer &
start/end offset in the process.
|
void |
copyTo(AttributeImpl target)
Copies the values from this Attribute into the passed-in
target attribute.
|
int |
endOffset()
Returns this Token's ending offset, one greater than the position of the
last character corresponding to this token in the source text.
|
boolean |
equals(Object obj) |
int |
getFlags()
Get the bitset for any bits that have been set.
|
Payload |
getPayload()
Returns this Token's payload.
|
int |
getPositionIncrement()
Returns the position increment of this Token.
|
int |
getPositionLength()
Get the position length.
|
int |
hashCode() |
void |
reflectWith(AttributeReflector reflector)
This method is for introspection of attributes, it should simply
add the key/values this attribute holds to the given
AttributeReflector . |
Token |
reinit(char[] newTermBuffer,
int newTermOffset,
int newTermLength,
int newStartOffset,
int newEndOffset)
Shorthand for calling
clear() ,
CharTermAttributeImpl.copyBuffer(char[], int, int) ,
setStartOffset(int) ,
setEndOffset(int)
setType(java.lang.String) on Token.DEFAULT_TYPE |
Token |
reinit(char[] newTermBuffer,
int newTermOffset,
int newTermLength,
int newStartOffset,
int newEndOffset,
String newType)
Shorthand for calling
clear() ,
CharTermAttributeImpl.copyBuffer(char[], int, int) ,
setStartOffset(int) ,
setEndOffset(int) ,
setType(java.lang.String) |
Token |
reinit(String newTerm,
int newStartOffset,
int newEndOffset)
Shorthand for calling
clear() ,
CharTermAttributeImpl.append(CharSequence) ,
setStartOffset(int) ,
setEndOffset(int)
setType(java.lang.String) on Token.DEFAULT_TYPE |
Token |
reinit(String newTerm,
int newTermOffset,
int newTermLength,
int newStartOffset,
int newEndOffset)
Shorthand for calling
clear() ,
CharTermAttributeImpl.append(CharSequence, int, int) ,
setStartOffset(int) ,
setEndOffset(int)
setType(java.lang.String) on Token.DEFAULT_TYPE |
Token |
reinit(String newTerm,
int newTermOffset,
int newTermLength,
int newStartOffset,
int newEndOffset,
String newType)
|
Token |
reinit(String newTerm,
int newStartOffset,
int newEndOffset,
String newType)
Shorthand for calling
clear() ,
CharTermAttributeImpl.append(CharSequence) ,
setStartOffset(int) ,
setEndOffset(int)
setType(java.lang.String) |
void |
reinit(Token prototype)
Copy the prototype token's fields into this one.
|
void |
reinit(Token prototype,
char[] newTermBuffer,
int offset,
int length)
Copy the prototype token's fields into this one, with a different term.
|
void |
reinit(Token prototype,
String newTerm)
Copy the prototype token's fields into this one, with a different term.
|
void |
setEndOffset(int offset)
Set the ending offset.
|
void |
setFlags(int flags) |
void |
setOffset(int startOffset,
int endOffset)
Set the starting and ending offset.
|
void |
setPayload(Payload payload)
Sets this Token's payload.
|
void |
setPositionIncrement(int positionIncrement)
Set the position increment.
|
void |
setPositionLength(int positionLength)
Set the position length.
|
void |
setStartOffset(int offset)
Set the starting offset.
|
void |
setType(String type)
Set the lexical type.
|
int |
startOffset()
Returns this Token's starting offset, the position of the first character
corresponding to this token in the source text.
|
String |
type()
Returns this Token's lexical type.
|
append, append, append, append, append, append, buffer, charAt, copyBuffer, length, resizeBuffer, resizeTermBuffer, setEmpty, setLength, setTermBuffer, setTermBuffer, setTermBuffer, setTermLength, subSequence, term, termBuffer, termLength, toString
reflectAsString
public static final AttributeSource.AttributeFactory TOKEN_ATTRIBUTE_FACTORY
Token
as implementation for the basic
attributes and return the default impl (with "Impl" appended) for all other
attributes.public Token()
public Token(int start, int end)
start
- start offset in the source textend
- end offset in the source textpublic Token(int start, int end, String typ)
start
- start offset in the source textend
- end offset in the source texttyp
- the lexical type of this Tokenpublic Token(int start, int end, int flags)
start
- start offset in the source textend
- end offset in the source textflags
- The bits to set for this tokenpublic Token(String text, int start, int end)
text
- term textstart
- start offsetend
- end offsetpublic Token(String text, int start, int end, String typ)
text
- term textstart
- start offsetend
- end offsettyp
- token typepublic Token(String text, int start, int end, int flags)
text
- start
- end
- flags
- token type bitspublic Token(char[] startTermBuffer, int termBufferOffset, int termBufferLength, int start, int end)
startTermBuffer
- termBufferOffset
- termBufferLength
- start
- end
- public void setPositionIncrement(int positionIncrement)
TokenStream
, used in phrase
searching.
The default value is one.
Some common uses for this are:
setPositionIncrement
in interface PositionIncrementAttribute
positionIncrement
- the distance from the prior termTermPositions
public int getPositionIncrement()
getPositionIncrement
in interface PositionIncrementAttribute
setPositionIncrement(int)
public void setPositionLength(int positionLength)
setPositionLength
in interface PositionLengthAttribute
positionLength
- how many positions this token
spans.PositionLengthAttribute
public int getPositionLength()
getPositionLength
in interface PositionLengthAttribute
PositionLengthAttribute
public final int startOffset()
CharTermAttributeImpl.length()
, as the term text may have been altered by a
stemmer or some other filter.startOffset
in interface OffsetAttribute
public void setStartOffset(int offset)
startOffset()
public final int endOffset()
endOffset
in interface OffsetAttribute
public void setEndOffset(int offset)
endOffset()
public void setOffset(int startOffset, int endOffset)
setOffset
in interface OffsetAttribute
and #endOffset()
public final String type()
type
in interface TypeAttribute
public final void setType(String type)
setType
in interface TypeAttribute
type()
public int getFlags()
type()
, although they do share similar purposes.
The flags can be used to encode information about the token for use by other TokenFilter
s.getFlags
in interface FlagsAttribute
public void setFlags(int flags)
setFlags
in interface FlagsAttribute
getFlags()
public Payload getPayload()
getPayload
in interface PayloadAttribute
public void setPayload(Payload payload)
setPayload
in interface PayloadAttribute
public void clear()
clear
in class CharTermAttributeImpl
public Object clone()
AttributeImpl
clone
in class CharTermAttributeImpl
public Token clone(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset)
CharTermAttributeImpl.copyBuffer(char[], int, int)
) because it saves a wasted copy of the old
termBuffer.public boolean equals(Object obj)
equals
in class CharTermAttributeImpl
public int hashCode()
hashCode
in class CharTermAttributeImpl
public Token reinit(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset, String newType)
clear()
,
CharTermAttributeImpl.copyBuffer(char[], int, int)
,
setStartOffset(int)
,
setEndOffset(int)
,
setType(java.lang.String)
public Token reinit(char[] newTermBuffer, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset)
clear()
,
CharTermAttributeImpl.copyBuffer(char[], int, int)
,
setStartOffset(int)
,
setEndOffset(int)
setType(java.lang.String)
on Token.DEFAULT_TYPEpublic Token reinit(String newTerm, int newStartOffset, int newEndOffset, String newType)
clear()
,
CharTermAttributeImpl.append(CharSequence)
,
setStartOffset(int)
,
setEndOffset(int)
setType(java.lang.String)
public Token reinit(String newTerm, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset, String newType)
clear()
,
CharTermAttributeImpl.append(CharSequence, int, int)
,
setStartOffset(int)
,
setEndOffset(int)
setType(java.lang.String)
public Token reinit(String newTerm, int newStartOffset, int newEndOffset)
clear()
,
CharTermAttributeImpl.append(CharSequence)
,
setStartOffset(int)
,
setEndOffset(int)
setType(java.lang.String)
on Token.DEFAULT_TYPEpublic Token reinit(String newTerm, int newTermOffset, int newTermLength, int newStartOffset, int newEndOffset)
clear()
,
CharTermAttributeImpl.append(CharSequence, int, int)
,
setStartOffset(int)
,
setEndOffset(int)
setType(java.lang.String)
on Token.DEFAULT_TYPEpublic void reinit(Token prototype)
prototype
- public void reinit(Token prototype, String newTerm)
prototype
- newTerm
- public void reinit(Token prototype, char[] newTermBuffer, int offset, int length)
prototype
- newTermBuffer
- offset
- length
- public void copyTo(AttributeImpl target)
AttributeImpl
copyTo
in class CharTermAttributeImpl
public void reflectWith(AttributeReflector reflector)
AttributeImpl
AttributeReflector
.
The default implementation calls AttributeReflector.reflect(java.lang.Class<? extends org.apache.lucene.util.Attribute>, java.lang.String, java.lang.Object)
for all
non-static fields from the implementing class, using the field name as key
and the field value as value. The Attribute class is also determined by reflection.
Please note that the default implementation can only handle single-Attribute
implementations.
Custom implementations look like this (e.g. for a combined attribute implementation):
public void reflectWith(AttributeReflector reflector) { reflector.reflect(CharTermAttribute.class, "term", term()); reflector.reflect(PositionIncrementAttribute.class, "positionIncrement", getPositionIncrement()); }
If you implement this method, make sure that for each invocation, the same set of Attribute
interfaces and keys are passed to AttributeReflector.reflect(java.lang.Class<? extends org.apache.lucene.util.Attribute>, java.lang.String, java.lang.Object)
in the same order, but possibly
different values. So don't automatically exclude e.g. null
properties!
reflectWith
in class CharTermAttributeImpl
AttributeImpl.reflectAsString(boolean)