Package org.apache.lucene.analysis

Class Token

  • All Implemented Interfaces:
    Appendable, CharSequence, Cloneable, CharTermAttribute, FlagsAttribute, OffsetAttribute, PayloadAttribute, PositionIncrementAttribute, PositionLengthAttribute, TermFrequencyAttribute, TermToBytesRefAttribute, TypeAttribute, Attribute
    public class Token
extends PackedTokenAttributeImpl
implements FlagsAttribute, PayloadAttribute
    A Token is an occurrence of a term from the text of a field. It consists of a term's text, the start and end offset of the term in the text of the field, and a type string.

    The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC display, etc.

    The type is a string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".

    A Token can optionally have metadata (a.k.a. payload) in the form of a variable length byte array. Use PostingsEnum.getPayload() to retrieve the payloads from the index.

    A few things to note:

    • clear() initializes all of the fields to default values. This was changed in contrast to Lucene 2.4, but should affect no one.
    • Because TokenStreams can be chained, one cannot assume that the Token's current type is correct.
    • The startOffset and endOffset represent the start and offset in the source text, so be careful in adjusting them.
    • When caching a reusable token, clone it. When injecting a cached token into a stream that can be reset, clone it again.

    • Field Detail

      • TOKEN_ATTRIBUTE_FACTORY

        public static final AttributeFactory TOKEN_ATTRIBUTE_FACTORY
        Convenience factory that returns Token as implementation for the basic attributes and return the default impl (with "Impl" appended) for all other attributes.
        Since:
        3.0

    • Constructor Detail

      • Token

        public Token()
        Constructs a Token will null text.

      • Token

        public Token​(CharSequence text,
             int start,
             int end)
        Constructs a Token with the given term text, start and end offsets. The type defaults to "word." NOTE: for better indexing speed you should instead use the char[] termBuffer methods to set the term text.
        Parameters:
        text - term text
        start - start offset in the source text
        end - end offset in the source text

      • Token

        public Token​(CharSequence text,
             int posInc,
             int start,
             int end)
        Constructs a Token with the given term text, position increment, start and end offsets

      • Token

        public Token​(CharSequence text,
             int posInc,
             int start,
             int end,
             int posLength)