org.apache.lucene.analysis.ngram

Class EdgeNGramTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.ngram.NGramTokenizer

org.apache.lucene.analysis.ngram.EdgeNGramTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

public class EdgeNGramTokenizer
extends NGramTokenizer

Tokenizes the input from an edge into n-grams of given size(s).

This Tokenizer create n-grams from the beginning edge or ending edge of a input token.

As of Lucene 4.4, this tokenizer

• can handle maxGram larger than 1024 chars, but beware that this will result in increased memory usage
• doesn't trim the input,
• sets position increments equal to 1 instead of 1 for the first token and 0 for all other ones
• doesn't support backward n-grams anymore.
• supports pre-tokenization,
• correctly handles supplementary characters.

Although highly discouraged, it is still possible to use the old behavior through Lucene43EdgeNGramTokenizer.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_MAX_GRAM_SIZE
static int	DEFAULT_MIN_GRAM_SIZE

Fields inherited from class org.apache.lucene.analysis.ngram.NGramTokenizer

DEFAULT_MAX_NGRAM_SIZE, DEFAULT_MIN_NGRAM_SIZE

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Fields inherited from class org.apache.lucene.util.AttributeSource

DEFAULT_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor and Description
EdgeNGramTokenizer(AttributeFactory factory, Reader input, int minGram, int maxGram) Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
EdgeNGramTokenizer(Reader input, int minGram, int maxGram) Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
EdgeNGramTokenizer(Version version, AttributeFactory factory, Reader input, int minGram, int maxGram) Deprecated. Use EdgeNGramTokenizer(AttributeFactory, Reader, int, int)
EdgeNGramTokenizer(Version version, Reader input, int minGram, int maxGram) Deprecated. Use EdgeNGramTokenizer(Reader, int, int)

Method Summary

Methods inherited from class org.apache.lucene.analysis.ngram.NGramTokenizer

end, incrementToken, isTokenChar, reset

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close, correctOffset, setReader

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_MAX_GRAM_SIZE

public static final int DEFAULT_MAX_GRAM_SIZE

See Also:

Constant Field Values

DEFAULT_MIN_GRAM_SIZE

public static final int DEFAULT_MIN_GRAM_SIZE

See Also:

Constant Field Values

Constructor Detail

EdgeNGramTokenizer

public EdgeNGramTokenizer(Reader input,
                  int minGram,
                  int maxGram)

Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range

Parameters:

input - Reader holding the input to be tokenized

minGram - the smallest n-gram to generate

maxGram - the largest n-gram to generate

EdgeNGramTokenizer

@Deprecated
public EdgeNGramTokenizer(Version version,
                             Reader input,
                             int minGram,
                             int maxGram)

Deprecated. Use EdgeNGramTokenizer(Reader, int, int)

EdgeNGramTokenizer

public EdgeNGramTokenizer(AttributeFactory factory,
                  Reader input,
                  int minGram,
                  int maxGram)

Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range

Parameters:

factory - AttributeFactory to use

input - Reader holding the input to be tokenized

minGram - the smallest n-gram to generate

maxGram - the largest n-gram to generate

EdgeNGramTokenizer

@Deprecated
public EdgeNGramTokenizer(Version version,
                             AttributeFactory factory,
                             Reader input,
                             int minGram,
                             int maxGram)

Deprecated. Use EdgeNGramTokenizer(AttributeFactory, Reader, int, int)