Package org.apache.lucene.analysis.ngram

Class NGramTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.ngram.NGramTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

EdgeNGramTokenizer

public class NGramTokenizer
extends Tokenizer

Tokenizes the input into n-grams of the given size(s).

On the contrary to NGramTokenFilter, this class sets offsets so that characters between startOffset and endOffset in the original stream are the same as the term chars.

For example, "abcde" would be tokenized as (minGram=2, maxGram=3):

ngram tokens example

Term	ab	abc	bc	bcd	cd	cde	de
Position increment	1	1	1	1	1	1	1
Position length	1	1	1	1	1	1	1
Offsets	[0,2[	[0,3[	[1,3[	[1,4[	[2,4[	[2,5[	[3,5[

This tokenizer changed a lot in Lucene 4.4 in order to:

• tokenize in a streaming fashion to support streams which are larger than 1024 chars (limit of the previous version),
• count grams based on unicode code points instead of java chars (and never split in the middle of surrogate pairs),
• give the ability to pre-tokenize the stream before computing n-grams.

Additionally, this class doesn't trim trailing whitespaces and emits tokens in a different order, tokens are now emitted by increasing start offsets while they used to be emitted by increasing lengths (which prevented from supporting large input streams).

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_MAX_NGRAM_SIZE
static final int	DEFAULT_MIN_NGRAM_SIZE

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor	Description
NGramTokenizer()	Creates NGramTokenizer with default min and max n-grams.
NGramTokenizer(int minGram, int maxGram)	Creates NGramTokenizer with given min and max n-grams.
NGramTokenizer(AttributeFactory factory, int minGram, int maxGram)	Creates NGramTokenizer with given min and max n-grams.

Method Summary

Modifier and Type	Method	Description
final void	end()
final boolean	incrementToken()
protected boolean	isTokenChar(int chr)	Only collect characters which satisfy this condition.
final void	reset()

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close, correctOffset, setReader, setReaderTestPoint

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

DEFAULT_MIN_NGRAM_SIZE

public static final int DEFAULT_MIN_NGRAM_SIZE

See Also:

• Constant Field Values

DEFAULT_MAX_NGRAM_SIZE

public static final int DEFAULT_MAX_NGRAM_SIZE

See Also:

• Constant Field Values

Constructor Details

NGramTokenizer

public NGramTokenizer(int minGram,
 int maxGram)

Creates NGramTokenizer with given min and max n-grams.

Parameters:

minGram - the smallest n-gram to generate

maxGram - the largest n-gram to generate

NGramTokenizer

public NGramTokenizer(AttributeFactory factory,
 int minGram,
 int maxGram)

Creates NGramTokenizer with given min and max n-grams.

Parameters:

factory - AttributeFactory to use

minGram - the smallest n-gram to generate

maxGram - the largest n-gram to generate

NGramTokenizer

public NGramTokenizer()

Creates NGramTokenizer with default min and max n-grams.

Method Details

incrementToken

public final boolean incrementToken()
                             throws IOException

Specified by:

incrementToken in class TokenStream

Throws:

IOException

isTokenChar

protected boolean isTokenChar(int chr)

Only collect characters which satisfy this condition.

end

public final void end()
               throws IOException

Overrides:

end in class TokenStream

Throws:

IOException

reset

public final void reset()
                 throws IOException

Overrides:

reset in class Tokenizer

Throws:

IOException