public class NGramTokenizer extends Tokenizer
On the contrary to NGramTokenFilter
, this class sets offsets so
that characters between startOffset and endOffset in the original stream are
the same as the term chars.
For example, "abcde" would be tokenized as (minGram=2, maxGram=3):
Term | ab | abc | bc | bcd | cd | cde | de |
---|---|---|---|---|---|---|---|
Position increment | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Position length | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Offsets | [0,2[ | [0,3[ | [1,3[ | [1,4[ | [2,4[ | [2,5[ | [3,5[ |
This tokenizer changed a lot in Lucene 4.4 in order to:
pre-tokenize
the stream
before computing n-grams.Additionally, this class doesn't trim trailing whitespaces and emits tokens in a different order, tokens are now emitted by increasing start offsets while they used to be emitted by increasing lengths (which prevented from supporting large input streams).
AttributeSource.State
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_MAX_NGRAM_SIZE |
static int |
DEFAULT_MIN_NGRAM_SIZE |
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Constructor and Description |
---|
NGramTokenizer()
Creates NGramTokenizer with default min and max n-grams.
|
NGramTokenizer(AttributeFactory factory,
int minGram,
int maxGram)
Creates NGramTokenizer with given min and max n-grams.
|
NGramTokenizer(int minGram,
int maxGram)
Creates NGramTokenizer with given min and max n-grams.
|
Modifier and Type | Method and Description |
---|---|
void |
end() |
boolean |
incrementToken() |
protected boolean |
isTokenChar(int chr)
Only collect characters which satisfy this condition.
|
void |
reset() |
close, correctOffset, setReader
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
public static final int DEFAULT_MIN_NGRAM_SIZE
public static final int DEFAULT_MAX_NGRAM_SIZE
public NGramTokenizer(int minGram, int maxGram)
minGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generatepublic NGramTokenizer(AttributeFactory factory, int minGram, int maxGram)
factory
- AttributeFactory
to useminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generatepublic NGramTokenizer()
public final boolean incrementToken() throws IOException
incrementToken
in class TokenStream
IOException
protected boolean isTokenChar(int chr)
public final void end() throws IOException
end
in class TokenStream
IOException
public final void reset() throws IOException
reset
in class Tokenizer
IOException
Copyright © 2000-2019 Apache Software Foundation. All Rights Reserved.