public final class FixedShingleFilter extends TokenFilter
Unlike the ShingleFilter
, FixedShingleFilter only emits shingles of a
fixed size, and never emits unigrams, even at the end of a TokenStream. In
addition, if the filter encounters stacked tokens (eg synonyms), then it will
output stacked shingles
For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles".
This filter handles position increments > 1 by inserting filler tokens (tokens with termtext "_").
AttributeSource.State
input
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Constructor and Description |
---|
FixedShingleFilter(TokenStream input,
int shingleSize)
Creates a FixedShingleFilter over an input token stream
|
FixedShingleFilter(TokenStream input,
int shingleSize,
String tokenSeparator,
String fillerToken)
Creates a FixedShingleFilter over an input token stream
|
Modifier and Type | Method and Description |
---|---|
void |
end() |
boolean |
incrementToken() |
void |
reset() |
close
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
public FixedShingleFilter(TokenStream input, int shingleSize)
input
- the input streamshingleSize
- the shingle sizepublic FixedShingleFilter(TokenStream input, int shingleSize, String tokenSeparator, String fillerToken)
input
- the input tokenstreamshingleSize
- the shingle sizetokenSeparator
- a String to use as a token separatorfillerToken
- a String to use to represent gaps in the input stream (due to eg stopwords)public boolean incrementToken() throws IOException
incrementToken
in class TokenStream
IOException
public void reset() throws IOException
reset
in class TokenFilter
IOException
public void end() throws IOException
end
in class TokenFilter
IOException
Copyright © 2000-2018 Apache Software Foundation. All Rights Reserved.