public final class FixedShingleFilter extends GraphTokenFilter
Unlike the ShingleFilter
, FixedShingleFilter only emits shingles of a
fixed size, and never emits unigrams, even at the end of a TokenStream. In
addition, if the filter encounters stacked tokens (eg synonyms), then it will
output stacked shingles
For example, the sentence "please divide this sentence into shingles" might be tokenized into shingles "please divide", "divide this", "this sentence", "sentence into", and "into shingles".
This filter handles position increments > 1 by inserting filler tokens (tokens with termtext "_").
AttributeSource.State
MAX_GRAPH_STACK_SIZE, MAX_TOKEN_CACHE_SIZE
input
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Constructor and Description |
---|
FixedShingleFilter(TokenStream input,
int shingleSize)
Creates a FixedShingleFilter over an input token stream
|
FixedShingleFilter(TokenStream input,
int shingleSize,
String tokenSeparator,
String fillerToken)
Creates a FixedShingleFilter over an input token stream
|
Modifier and Type | Method and Description |
---|---|
boolean |
incrementToken() |
end, getTrailingPositions, incrementBaseToken, incrementGraph, incrementGraphToken, reset
close
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
public FixedShingleFilter(TokenStream input, int shingleSize)
input
- the input streamshingleSize
- the shingle sizepublic FixedShingleFilter(TokenStream input, int shingleSize, String tokenSeparator, String fillerToken)
input
- the input tokenstreamshingleSize
- the shingle sizetokenSeparator
- a String to use as a token separatorfillerToken
- a String to use to represent gaps in the input stream (due to eg stopwords)public boolean incrementToken() throws IOException
incrementToken
in class TokenStream
IOException
Copyright © 2000-2019 Apache Software Foundation. All Rights Reserved.