public abstract class GraphTokenFilter extends TokenFilter
incrementBaseToken()
to move the root of the graph to the next
position in the TokenStream, incrementGraphToken()
to move along
the current graph, and incrementGraph()
to reset to the next graph
based at the current root.
For example, given the stream 'a b/c:2 d e`, then with the base token at
'a', incrementGraphToken() will produce the stream 'a b d e', and then
after calling incrementGraph() will produce the stream 'a c e'.AttributeSource.State
Modifier and Type | Field and Description |
---|---|
static int |
MAX_GRAPH_STACK_SIZE
The maximum permitted number of routes through a graph
|
static int |
MAX_TOKEN_CACHE_SIZE
The maximum permitted read-ahead in the token stream
|
input
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Constructor and Description |
---|
GraphTokenFilter(TokenStream input)
Create a new GraphTokenFilter
|
Modifier and Type | Method and Description |
---|---|
void |
end()
This method is called by the consumer after the last token has been
consumed, after
TokenStream.incrementToken() returned false
(using the new TokenStream API). |
int |
getTrailingPositions()
Return the number of trailing positions at the end of the graph
NB this should only be called after
incrementGraphToken() has returned false |
protected boolean |
incrementBaseToken()
Move the root of the graph to the next token in the wrapped TokenStream
|
protected boolean |
incrementGraph()
Reset to the root token again, and move down the next route through the graph
|
protected boolean |
incrementGraphToken()
Move to the next token in the current route through the graph
|
void |
reset()
This method is called by a consumer before it begins consumption using
TokenStream.incrementToken() . |
close
incrementToken
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
public static final int MAX_GRAPH_STACK_SIZE
public static final int MAX_TOKEN_CACHE_SIZE
public GraphTokenFilter(TokenStream input)
protected final boolean incrementBaseToken() throws IOException
false
if the underlying stream is exhaustedIOException
protected final boolean incrementGraphToken() throws IOException
false
if there are not more tokens in the current graphIOException
protected final boolean incrementGraph() throws IOException
IOException
public int getTrailingPositions()
incrementGraphToken()
has returned false
public void end() throws IOException
TokenFilter
TokenStream.incrementToken()
returned false
(using the new TokenStream
API). Streams implementing the old API
should upgrade to use this feature.
This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.
Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.
If you override this method, always call super.end()
.
NOTE:
The default implementation chains the call to the input TokenStream, so
be sure to call super.end()
first when overriding this method.
end
in class TokenFilter
IOException
- If an I/O error occurspublic void reset() throws IOException
TokenFilter
TokenStream.incrementToken()
.
Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call super.reset()
, otherwise
some internal state will not be correctly reset (e.g., Tokenizer
will
throw IllegalStateException
on further usage).
NOTE:
The default implementation chains the call to the input TokenStream, so
be sure to call super.reset()
when overriding this method.
reset
in class TokenFilter
IOException
Copyright © 2000-2021 Apache Software Foundation. All Rights Reserved.