public final class SimplePatternTokenizer extends Tokenizer
RegExp
or (expert usage) a pre-built determinized Automaton
, to locate tokens.
The regexp syntax is more limited than PatternTokenizer
, but the tokenization is quite a bit faster. The provided
regex should match valid token characters (not token separator characters, like String.split
). The matching is greedy:
the longest match at a given start point will be the next token. Empty string tokens are never produced.AttributeSource.State
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Constructor and Description |
---|
SimplePatternTokenizer(AttributeFactory factory,
Automaton dfa)
Runs a pre-built automaton.
|
SimplePatternTokenizer(AttributeFactory factory,
String regexp,
int determinizeWorkLimit)
See
RegExp for the accepted syntax. |
SimplePatternTokenizer(Automaton dfa)
Runs a pre-built automaton.
|
SimplePatternTokenizer(String regexp)
See
RegExp for the accepted syntax. |
Modifier and Type | Method and Description |
---|---|
void |
end() |
boolean |
incrementToken() |
void |
reset() |
close, correctOffset, setReader
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
public SimplePatternTokenizer(String regexp)
RegExp
for the accepted syntax.public SimplePatternTokenizer(Automaton dfa)
public SimplePatternTokenizer(AttributeFactory factory, String regexp, int determinizeWorkLimit)
RegExp
for the accepted syntax.public SimplePatternTokenizer(AttributeFactory factory, Automaton dfa)
public boolean incrementToken() throws IOException
incrementToken
in class TokenStream
IOException
public void end() throws IOException
end
in class TokenStream
IOException
public void reset() throws IOException
reset
in class Tokenizer
IOException
Copyright © 2000-2021 Apache Software Foundation. All Rights Reserved.