HMMChineseTokenizer
instead@Deprecated public final class SentenceTokenizer extends Tokenizer
The output tokens can then be broken into words with WordTokenFilter
AttributeSource.State
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Constructor and Description |
---|
SentenceTokenizer()
Deprecated.
|
SentenceTokenizer(AttributeFactory factory)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
void |
end()
Deprecated.
|
boolean |
incrementToken()
Deprecated.
|
void |
reset()
Deprecated.
|
close, correctOffset, setReader
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
public SentenceTokenizer()
public SentenceTokenizer(AttributeFactory factory)
public boolean incrementToken() throws IOException
incrementToken
in class TokenStream
IOException
public void reset() throws IOException
reset
in class Tokenizer
IOException
public void end() throws IOException
end
in class TokenStream
IOException
Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.