HMMChineseTokenizer
instead@Deprecated public final class SentenceTokenizer extends Tokenizer
The output tokens can then be broken into words with WordTokenFilter
AttributeSource.State
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
DEFAULT_ATTRIBUTE_FACTORY
Constructor and Description |
---|
SentenceTokenizer(AttributeFactory factory,
Reader reader)
Deprecated.
|
SentenceTokenizer(Reader reader)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
void |
end()
Deprecated.
|
boolean |
incrementToken()
Deprecated.
|
void |
reset()
Deprecated.
|
close, correctOffset, setReader
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
public SentenceTokenizer(Reader reader)
public SentenceTokenizer(AttributeFactory factory, Reader reader)
public boolean incrementToken() throws IOException
incrementToken
in class TokenStream
IOException
public void reset() throws IOException
reset
in class Tokenizer
IOException
public void end() throws IOException
end
in class TokenStream
IOException
Copyright © 2000-2014 Apache Software Foundation. All Rights Reserved.