Class OpenNLPTokenizer
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.Tokenizer
-
- org.apache.lucene.analysis.util.SegmentingTokenizerBase
-
- org.apache.lucene.analysis.opennlp.OpenNLPTokenizer
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
public final class OpenNLPTokenizer extends SegmentingTokenizerBase
Run OpenNLP SentenceDetector and Tokenizer. The index of each sentence is stored in SentenceAttribute.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State
-
-
Field Summary
-
Fields inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase
buffer, BUFFERMAX, offset
-
Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
-
-
Constructor Summary
Constructors Constructor Description OpenNLPTokenizer(AttributeFactory factory, NLPSentenceDetectorOp sentenceOp, NLPTokenizerOp tokenizerOp)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
protected boolean
incrementWord()
void
reset()
protected void
setNextSentence(int sentenceStart, int sentenceEnd)
-
Methods inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase
end, incrementToken, isSafeEnd
-
Methods inherited from class org.apache.lucene.analysis.Tokenizer
correctOffset, setReader, setReaderTestPoint
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
-
-
-
Constructor Detail
-
OpenNLPTokenizer
public OpenNLPTokenizer(AttributeFactory factory, NLPSentenceDetectorOp sentenceOp, NLPTokenizerOp tokenizerOp) throws IOException
- Throws:
IOException
-
-
Method Detail
-
close
public void close() throws IOException
- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Overrides:
close
in classTokenizer
- Throws:
IOException
-
setNextSentence
protected void setNextSentence(int sentenceStart, int sentenceEnd)
- Specified by:
setNextSentence
in classSegmentingTokenizerBase
-
incrementWord
protected boolean incrementWord()
- Specified by:
incrementWord
in classSegmentingTokenizerBase
-
reset
public void reset() throws IOException
- Overrides:
reset
in classSegmentingTokenizerBase
- Throws:
IOException
-
-