Package org.apache.lucene.analysis.th

Class ThaiTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.util.SegmentingTokenizerBase

org.apache.lucene.analysis.th.ThaiTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

public class ThaiTokenizer
extends SegmentingTokenizerBase

Tokenizer that use BreakIterator to tokenize Thai text.

WARNING: this tokenizer may not be supported by all JREs. It is known to work with Sun/Oracle and Harmony JREs. If your application needs to be fully portable, consider using ICUTokenizer instead, which uses an ICU Thai BreakIterator that will always be available.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields

Modifier and Type	Field	Description
static final boolean	DBBI_AVAILABLE	True if the JRE supports a working dictionary-based breakiterator for Thai.

Fields inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

buffer, BUFFERMAX, offset

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor	Description
ThaiTokenizer()	Creates a new ThaiTokenizer
ThaiTokenizer(AttributeFactory factory)	Creates a new ThaiTokenizer, supplying the AttributeFactory

Method Summary

Modifier and Type	Method	Description
protected boolean	incrementWord()	Returns true if another word is available
protected void	setNextSentence(int sentenceStart, int sentenceEnd)	Provides the next input sentence for analysis

Methods inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

end, incrementToken, isSafeEnd, reset

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close, correctOffset, setReader, setReaderTestPoint

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

DBBI_AVAILABLE

public static final boolean DBBI_AVAILABLE

True if the JRE supports a working dictionary-based breakiterator for Thai. If this is false, this tokenizer will not work at all!

Constructor Details

ThaiTokenizer

public ThaiTokenizer()

Creates a new ThaiTokenizer

ThaiTokenizer

public ThaiTokenizer(AttributeFactory factory)

Creates a new ThaiTokenizer, supplying the AttributeFactory

Method Details

setNextSentence

protected void setNextSentence(int sentenceStart,
 int sentenceEnd)

Description copied from class: SegmentingTokenizerBase

Provides the next input sentence for analysis

Specified by:

setNextSentence in class SegmentingTokenizerBase

incrementWord

protected boolean incrementWord()

Description copied from class: SegmentingTokenizerBase

Returns true if another word is available

Specified by:

incrementWord in class SegmentingTokenizerBase