ThaiTokenizer (Lucene 8.5.1 API)

java.lang.Object
- org.apache.lucene.util.AttributeSource
- - org.apache.lucene.analysis.TokenStream
  - - org.apache.lucene.analysis.Tokenizer
    - - org.apache.lucene.analysis.util.SegmentingTokenizerBase
      - org.apache.lucene.analysis.th.ThaiTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable
```
public class ThaiTokenizer
extends SegmentingTokenizerBase
```
Tokenizer that use BreakIterator to tokenize Thai text.
WARNING: this tokenizer may not be supported by all JREs. It is known to work with Sun/Oracle and Harmony JREs. If your application needs to be fully portable, consider using ICUTokenizer instead, which uses an ICU Thai BreakIterator that will always be available.

Nested Class Summary
- Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
  AttributeSource.State

Field Summary

Fields
Modifier and Type Field Description

static boolean DBBI_AVAILABLE
True if the JRE supports a working dictionary-based breakiterator for Thai.
- Fields inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase
  buffer, BUFFERMAX, offset
- Fields inherited from class org.apache.lucene.analysis.Tokenizer
  input
- Fields inherited from class org.apache.lucene.analysis.TokenStream
  DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors
Constructor Description

ThaiTokenizer()
Creates a new ThaiTokenizer

ThaiTokenizer(AttributeFactory factory)
Creates a new ThaiTokenizer, supplying the AttributeFactory

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`protected boolean`	`incrementWord()`	Returns true if another word is available
`protected void`	`setNextSentence(int sentenceStart, int sentenceEnd)`	Provides the next input sentence for analysis

Methods inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase
end, incrementToken, isSafeEnd, reset

Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset, setReader

Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

- Field Detail
  - DBBI_AVAILABLE
```
public static final boolean DBBI_AVAILABLE
```
    True if the JRE supports a working dictionary-based breakiterator for Thai. If this is false, this tokenizer will not work at all!
- Constructor Detail
  - ThaiTokenizer
```
public ThaiTokenizer()
```
    Creates a new ThaiTokenizer
  - ThaiTokenizer
```
public ThaiTokenizer(AttributeFactory factory)
```
    Creates a new ThaiTokenizer, supplying the AttributeFactory
- Method Detail
  - setNextSentence
```
protected void setNextSentence(int sentenceStart,
                               int sentenceEnd)
```
    Description copied from class: SegmentingTokenizerBase
    
    Provides the next input sentence for analysis
    
    Specified by:
    
    setNextSentence in class SegmentingTokenizerBase
  - incrementWord
```
protected boolean incrementWord()
```
    Description copied from class: SegmentingTokenizerBase
    
    Returns true if another word is available
    
    Specified by:
    
    incrementWord in class SegmentingTokenizerBase