org.apache.lucene.analysis.icu.segmentation

Class DefaultICUTokenizerConfig

java.lang.Object

org.apache.lucene.analysis.icu.segmentation.ICUTokenizerConfig

org.apache.lucene.analysis.icu.segmentation.DefaultICUTokenizerConfig

public class DefaultICUTokenizerConfig
extends ICUTokenizerConfig

Default ICUTokenizerConfig that is generally applicable to many languages.

Generally tokenizes Unicode text according to UAX#29 (BreakIterator.getWordInstance(ULocale.ROOT)), but with the following tailorings:

• Thai, Lao, Myanmar, and CJK text is broken into words with a dictionary.
• Khmer text is broken into syllables based on custom BreakIterator rules.

WARNING: This API is experimental and might change in incompatible ways in the next release.

Field Summary

Fields

Modifier and Type	Field and Description
static String	WORD_HANGUL Token type for words containing Korean hangul
static String	WORD_HIRAGANA Token type for words containing Japanese hiragana
static String	WORD_IDEO Token type for words containing ideographic characters
static String	WORD_KATAKANA Token type for words containing Japanese katakana
static String	WORD_LETTER Token type for words that contain letters
static String	WORD_NUMBER Token type for words that appear to be numbers

Constructor Summary

Constructors

Constructor and Description
DefaultICUTokenizerConfig(boolean cjkAsWords) Creates a new config.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	combineCJ() true if Han, Hiragana, and Katakana scripts should all be returned as Japanese
com.ibm.icu.text.BreakIterator	getBreakIterator(int script) Return a breakiterator capable of processing a given script.
String	getType(int script, int ruleStatus) Return a token type value for a given script and BreakIterator rule status.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WORD_IDEO

public static final String WORD_IDEO

Token type for words containing ideographic characters

WORD_HIRAGANA

public static final String WORD_HIRAGANA

Token type for words containing Japanese hiragana

WORD_KATAKANA

public static final String WORD_KATAKANA

Token type for words containing Japanese katakana

WORD_HANGUL

public static final String WORD_HANGUL

Token type for words containing Korean hangul

WORD_LETTER

public static final String WORD_LETTER

Token type for words that contain letters

WORD_NUMBER

public static final String WORD_NUMBER

Token type for words that appear to be numbers

Constructor Detail

DefaultICUTokenizerConfig

public DefaultICUTokenizerConfig(boolean cjkAsWords)

Creates a new config. This object is lightweight, but the first time the class is referenced, breakiterators will be initialized.

Parameters:

cjkAsWords - true if cjk text should undergo dictionary-based segmentation, otherwise text will be segmented according to UAX#29 defaults. If this is true, all Han+Hiragana+Katakana words will be tagged as IDEOGRAPHIC.

Method Detail

combineCJ

public boolean combineCJ()

Description copied from class: ICUTokenizerConfig

true if Han, Hiragana, and Katakana scripts should all be returned as Japanese

Specified by:

combineCJ in class ICUTokenizerConfig

getBreakIterator

public com.ibm.icu.text.BreakIterator getBreakIterator(int script)

Description copied from class: ICUTokenizerConfig

Return a breakiterator capable of processing a given script.

Specified by:

getBreakIterator in class ICUTokenizerConfig

getType

public String getType(int script,
             int ruleStatus)

Description copied from class: ICUTokenizerConfig

Return a token type value for a given script and BreakIterator rule status.

Specified by:

getType in class ICUTokenizerConfig