org.apache.lucene.analysis.icu.segmentation

Class ICUTokenizerFactory

java.lang.Object

org.apache.lucene.analysis.util.AbstractAnalysisFactory

org.apache.lucene.analysis.util.TokenizerFactory

org.apache.lucene.analysis.icu.segmentation.ICUTokenizerFactory

All Implemented Interfaces:

ResourceLoaderAware

public class ICUTokenizerFactory
extends TokenizerFactory
implements ResourceLoaderAware

Factory for ICUTokenizer. Words are broken across script boundaries, then segmented according to the BreakIterator and typing provided by the DefaultICUTokenizerConfig.

To use the default set of per-script rules:

 <fieldType name="text_icu" class="solr.TextField" positionIncrementGap="100">
   <analyzer>
     <tokenizer class="solr.ICUTokenizerFactory"/>
   </analyzer>
 </fieldType>

You can customize this tokenizer's behavior by specifying per-script rule files, which are compiled by the ICU RuleBasedBreakIterator. See the ICU RuleBasedBreakIterator syntax reference. To add per-script rules, add a "rulefiles" argument, which should contain a comma-separated list of code:rulefile pairs in the following format: four-letter ISO 15924 script code, followed by a colon, then a resource path. E.g. to specify rules for Latin (script code "Latn") and Cyrillic (script code "Cyrl"):

 <fieldType name="text_icu_custom" class="solr.TextField" positionIncrementGap="100">
   <analyzer>
     <tokenizer class="solr.ICUTokenizerFactory"
                rulefiles="Latn:my.Latin.rules.rbbi,Cyrl:my.Cyrillic.rules.rbbi"/>
   </analyzer>
 </fieldType>

Field Summary

Fields inherited from class org.apache.lucene.analysis.util.AbstractAnalysisFactory

args, luceneMatchVersion

Constructor Summary

Constructors

Constructor and Description
ICUTokenizerFactory() Sole constructor.

Method Summary

Methods

Modifier and Type	Method and Description
Tokenizer	create(Reader input)
void	inform(ResourceLoader loader)
void	init(Map<String,String> args)

Methods inherited from class org.apache.lucene.analysis.util.TokenizerFactory

availableTokenizers, forName, lookupClass, reloadTokenizers

Methods inherited from class org.apache.lucene.analysis.util.AbstractAnalysisFactory

assureMatchVersion, getArgs, getBoolean, getBoolean, getInt, getInt, getInt, getLines, getLuceneMatchVersion, getPattern, getSnowballWordSet, getWordSet, setLuceneMatchVersion, splitFileNames

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ICUTokenizerFactory

public ICUTokenizerFactory()

Sole constructor. See AbstractAnalysisFactory for initialization lifecycle.

Method Detail

init

public void init(Map<String,String> args)

Overrides:

init in class AbstractAnalysisFactory

inform

public void inform(ResourceLoader loader)
            throws IOException

Specified by:

inform in interface ResourceLoaderAware

Throws:

IOException

create

public Tokenizer create(Reader input)

Specified by:

create in class TokenizerFactory