org.apache.lucene.collation
Class ICUCollationKeyAnalyzer
java.lang.Object
org.apache.lucene.analysis.Analyzer
org.apache.lucene.collation.ICUCollationKeyAnalyzer
public class ICUCollationKeyAnalyzer
- extends org.apache.lucene.analysis.Analyzer
Filters KeywordTokenizer
with ICUCollationKeyFilter
.
Converts the token into its CollationKey
, and
then encodes the CollationKey with
IndexableBinaryStringTools
, to allow it to
be stored as an index term.
WARNING: Make sure you use exactly the same Collator at
index and query time -- CollationKeys are only comparable when produced by
the same Collator. RuleBasedCollator
s are
independently versioned, so it is safe to search against stored
CollationKeys if the following are exactly the same (best practice is
to store this information with the index and check that they remain the
same at query time):
-
Collator version - see
Collator.getVersion()
-
The collation strength used - see
Collator.setStrength(int)
CollationKeys generated by ICU Collators are not compatible with those
generated by java.text.Collators. Specifically, if you use
ICUCollationKeyAnalyzer to generate index terms, do not use
CollationKeyAnalyzer
on the query side, or vice versa.
ICUCollationKeyAnalyzer is significantly faster and generates significantly
shorter keys than CollationKeyAnalyzer. See
http://site.icu-project.org/charts/collation-icu4j-sun for key
generation timing and key length comparisons between ICU4J and
java.text.Collator over several languages.
Fields inherited from class org.apache.lucene.analysis.Analyzer |
overridesTokenStreamMethod |
Methods inherited from class org.apache.lucene.analysis.Analyzer |
close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setOverridesTokenStreamMethod, setPreviousTokenStream |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
ICUCollationKeyAnalyzer
public ICUCollationKeyAnalyzer(com.ibm.icu.text.Collator collator)
tokenStream
public org.apache.lucene.analysis.TokenStream tokenStream(String fieldName,
Reader reader)
- Specified by:
tokenStream
in class org.apache.lucene.analysis.Analyzer
reusableTokenStream
public org.apache.lucene.analysis.TokenStream reusableTokenStream(String fieldName,
Reader reader)
throws IOException
- Overrides:
reusableTokenStream
in class org.apache.lucene.analysis.Analyzer
- Throws:
IOException
Copyright © 2000-2010 Apache Software Foundation. All Rights Reserved.