public final class ICUCollationKeyFilter extends TokenFilter
Converts each token into its CollationKey, and
then encodes the CollationKey with IndexableBinaryStringTools, to
allow it to be stored as an index term.
WARNING: Make sure you use exactly the same Collator at
index and query time -- CollationKeys are only comparable when produced by
the same Collator. RuleBasedCollators are
independently versioned, so it is safe to search against stored
CollationKeys if the following are exactly the same (best practice is
to store this information with the index and check that they remain the
same at query time):
Collator.getVersion()
Collator.setStrength(int)
CollationKeys generated by ICU Collators are not compatible with those
generated by java.text.Collators. Specifically, if you use
ICUCollationKeyFilter to generate index terms, do not use
CollationKeyFilter on the query side, or vice versa.
ICUCollationKeyFilter is significantly faster and generates significantly shorter keys than CollationKeyFilter. See http://site.icu-project.org/charts/collation-icu4j-sun for key generation timing and key length comparisons between ICU4J and java.text.Collator over several languages.
AttributeSource.AttributeFactory, AttributeSource.Stateinput| Constructor and Description |
|---|
ICUCollationKeyFilter(TokenStream input,
com.ibm.icu.text.Collator collator) |
| Modifier and Type | Method and Description |
|---|---|
boolean |
incrementToken()
Consumers (i.e.,
IndexWriter) use this method to advance the stream to
the next token. |
close, end, resetaddAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toStringpublic ICUCollationKeyFilter(TokenStream input, com.ibm.icu.text.Collator collator)
input - Source token streamcollator - CollationKey generatorpublic boolean incrementToken()
throws IOException
TokenStreamIndexWriter) use this method to advance the stream to
the next token. Implementing classes must implement this method and update
the appropriate AttributeImpls with the attributes of the next
token.
The producer must make no assumptions about the attributes after the method
has been returned: the caller may arbitrarily change it. If the producer
needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState() to create a copy of the current attribute state.
This method is called for every token of a document, so an efficient
implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class) and AttributeSource.getAttribute(Class),
references to all AttributeImpls that this stream uses should be
retrieved during instantiation.
To ensure that filters and consumers know which attributes are available,
the attributes must be added during instantiation. Filters and consumers
are not required to check for availability of attributes in
TokenStream.incrementToken().
incrementToken in class TokenStreamIOException