Class CJKBigramFilter

  • All Implemented Interfaces:
    Closeable, AutoCloseable

    public final class CJKBigramFilter
    extends TokenFilter
    Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.

    CJK types are set by these tokenizers, but you can also use CJKBigramFilter(TokenStream, int) to explicitly control which of the CJK scripts are turned into bigrams.

    By default, when a CJK character has no adjacent characters to form a bigram, it is output in unigram form. If you want to always output both unigrams and bigrams, set the outputUnigrams flag in CJKBigramFilter(TokenStream, int, boolean). This can be used for a combined unigram+bigram approach.

    Unlike ICUTokenizer, StandardTokenizer does not split at script boundaries. Korean Hangul characters are treated the same as many other scripts' letters, and as a result, StandardTokenizer can produce tokens that mix Hangul and non-Hangul characters, e.g. "한국abc". Such mixed-script tokens are typed as <ALPHANUM> rather than <HANGUL>, and as a result, will not be converted to bigrams by CJKBigramFilter. In all cases, all non-CJK input is passed thru unmodified.