Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.
Use StandardTokenizer, CJKWidthFilter, CJKBigramFilter, and LowerCaseFilter instead.
TokenFilter that normalizes CJK width differences:
Folds fullwidth ASCII variants into the equivalent basic latin
Folds halfwidth Katakana variants into the equivalent kana
Three analyzers are provided for Chinese, each of which treats Chinese text in a different way.