org.apache.lucene.analysis.cn.smart.hhmm
Class SegTokenFilter

java.lang.Object
  org.apache.lucene.analysis.cn.smart.hhmm.SegTokenFilter

public class SegTokenFilter
extends Object

Filters a SegToken by converting full-width latin to half-width, then lowercasing latin. Additionally, all punctuation is converted into Utility.COMMON_DELIMITER

WARNING: This API is experimental and might change in incompatible ways in the next release.

Constructor Summary

SegTokenFilter()

Method Summary

SegToken

filter(SegToken token) Filter an input SegToken

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SegTokenFilter

public SegTokenFilter()

Method Detail

filter

public SegToken filter(SegToken token)

Filter an input SegToken

Full-width latin will be converted to half-width, then all latin will be lowercased. All punctuation is converted into Utility.COMMON_DELIMITER

Parameters:

token - input SegToken

Returns:

normalized SegToken