Class SegTokenFilter

  extended by

public class SegTokenFilter
extends Object

Filters a SegToken by converting full-width latin to half-width, then lowercasing latin. Additionally, all punctuation is converted into Utility.COMMON_DELIMITER

WARNING: The status of the analyzers/smartcn package is experimental. The APIs and file formats introduced here might change in the future and will not be supported anymore in such a case.

Constructor Summary
Method Summary
 SegToken filter(SegToken token)
          Filter an input SegToken
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail


public SegTokenFilter()
Method Detail


public SegToken filter(SegToken token)
Filter an input SegToken

Full-width latin will be converted to half-width, then all latin will be lowercased. All punctuation is converted into Utility.COMMON_DELIMITER

token - input SegToken
normalized SegToken

Copyright © 2000-2010 Apache Software Foundation. All Rights Reserved.