SegTokenFilter (Lucene 9.0.0 smartcn API)

java.lang.Object
- org.apache.lucene.analysis.cn.smart.hhmm.SegTokenFilter

```
public class SegTokenFilter
extends Object
```
Filters a SegToken by converting full-width latin to half-width, then lowercasing latin. Additionally, all punctuation is converted into Utility.COMMON_DELIMITER

WARNING: This API is experimental and might change in incompatible ways in the next release.

- Constructor Summary
  
  Constructors
  Constructor Description
  
  SegTokenFilter()
- Method Summary
  
  All Methods Instance Methods Concrete Methods
  Modifier and Type Method Description
  
  SegToken filter(SegToken token)
  Filter an input SegToken
  - Methods inherited from class java.lang.Object
    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - SegTokenFilter
```
public SegTokenFilter()
```
- Method Detail
  - filter
```
public SegToken filter(SegToken token)
```
    Filter an input SegToken
    Full-width latin will be converted to half-width, then all latin will be lowercased. All punctuation is converted into Utility.COMMON_DELIMITER
    
    Parameters:
    
    token - input SegToken
    
    Returns:
    
    normalized SegToken