org.apache.lucene.analysis.pattern

Class SimplePatternSplitTokenizerFactory

java.lang.Object

org.apache.lucene.analysis.util.AbstractAnalysisFactory

org.apache.lucene.analysis.util.TokenizerFactory

org.apache.lucene.analysis.pattern.SimplePatternSplitTokenizerFactory

public class SimplePatternSplitTokenizerFactory
extends TokenizerFactory

Factory for SimplePatternSplitTokenizer, for producing tokens by splitting according to the provided regexp.

This tokenizer uses Lucene RegExp pattern matching to construct distinct tokens for the input stream. The syntax is more limited than PatternTokenizer, but the tokenization is quite a bit faster. It takes two arguments:

• "pattern" (required) is the regular expression, according to the syntax described at RegExp
• "maxDeterminizedStates" (optional, default 10000) the limit on total state count for the determined automaton computed from the regexp

The pattern matches the characters that should split tokens, like String.split, and the matching is greedy such that the longest token separator matching at a given point is matched. Empty tokens are never created.

For example, to match tokens delimited by simple whitespace characters:

 <fieldType name="text_ptn" class="solr.TextField" positionIncrementGap="100">
   <analyzer>
     <tokenizer class="solr.SimplePatternSplitTokenizerFactory" pattern="[ \t\r\n]+"/>
   </analyzer>
 </fieldType>

Since:

6.5.0

See Also:

SimplePatternSplitTokenizer

WARNING: This API is experimental and might change in incompatible ways in the next release.

SPI Name (Note: This is case-insensitive. e.g., if the name is 'htmlStrip', 'htmlstrip' can be used when looking up the service):

"simplePatternSplit"

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME SPI name
static String	PATTERN

Fields inherited from class org.apache.lucene.analysis.util.AbstractAnalysisFactory

LUCENE_MATCH_VERSION_PARAM, luceneMatchVersion

Constructor Summary

Constructors

Constructor and Description
SimplePatternSplitTokenizerFactory(Map<String,String> args) Creates a new SimpleSplitPatternTokenizerFactory

Method Summary

Modifier and Type	Method and Description
SimplePatternSplitTokenizer	create(AttributeFactory factory) Creates a TokenStream of the specified input using the given AttributeFactory

Methods inherited from class org.apache.lucene.analysis.util.TokenizerFactory

availableTokenizers, create, findSPIName, forName, lookupClass, reloadTokenizers

Methods inherited from class org.apache.lucene.analysis.util.AbstractAnalysisFactory

get, get, get, get, get, getBoolean, getChar, getClassArg, getFloat, getInt, getLines, getLuceneMatchVersion, getOriginalArgs, getPattern, getSet, getSnowballWordSet, getWordSet, isExplicitLuceneMatchVersion, require, require, require, requireBoolean, requireChar, requireFloat, requireInt, setExplicitLuceneMatchVersion, splitAt, splitFileNames

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NAME

public static final String NAME

SPI name

See Also:

Constant Field Values

PATTERN

public static final String PATTERN

See Also:

Constant Field Values

Constructor Detail

SimplePatternSplitTokenizerFactory

public SimplePatternSplitTokenizerFactory(Map<String,String> args)

Creates a new SimpleSplitPatternTokenizerFactory

Method Detail

create

public SimplePatternSplitTokenizer create(AttributeFactory factory)

Description copied from class: TokenizerFactory

Creates a TokenStream of the specified input using the given AttributeFactory

Specified by:

create in class TokenizerFactory