public class PatternTokenizerFactory extends TokenizerFactory
PatternTokenizer
.
This tokenizer uses regex pattern matching to construct distinct tokens
for the input stream. It takes two arguments: "pattern" and "group".
group=-1 (the default) is equivalent to "split". In this case, the tokens will
be equivalent to the output from (without empty tokens):
String.split(java.lang.String)
Using group >= 0 selects the matching group as the token. For example, if you have:
pattern = \'([^\']+)\' group = 0 input = aaa 'bbb' 'ccc'the output will be two tokens: 'bbb' and 'ccc' (including the ' marks). With the same input but using group=1, the output would be: bbb and ccc (no ' marks)
NOTE: This Tokenizer does not output tokens that are of zero length.
<fieldType name="text_ptn" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizer class="solr.PatternTokenizerFactory" pattern="\'([^\']+)\'" group="1"/> </analyzer> </fieldType>
PatternTokenizer
Modifier and Type | Field and Description |
---|---|
protected int |
group |
static String |
GROUP |
protected Pattern |
pattern |
static String |
PATTERN |
args, luceneMatchVersion
Constructor and Description |
---|
PatternTokenizerFactory() |
Modifier and Type | Method and Description |
---|---|
Tokenizer |
create(Reader in)
Split the input using configured pattern
|
void |
init(Map<String,String> args)
Require a configured pattern
|
availableTokenizers, forName, lookupClass, reloadTokenizers
assureMatchVersion, getArgs, getBoolean, getBoolean, getInt, getInt, getInt, getLines, getLuceneMatchVersion, getPattern, getSnowballWordSet, getWordSet, setLuceneMatchVersion, splitFileNames
public static final String PATTERN
public static final String GROUP
protected Pattern pattern
protected int group
public void init(Map<String,String> args)
init
in class AbstractAnalysisFactory
public Tokenizer create(Reader in)
create
in class TokenizerFactory
Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.