org.apache.lucene.analysis.pattern

Class PatternTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.pattern.PatternTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

public final class PatternTokenizer
extends Tokenizer

This tokenizer uses regex pattern matching to construct distinct tokens for the input stream. It takes two arguments: "pattern" and "group".

• "pattern" is the regular expression.
• "group" says which group to extract into tokens.

group=-1 (the default) is equivalent to "split". In this case, the tokens will be equivalent to the output from (without empty tokens): String.split(java.lang.String)

Using group >= 0 selects the matching group as the token. For example, if you have:

  pattern = \'([^\']+)\'
  group = 0
  input = aaa 'bbb' 'ccc'

the output will be two tokens: 'bbb' and 'ccc' (including the ' marks). With the same input but using group=1, the output would be: bbb and ccc (no ' marks)

NOTE: This Tokenizer does not output tokens that are of zero length.

See Also:

Pattern

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor and Description
PatternTokenizer(AttributeFactory factory, Pattern pattern, int group) creates a new PatternTokenizer returning tokens from group (-1 for split functionality)
PatternTokenizer(Pattern pattern, int group) creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

Method Summary

Modifier and Type	Method and Description
void	close()
void	end()
boolean	incrementToken()
void	reset()

Methods inherited from class org.apache.lucene.analysis.Tokenizer

correctOffset, setReader

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

PatternTokenizer

public PatternTokenizer(Pattern pattern,
                        int group)

creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

PatternTokenizer

public PatternTokenizer(AttributeFactory factory,
                        Pattern pattern,
                        int group)

creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

Method Detail

incrementToken

public boolean incrementToken()

Specified by:

incrementToken in class TokenStream

end

public void end()
         throws IOException

Overrides:

end in class TokenStream

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class Tokenizer

Throws:

IOException

reset

public void reset()
           throws IOException

Overrides:

reset in class Tokenizer

Throws:

IOException