Package org.apache.lucene.tests.analysis

Class MockTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.tests.analysis.MockTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

public class MockTokenizer
extends Tokenizer

Tokenizer for testing.

This tokenizer is a replacement for WHITESPACE, SIMPLE, and KEYWORD tokenizers. If you are writing a component such as a TokenFilter, it's a great idea to test it wrapping this tokenizer instead for extra checks. This tokenizer has the following behavior:

• An internal state-machine is used for checking consumer consistency. These checks can be disabled with setEnableChecks(boolean).
• For convenience, optionally lowercases terms that it outputs.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_MAX_TOKEN_LENGTH
static final CharacterRunAutomaton	KEYWORD	Acts Similar to KeywordTokenizer.
static final CharacterRunAutomaton	SIMPLE	Acts like LetterTokenizer.
static final CharacterRunAutomaton	WHITESPACE	Acts Similar to WhitespaceTokenizer

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor	Description
MockTokenizer()	Calls MockTokenizer(Reader, WHITESPACE, true)
MockTokenizer(AttributeFactory factory)	Calls MockTokenizer(AttributeFactory, Reader, WHITESPACE, true)
MockTokenizer(AttributeFactory factory, CharacterRunAutomaton runAutomaton, boolean lowerCase)
MockTokenizer(AttributeFactory factory, CharacterRunAutomaton runAutomaton, boolean lowerCase, int maxTokenLength)
MockTokenizer(CharacterRunAutomaton runAutomaton, boolean lowerCase)
MockTokenizer(CharacterRunAutomaton runAutomaton, boolean lowerCase, int maxTokenLength)

Method Summary

Modifier and Type	Method	Description
void	close()
void	end()
final boolean	incrementToken()
protected boolean	isTokenChar(int c)
protected int	normalize(int c)
protected int	readChar()
protected int	readCodePoint()
void	reset()
void	setEnableChecks(boolean enableChecks)	Toggle consumer workflow checking: if your test consumes tokenstreams normally you should leave this enabled.
protected void	setReaderTestPoint()

Methods inherited from class org.apache.lucene.analysis.Tokenizer

correctOffset, setReader

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

WHITESPACE

public static final CharacterRunAutomaton WHITESPACE

Acts Similar to WhitespaceTokenizer

KEYWORD

public static final CharacterRunAutomaton KEYWORD

Acts Similar to KeywordTokenizer. TODO: Keyword returns an "empty" token for an empty reader...

SIMPLE

public static final CharacterRunAutomaton SIMPLE

Acts like LetterTokenizer.

DEFAULT_MAX_TOKEN_LENGTH

public static final int DEFAULT_MAX_TOKEN_LENGTH

See Also:

• Constant Field Values

Constructor Details

MockTokenizer

public MockTokenizer(AttributeFactory factory,
 CharacterRunAutomaton runAutomaton,
 boolean lowerCase,
 int maxTokenLength)

MockTokenizer

public MockTokenizer(CharacterRunAutomaton runAutomaton,
 boolean lowerCase,
 int maxTokenLength)

MockTokenizer

public MockTokenizer(CharacterRunAutomaton runAutomaton,
 boolean lowerCase)

MockTokenizer

public MockTokenizer()

Calls MockTokenizer(Reader, WHITESPACE, true)

MockTokenizer

public MockTokenizer(AttributeFactory factory,
 CharacterRunAutomaton runAutomaton,
 boolean lowerCase)

MockTokenizer

public MockTokenizer(AttributeFactory factory)

Calls MockTokenizer(AttributeFactory, Reader, WHITESPACE, true)

Method Details

incrementToken

public final boolean incrementToken()
                             throws IOException

Specified by:

incrementToken in class TokenStream

Throws:

IOException

readCodePoint

protected int readCodePoint()
                     throws IOException

Throws:

IOException

readChar

protected int readChar()
                throws IOException

Throws:

IOException

isTokenChar

protected boolean isTokenChar(int c)

normalize

protected int normalize(int c)

reset

public void reset()
           throws IOException

Overrides:

reset in class Tokenizer

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Overrides:

close in class Tokenizer

Throws:

IOException

setReaderTestPoint

protected void setReaderTestPoint()

Overrides:

setReaderTestPoint in class Tokenizer

end

public void end()
         throws IOException

Overrides:

end in class TokenStream

Throws:

IOException

setEnableChecks

public void setEnableChecks(boolean enableChecks)

Toggle consumer workflow checking: if your test consumes tokenstreams normally you should leave this enabled.