|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.lucene.util.AttributeSource org.apache.lucene.analysis.TokenStream org.apache.lucene.analysis.Tokenizer org.apache.lucene.analysis.MockTokenizer
public class MockTokenizer
Tokenizer for testing.
This tokenizer is a replacement for WHITESPACE
, SIMPLE
, and KEYWORD
tokenizers. If you are writing a component such as a TokenFilter, its a great idea to test
it wrapping this tokenizer instead for extra checks. This tokenizer has the following behavior:
setEnableChecks(boolean)
.
Nested Class Summary |
---|
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource |
---|
AttributeSource.AttributeFactory |
Field Summary | |
---|---|
static int |
DEFAULT_MAX_TOKEN_LENGTH
|
static CharacterRunAutomaton |
KEYWORD
Acts Similar to KeywordTokenizer. |
static CharacterRunAutomaton |
SIMPLE
Acts like LetterTokenizer. |
static CharacterRunAutomaton |
WHITESPACE
Acts Similar to WhitespaceTokenizer |
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
---|
input |
Constructor Summary | |
---|---|
MockTokenizer(AttributeSource.AttributeFactory factory,
Reader input)
Calls MockTokenizer(AttributeFactory, Reader, WHITESPACE, true) |
|
MockTokenizer(AttributeSource.AttributeFactory factory,
Reader input,
CharacterRunAutomaton runAutomaton,
boolean lowerCase)
|
|
MockTokenizer(AttributeSource.AttributeFactory factory,
Reader input,
CharacterRunAutomaton runAutomaton,
boolean lowerCase,
int maxTokenLength)
|
|
MockTokenizer(Reader input)
Calls MockTokenizer(Reader, WHITESPACE, true) |
|
MockTokenizer(Reader input,
CharacterRunAutomaton runAutomaton,
boolean lowerCase)
|
|
MockTokenizer(Reader input,
CharacterRunAutomaton runAutomaton,
boolean lowerCase,
int maxTokenLength)
|
Method Summary | |
---|---|
void |
close()
|
void |
end()
|
boolean |
incrementToken()
|
protected boolean |
isTokenChar(int c)
|
protected int |
normalize(int c)
|
protected int |
readChar()
|
protected int |
readCodePoint()
|
void |
reset()
|
void |
setEnableChecks(boolean enableChecks)
Toggle consumer workflow checking: if your test consumes tokenstreams normally you should leave this enabled. |
Methods inherited from class org.apache.lucene.analysis.Tokenizer |
---|
correctOffset, setReader |
Methods inherited from class org.apache.lucene.util.AttributeSource |
---|
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final CharacterRunAutomaton WHITESPACE
public static final CharacterRunAutomaton KEYWORD
public static final CharacterRunAutomaton SIMPLE
public static final int DEFAULT_MAX_TOKEN_LENGTH
Constructor Detail |
---|
public MockTokenizer(AttributeSource.AttributeFactory factory, Reader input, CharacterRunAutomaton runAutomaton, boolean lowerCase, int maxTokenLength)
public MockTokenizer(Reader input, CharacterRunAutomaton runAutomaton, boolean lowerCase, int maxTokenLength)
public MockTokenizer(Reader input, CharacterRunAutomaton runAutomaton, boolean lowerCase)
public MockTokenizer(Reader input)
MockTokenizer(Reader, WHITESPACE, true)
public MockTokenizer(AttributeSource.AttributeFactory factory, Reader input, CharacterRunAutomaton runAutomaton, boolean lowerCase)
public MockTokenizer(AttributeSource.AttributeFactory factory, Reader input)
MockTokenizer(AttributeFactory, Reader, WHITESPACE, true)
Method Detail |
---|
public final boolean incrementToken() throws IOException
incrementToken
in class TokenStream
IOException
protected int readCodePoint() throws IOException
IOException
protected int readChar() throws IOException
IOException
protected boolean isTokenChar(int c)
protected int normalize(int c)
public void reset() throws IOException
reset
in class TokenStream
IOException
public void close() throws IOException
close
in interface Closeable
close
in class Tokenizer
IOException
public void end() throws IOException
end
in class TokenStream
IOException
public void setEnableChecks(boolean enableChecks)
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |