|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.lucene.util.AttributeSource org.apache.lucene.analysis.TokenStream org.apache.lucene.analysis.Tokenizer org.apache.lucene.wikipedia.analysis.WikipediaTokenizer
public class WikipediaTokenizer
Extension of StandardTokenizer that is aware of Wikipedia syntax. It is based off of the Wikipedia tutorial available at http://en.wikipedia.org/wiki/Wikipedia:Tutorial, but it may not be complete.
EXPERIMENTAL !!!!!!!!! NOTE: This Tokenizer is considered experimental and the grammar is subject to change in the trunk and in follow up releases.
Nested Class Summary |
---|
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource |
---|
AttributeSource.AttributeFactory, AttributeSource.State |
Field Summary | |
---|---|
static int |
ACRONYM_ID
|
static int |
ALPHANUM_ID
|
static int |
APOSTROPHE_ID
|
static String |
BOLD
|
static int |
BOLD_ID
|
static String |
BOLD_ITALICS
|
static int |
BOLD_ITALICS_ID
|
static int |
BOTH
Output the both the untokenized token and the splits |
static String |
CATEGORY
|
static int |
CATEGORY_ID
|
static String |
CITATION
|
static int |
CITATION_ID
|
static int |
CJ_ID
|
static int |
COMPANY_ID
|
static int |
EMAIL_ID
|
static String |
EXTERNAL_LINK
|
static int |
EXTERNAL_LINK_ID
|
static String |
EXTERNAL_LINK_URL
|
static int |
EXTERNAL_LINK_URL_ID
|
static String |
HEADING
|
static int |
HEADING_ID
|
static int |
HOST_ID
|
static String |
INTERNAL_LINK
|
static int |
INTERNAL_LINK_ID
|
static String |
ITALICS
|
static int |
ITALICS_ID
|
static int |
NUM_ID
|
static String |
SUB_HEADING
|
static int |
SUB_HEADING_ID
|
static String[] |
TOKEN_TYPES
String token types that correspond to token type int constants |
static String[] |
tokenImage
Deprecated. Please use TOKEN_TYPES instead |
static int |
TOKENS_ONLY
Only output tokens |
static int |
UNTOKENIZED_ONLY
Only output untokenized tokens, which are tokens that would normally be split into several tokens |
static int |
UNTOKENIZED_TOKEN_FLAG
This flag is used to indicate that the produced "Token" would, if TOKENS_ONLY was used, produce multiple tokens. |
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
---|
input |
Constructor Summary | |
---|---|
WikipediaTokenizer(AttributeSource.AttributeFactory factory,
Reader input,
int tokenOutput,
Set untokenizedTypes)
Creates a new instance of the WikipediaTokenizer . |
|
WikipediaTokenizer(AttributeSource source,
Reader input,
int tokenOutput,
Set untokenizedTypes)
Creates a new instance of the WikipediaTokenizer . |
|
WikipediaTokenizer(Reader input)
Creates a new instance of the WikipediaTokenizer . |
|
WikipediaTokenizer(Reader input,
int tokenOutput,
Set untokenizedTypes)
Creates a new instance of the WikipediaTokenizer . |
Method Summary | |
---|---|
void |
end()
This method is called by the consumer after the last token has been consumed, after TokenStream.incrementToken() returned false
(using the new TokenStream API). |
boolean |
incrementToken()
Consumers (i.e., IndexWriter ) use this method to advance the stream to
the next token. |
Token |
next()
Deprecated. Will be removed in Lucene 3.0. This method is final, as it should not be overridden. Delegates to the backwards compatibility layer. |
Token |
next(Token reusableToken)
Deprecated. Will be removed in Lucene 3.0. This method is final, as it should not be overridden. Delegates to the backwards compatibility layer. |
void |
reset()
Resets this stream to the beginning. |
void |
reset(Reader reader)
Expert: Reset the tokenizer to a new reader. |
Methods inherited from class org.apache.lucene.analysis.Tokenizer |
---|
close, correctOffset |
Methods inherited from class org.apache.lucene.analysis.TokenStream |
---|
getOnlyUseNewAPI, setOnlyUseNewAPI |
Methods inherited from class org.apache.lucene.util.AttributeSource |
---|
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
Field Detail |
---|
public static final String INTERNAL_LINK
public static final String EXTERNAL_LINK
public static final String EXTERNAL_LINK_URL
public static final String CITATION
public static final String CATEGORY
public static final String BOLD
public static final String ITALICS
public static final String BOLD_ITALICS
public static final String HEADING
public static final String SUB_HEADING
public static final int ALPHANUM_ID
public static final int APOSTROPHE_ID
public static final int ACRONYM_ID
public static final int COMPANY_ID
public static final int EMAIL_ID
public static final int HOST_ID
public static final int NUM_ID
public static final int CJ_ID
public static final int INTERNAL_LINK_ID
public static final int EXTERNAL_LINK_ID
public static final int CITATION_ID
public static final int CATEGORY_ID
public static final int BOLD_ID
public static final int ITALICS_ID
public static final int BOLD_ITALICS_ID
public static final int HEADING_ID
public static final int SUB_HEADING_ID
public static final int EXTERNAL_LINK_URL_ID
public static final String[] TOKEN_TYPES
public static final String[] tokenImage
TOKEN_TYPES
insteadpublic static final int TOKENS_ONLY
public static final int UNTOKENIZED_ONLY
public static final int BOTH
public static final int UNTOKENIZED_TOKEN_FLAG
TOKENS_ONLY
was used, produce multiple tokens.
Constructor Detail |
---|
public WikipediaTokenizer(Reader input)
WikipediaTokenizer
. Attaches the
input
to a newly created JFlex scanner.
input
- The Input Readerpublic WikipediaTokenizer(Reader input, int tokenOutput, Set untokenizedTypes)
WikipediaTokenizer
. Attaches the
input
to a the newly created JFlex scanner.
input
- The inputtokenOutput
- One of TOKENS_ONLY
, UNTOKENIZED_ONLY
, BOTH
untokenizedTypes
- public WikipediaTokenizer(AttributeSource.AttributeFactory factory, Reader input, int tokenOutput, Set untokenizedTypes)
WikipediaTokenizer
. Attaches the
input
to a the newly created JFlex scanner. Uses the given AttributeSource.AttributeFactory
.
input
- The inputtokenOutput
- One of TOKENS_ONLY
, UNTOKENIZED_ONLY
, BOTH
untokenizedTypes
- public WikipediaTokenizer(AttributeSource source, Reader input, int tokenOutput, Set untokenizedTypes)
WikipediaTokenizer
. Attaches the
input
to a the newly created JFlex scanner. Uses the given AttributeSource
.
input
- The inputtokenOutput
- One of TOKENS_ONLY
, UNTOKENIZED_ONLY
, BOTH
untokenizedTypes
- Method Detail |
---|
public final Token next(Token reusableToken) throws IOException
TokenStream
This implicitly defines a "contract" between consumers (callers of this method) and producers (implementations of this method that are the source for tokens):
Token
before calling this method again.Token.clear()
before setting the fields in
it and returning itToken
after it
has been returned: the caller may arbitrarily change it. If the producer
needs to hold onto the Token
for subsequent calls, it must clone()
it before storing it. Note that a TokenFilter
is considered a
consumer.
next
in class TokenStream
reusableToken
- a Token
that may or may not be used to return;
this parameter should never be null (the callee is not required to
check for null before using it, but it is a good idea to assert that
it is not null.)
Token
in the stream or null if end-of-stream was hit
IOException
public final Token next() throws IOException
TokenStream
Token
in the stream, or null at EOS.
next
in class TokenStream
IOException
public final boolean incrementToken() throws IOException
TokenStream
IndexWriter
) use this method to advance the stream to
the next token. Implementing classes must implement this method and update
the appropriate AttributeImpl
s with the attributes of the next
token.
The producer must make no assumptions about the attributes after the method
has been returned: the caller may arbitrarily change it. If the producer
needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.
This method is called for every token of a document, so an efficient
implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
and AttributeSource.getAttribute(Class)
or downcasts,
references to all AttributeImpl
s that this stream uses should be
retrieved during instantiation.
To ensure that filters and consumers know which attributes are available,
the attributes must be added during instantiation. Filters and consumers
are not required to check for availability of attributes in
TokenStream.incrementToken()
.
incrementToken
in class TokenStream
Note that this method will be defined abstract in Lucene 3.0.
IOException
public void reset() throws IOException
TokenStream
TokenStream.reset()
is not needed for
the standard indexing process. However, if the tokens of a
TokenStream
are intended to be consumed more than once, it is
necessary to implement TokenStream.reset()
. Note that if your TokenStream
caches tokens and feeds them back again after a reset, it is imperative
that you clone the tokens when you store them away (on the first pass) as
well as when you return them (on future passes after TokenStream.reset()
).
reset
in class TokenStream
IOException
public void reset(Reader reader) throws IOException
Tokenizer
reset
in class Tokenizer
IOException
public void end() throws IOException
TokenStream
TokenStream.incrementToken()
returned false
(using the new TokenStream
API). Streams implementing the old API
should upgrade to use this feature.
This method can be used to perform any end-of-stream operations, such as
setting the final offset of a stream. The final offset of a stream might
differ from the offset of the last token eg in case one or more whitespaces
followed after the last token, but a WhitespaceTokenizer
was used.
end
in class TokenStream
IOException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |