|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.lucene.util.AttributeSource org.apache.lucene.analysis.TokenStream org.apache.lucene.analysis.Tokenizer org.apache.lucene.analysis.wikipedia.WikipediaTokenizer
public final class WikipediaTokenizer
Extension of StandardTokenizer that is aware of Wikipedia syntax. It is based off of the Wikipedia tutorial available at http://en.wikipedia.org/wiki/Wikipedia:Tutorial, but it may not be complete.
Nested Class Summary |
---|
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource |
---|
AttributeSource.AttributeFactory, AttributeSource.State |
Field Summary | |
---|---|
static int |
ACRONYM_ID
|
static int |
ALPHANUM_ID
|
static int |
APOSTROPHE_ID
|
static String |
BOLD
|
static int |
BOLD_ID
|
static String |
BOLD_ITALICS
|
static int |
BOLD_ITALICS_ID
|
static int |
BOTH
Output the both the untokenized token and the splits |
static String |
CATEGORY
|
static int |
CATEGORY_ID
|
static String |
CITATION
|
static int |
CITATION_ID
|
static int |
CJ_ID
|
static int |
COMPANY_ID
|
static int |
EMAIL_ID
|
static String |
EXTERNAL_LINK
|
static int |
EXTERNAL_LINK_ID
|
static String |
EXTERNAL_LINK_URL
|
static int |
EXTERNAL_LINK_URL_ID
|
static String |
HEADING
|
static int |
HEADING_ID
|
static int |
HOST_ID
|
static String |
INTERNAL_LINK
|
static int |
INTERNAL_LINK_ID
|
static String |
ITALICS
|
static int |
ITALICS_ID
|
static int |
NUM_ID
|
static String |
SUB_HEADING
|
static int |
SUB_HEADING_ID
|
static String[] |
TOKEN_TYPES
String token types that correspond to token type int constants |
static int |
TOKENS_ONLY
Only output tokens |
static int |
UNTOKENIZED_ONLY
Only output untokenized tokens, which are tokens that would normally be split into several tokens |
static int |
UNTOKENIZED_TOKEN_FLAG
This flag is used to indicate that the produced "Token" would, if TOKENS_ONLY was used, produce multiple tokens. |
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
---|
input |
Constructor Summary | |
---|---|
WikipediaTokenizer(AttributeSource.AttributeFactory factory,
Reader input,
int tokenOutput,
Set<String> untokenizedTypes)
Creates a new instance of the WikipediaTokenizer . |
|
WikipediaTokenizer(AttributeSource source,
Reader input,
int tokenOutput,
Set<String> untokenizedTypes)
Creates a new instance of the WikipediaTokenizer . |
|
WikipediaTokenizer(Reader input)
Creates a new instance of the WikipediaTokenizer . |
|
WikipediaTokenizer(Reader input,
int tokenOutput,
Set<String> untokenizedTypes)
Creates a new instance of the WikipediaTokenizer . |
Method Summary | |
---|---|
void |
end()
|
boolean |
incrementToken()
|
void |
reset()
|
Methods inherited from class org.apache.lucene.analysis.Tokenizer |
---|
close, correctOffset, setReader |
Methods inherited from class org.apache.lucene.util.AttributeSource |
---|
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final String INTERNAL_LINK
public static final String EXTERNAL_LINK
public static final String EXTERNAL_LINK_URL
public static final String CITATION
public static final String CATEGORY
public static final String BOLD
public static final String ITALICS
public static final String BOLD_ITALICS
public static final String HEADING
public static final String SUB_HEADING
public static final int ALPHANUM_ID
public static final int APOSTROPHE_ID
public static final int ACRONYM_ID
public static final int COMPANY_ID
public static final int EMAIL_ID
public static final int HOST_ID
public static final int NUM_ID
public static final int CJ_ID
public static final int INTERNAL_LINK_ID
public static final int EXTERNAL_LINK_ID
public static final int CITATION_ID
public static final int CATEGORY_ID
public static final int BOLD_ID
public static final int ITALICS_ID
public static final int BOLD_ITALICS_ID
public static final int HEADING_ID
public static final int SUB_HEADING_ID
public static final int EXTERNAL_LINK_URL_ID
public static final String[] TOKEN_TYPES
public static final int TOKENS_ONLY
public static final int UNTOKENIZED_ONLY
public static final int BOTH
public static final int UNTOKENIZED_TOKEN_FLAG
TOKENS_ONLY
was used, produce multiple tokens.
Constructor Detail |
---|
public WikipediaTokenizer(Reader input)
WikipediaTokenizer
. Attaches the
input
to a newly created JFlex scanner.
input
- The Input Readerpublic WikipediaTokenizer(Reader input, int tokenOutput, Set<String> untokenizedTypes)
WikipediaTokenizer
. Attaches the
input
to a the newly created JFlex scanner.
input
- The inputtokenOutput
- One of TOKENS_ONLY
, UNTOKENIZED_ONLY
, BOTH
public WikipediaTokenizer(AttributeSource.AttributeFactory factory, Reader input, int tokenOutput, Set<String> untokenizedTypes)
WikipediaTokenizer
. Attaches the
input
to a the newly created JFlex scanner. Uses the given AttributeSource.AttributeFactory
.
input
- The inputtokenOutput
- One of TOKENS_ONLY
, UNTOKENIZED_ONLY
, BOTH
public WikipediaTokenizer(AttributeSource source, Reader input, int tokenOutput, Set<String> untokenizedTypes)
WikipediaTokenizer
. Attaches the
input
to a the newly created JFlex scanner. Uses the given AttributeSource
.
input
- The inputtokenOutput
- One of TOKENS_ONLY
, UNTOKENIZED_ONLY
, BOTH
Method Detail |
---|
public final boolean incrementToken() throws IOException
incrementToken
in class TokenStream
IOException
public void reset() throws IOException
reset
in class TokenStream
IOException
public void end()
end
in class TokenStream
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |