org.apache.lucene.wikipedia.analysis
Class WikipediaTokenizer

java.lang.Object
  extended by org.apache.lucene.util.AttributeSource
      extended by org.apache.lucene.analysis.TokenStream
          extended by org.apache.lucene.analysis.Tokenizer
              extended by org.apache.lucene.wikipedia.analysis.WikipediaTokenizer
All Implemented Interfaces:
Closeable

public final class WikipediaTokenizer
extends org.apache.lucene.analysis.Tokenizer

Extension of StandardTokenizer that is aware of Wikipedia syntax. It is based off of the Wikipedia tutorial available at http://en.wikipedia.org/wiki/Wikipedia:Tutorial, but it may not be complete.

EXPERIMENTAL !!!!!!!!! NOTE: This Tokenizer is considered experimental and the grammar is subject to change in the trunk and in follow up releases.


Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State
 
Field Summary
static int ACRONYM_ID
           
static int ALPHANUM_ID
           
static int APOSTROPHE_ID
           
static String BOLD
           
static int BOLD_ID
           
static String BOLD_ITALICS
           
static int BOLD_ITALICS_ID
           
static int BOTH
          Output the both the untokenized token and the splits
static String CATEGORY
           
static int CATEGORY_ID
           
static String CITATION
           
static int CITATION_ID
           
static int CJ_ID
           
static int COMPANY_ID
           
static int EMAIL_ID
           
static String EXTERNAL_LINK
           
static int EXTERNAL_LINK_ID
           
static String EXTERNAL_LINK_URL
           
static int EXTERNAL_LINK_URL_ID
           
static String HEADING
           
static int HEADING_ID
           
static int HOST_ID
           
static String INTERNAL_LINK
           
static int INTERNAL_LINK_ID
           
static String ITALICS
           
static int ITALICS_ID
           
static int NUM_ID
           
static String SUB_HEADING
           
static int SUB_HEADING_ID
           
static String[] TOKEN_TYPES
          String token types that correspond to token type int constants
static int TOKENS_ONLY
          Only output tokens
static int UNTOKENIZED_ONLY
          Only output untokenized tokens, which are tokens that would normally be split into several tokens
static int UNTOKENIZED_TOKEN_FLAG
          This flag is used to indicate that the produced "Token" would, if TOKENS_ONLY was used, produce multiple tokens.
 
Fields inherited from class org.apache.lucene.analysis.Tokenizer
input
 
Constructor Summary
WikipediaTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, Reader input, int tokenOutput, Set<String> untokenizedTypes)
          Creates a new instance of the WikipediaTokenizer.
WikipediaTokenizer(org.apache.lucene.util.AttributeSource source, Reader input, int tokenOutput, Set<String> untokenizedTypes)
          Creates a new instance of the WikipediaTokenizer.
WikipediaTokenizer(Reader input)
          Creates a new instance of the WikipediaTokenizer.
WikipediaTokenizer(Reader input, int tokenOutput, Set<String> untokenizedTypes)
          Creates a new instance of the WikipediaTokenizer.
 
Method Summary
 void end()
           
 boolean incrementToken()
           
 void reset()
           
 void reset(Reader reader)
           
 
Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset
 
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Field Detail

INTERNAL_LINK

public static final String INTERNAL_LINK
See Also:
Constant Field Values

EXTERNAL_LINK

public static final String EXTERNAL_LINK
See Also:
Constant Field Values

EXTERNAL_LINK_URL

public static final String EXTERNAL_LINK_URL
See Also:
Constant Field Values

CITATION

public static final String CITATION
See Also:
Constant Field Values

CATEGORY

public static final String CATEGORY
See Also:
Constant Field Values

BOLD

public static final String BOLD
See Also:
Constant Field Values

ITALICS

public static final String ITALICS
See Also:
Constant Field Values

BOLD_ITALICS

public static final String BOLD_ITALICS
See Also:
Constant Field Values

HEADING

public static final String HEADING
See Also:
Constant Field Values

SUB_HEADING

public static final String SUB_HEADING
See Also:
Constant Field Values

ALPHANUM_ID

public static final int ALPHANUM_ID
See Also:
Constant Field Values

APOSTROPHE_ID

public static final int APOSTROPHE_ID
See Also:
Constant Field Values

ACRONYM_ID

public static final int ACRONYM_ID
See Also:
Constant Field Values

COMPANY_ID

public static final int COMPANY_ID
See Also:
Constant Field Values

EMAIL_ID

public static final int EMAIL_ID
See Also:
Constant Field Values

HOST_ID

public static final int HOST_ID
See Also:
Constant Field Values

NUM_ID

public static final int NUM_ID
See Also:
Constant Field Values

CJ_ID

public static final int CJ_ID
See Also:
Constant Field Values

INTERNAL_LINK_ID

public static final int INTERNAL_LINK_ID
See Also:
Constant Field Values

EXTERNAL_LINK_ID

public static final int EXTERNAL_LINK_ID
See Also:
Constant Field Values

CITATION_ID

public static final int CITATION_ID
See Also:
Constant Field Values

CATEGORY_ID

public static final int CATEGORY_ID
See Also:
Constant Field Values

BOLD_ID

public static final int BOLD_ID
See Also:
Constant Field Values

ITALICS_ID

public static final int ITALICS_ID
See Also:
Constant Field Values

BOLD_ITALICS_ID

public static final int BOLD_ITALICS_ID
See Also:
Constant Field Values

HEADING_ID

public static final int HEADING_ID
See Also:
Constant Field Values

SUB_HEADING_ID

public static final int SUB_HEADING_ID
See Also:
Constant Field Values

EXTERNAL_LINK_URL_ID

public static final int EXTERNAL_LINK_URL_ID
See Also:
Constant Field Values

TOKEN_TYPES

public static final String[] TOKEN_TYPES
String token types that correspond to token type int constants


TOKENS_ONLY

public static final int TOKENS_ONLY
Only output tokens

See Also:
Constant Field Values

UNTOKENIZED_ONLY

public static final int UNTOKENIZED_ONLY
Only output untokenized tokens, which are tokens that would normally be split into several tokens

See Also:
Constant Field Values

BOTH

public static final int BOTH
Output the both the untokenized token and the splits

See Also:
Constant Field Values

UNTOKENIZED_TOKEN_FLAG

public static final int UNTOKENIZED_TOKEN_FLAG
This flag is used to indicate that the produced "Token" would, if TOKENS_ONLY was used, produce multiple tokens.

See Also:
Constant Field Values
Constructor Detail

WikipediaTokenizer

public WikipediaTokenizer(Reader input)
Creates a new instance of the WikipediaTokenizer. Attaches the input to a newly created JFlex scanner.

Parameters:
input - The Input Reader

WikipediaTokenizer

public WikipediaTokenizer(Reader input,
                          int tokenOutput,
                          Set<String> untokenizedTypes)
Creates a new instance of the WikipediaTokenizer. Attaches the input to a the newly created JFlex scanner.

Parameters:
input - The input
tokenOutput - One of TOKENS_ONLY, UNTOKENIZED_ONLY, BOTH
untokenizedTypes -

WikipediaTokenizer

public WikipediaTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
                          Reader input,
                          int tokenOutput,
                          Set<String> untokenizedTypes)
Creates a new instance of the WikipediaTokenizer. Attaches the input to a the newly created JFlex scanner. Uses the given AttributeSource.AttributeFactory.

Parameters:
input - The input
tokenOutput - One of TOKENS_ONLY, UNTOKENIZED_ONLY, BOTH
untokenizedTypes -

WikipediaTokenizer

public WikipediaTokenizer(org.apache.lucene.util.AttributeSource source,
                          Reader input,
                          int tokenOutput,
                          Set<String> untokenizedTypes)
Creates a new instance of the WikipediaTokenizer. Attaches the input to a the newly created JFlex scanner. Uses the given AttributeSource.

Parameters:
input - The input
tokenOutput - One of TOKENS_ONLY, UNTOKENIZED_ONLY, BOTH
untokenizedTypes -
Method Detail

incrementToken

public final boolean incrementToken()
                             throws IOException
Specified by:
incrementToken in class org.apache.lucene.analysis.TokenStream
Throws:
IOException

reset

public void reset()
           throws IOException
Overrides:
reset in class org.apache.lucene.analysis.TokenStream
Throws:
IOException

reset

public void reset(Reader reader)
           throws IOException
Overrides:
reset in class org.apache.lucene.analysis.Tokenizer
Throws:
IOException

end

public void end()
         throws IOException
Overrides:
end in class org.apache.lucene.analysis.TokenStream
Throws:
IOException


Copyright © 2000-2010 Apache Software Foundation. All Rights Reserved.