org.apache.lucene.analysis.compound
Class CompoundWordTokenFilterBase

java.lang.Object
  extended by org.apache.lucene.util.AttributeSource
      extended by org.apache.lucene.analysis.TokenStream
          extended by org.apache.lucene.analysis.TokenFilter
              extended by org.apache.lucene.analysis.compound.CompoundWordTokenFilterBase
All Implemented Interfaces:
Closeable
Direct Known Subclasses:
DictionaryCompoundWordTokenFilter, HyphenationCompoundWordTokenFilter

public abstract class CompoundWordTokenFilterBase
extends TokenFilter

Base class for decomposition token filters.

You must specify the required Version compatibility when creating CompoundWordTokenFilterBase:


Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.AttributeFactory, AttributeSource.State
 
Field Summary
static int DEFAULT_MAX_SUBWORD_SIZE
          The default for maximal length of subwords that get propagated to the output of this filter
static int DEFAULT_MIN_SUBWORD_SIZE
          The default for minimal length of subwords that get propagated to the output of this filter
static int DEFAULT_MIN_WORD_SIZE
          The default for minimal word length that gets decomposed
protected  CharArraySet dictionary
           
protected  int maxSubwordSize
           
protected  int minSubwordSize
           
protected  int minWordSize
           
protected  boolean onlyLongestMatch
           
protected  LinkedList<Token> tokens
           
 
Fields inherited from class org.apache.lucene.analysis.TokenFilter
input
 
Constructor Summary
protected CompoundWordTokenFilterBase(TokenStream input, Set<?> dictionary)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set) instead
protected CompoundWordTokenFilterBase(TokenStream input, Set<?> dictionary, boolean onlyLongestMatch)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set, boolean) instead
protected CompoundWordTokenFilterBase(TokenStream input, Set<?> dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set, int, int, int, boolean) instead
protected CompoundWordTokenFilterBase(TokenStream input, String[] dictionary)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[]) instead
protected CompoundWordTokenFilterBase(TokenStream input, String[] dictionary, boolean onlyLongestMatch)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[], boolean) instead
protected CompoundWordTokenFilterBase(TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[], int, int, int, boolean) instead
protected CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, Set<?> dictionary)
           
protected CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, Set<?> dictionary, boolean onlyLongestMatch)
           
protected CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, Set<?> dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
           
protected CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, String[] dictionary)
           
protected CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, String[] dictionary, boolean onlyLongestMatch)
           
protected CompoundWordTokenFilterBase(Version matchVersion, TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
           
 
Method Summary
protected static void addAllLowerCase(CharArraySet target, Collection<?> col)
           
protected  Token createToken(int offset, int length, Token prototype)
           
protected  void decompose(Token token)
           
protected abstract  void decomposeInternal(Token token)
           
 boolean incrementToken()
          Consumers (i.e., IndexWriter) use this method to advance the stream to the next token.
static Set<?> makeDictionary(String[] dictionary)
          Create a set of words from an array The resulting Set does case insensitive matching TODO We should look for a faster dictionary lookup approach.
static Set<?> makeDictionary(Version matchVersion, String[] dictionary)
           
protected static char[] makeLowerCaseCopy(char[] buffer)
           
 void reset()
          Reset the filter as well as the input TokenStream.
 
Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end
 
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Field Detail

DEFAULT_MIN_WORD_SIZE

public static final int DEFAULT_MIN_WORD_SIZE
The default for minimal word length that gets decomposed

See Also:
Constant Field Values

DEFAULT_MIN_SUBWORD_SIZE

public static final int DEFAULT_MIN_SUBWORD_SIZE
The default for minimal length of subwords that get propagated to the output of this filter

See Also:
Constant Field Values

DEFAULT_MAX_SUBWORD_SIZE

public static final int DEFAULT_MAX_SUBWORD_SIZE
The default for maximal length of subwords that get propagated to the output of this filter

See Also:
Constant Field Values

dictionary

protected final CharArraySet dictionary

tokens

protected final LinkedList<Token> tokens

minWordSize

protected final int minWordSize

minSubwordSize

protected final int minSubwordSize

maxSubwordSize

protected final int maxSubwordSize

onlyLongestMatch

protected final boolean onlyLongestMatch
Constructor Detail

CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(TokenStream input,
                                                 String[] dictionary,
                                                 int minWordSize,
                                                 int minSubwordSize,
                                                 int maxSubwordSize,
                                                 boolean onlyLongestMatch)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[], int, int, int, boolean) instead


CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(TokenStream input,
                                                 String[] dictionary,
                                                 boolean onlyLongestMatch)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[], boolean) instead


CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(TokenStream input,
                                                 Set<?> dictionary,
                                                 boolean onlyLongestMatch)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set, boolean) instead


CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(TokenStream input,
                                                 String[] dictionary)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[]) instead


CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(TokenStream input,
                                                 Set<?> dictionary)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set) instead


CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(TokenStream input,
                                                 Set<?> dictionary,
                                                 int minWordSize,
                                                 int minSubwordSize,
                                                 int maxSubwordSize,
                                                 boolean onlyLongestMatch)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set, int, int, int, boolean) instead


CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(Version matchVersion,
                                      TokenStream input,
                                      String[] dictionary,
                                      int minWordSize,
                                      int minSubwordSize,
                                      int maxSubwordSize,
                                      boolean onlyLongestMatch)

CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(Version matchVersion,
                                      TokenStream input,
                                      String[] dictionary,
                                      boolean onlyLongestMatch)

CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(Version matchVersion,
                                      TokenStream input,
                                      Set<?> dictionary,
                                      boolean onlyLongestMatch)

CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(Version matchVersion,
                                      TokenStream input,
                                      String[] dictionary)

CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(Version matchVersion,
                                      TokenStream input,
                                      Set<?> dictionary)

CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(Version matchVersion,
                                      TokenStream input,
                                      Set<?> dictionary,
                                      int minWordSize,
                                      int minSubwordSize,
                                      int maxSubwordSize,
                                      boolean onlyLongestMatch)
Method Detail

makeDictionary

public static final Set<?> makeDictionary(String[] dictionary)
Create a set of words from an array The resulting Set does case insensitive matching TODO We should look for a faster dictionary lookup approach.

Parameters:
dictionary -
Returns:
Set of lowercased terms

makeDictionary

public static final Set<?> makeDictionary(Version matchVersion,
                                          String[] dictionary)

incrementToken

public final boolean incrementToken()
                             throws IOException
Description copied from class: TokenStream
Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate AttributeImpls with the attributes of the next token.

The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use AttributeSource.captureState() to create a copy of the current attribute state.

This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to AttributeSource.addAttribute(Class) and AttributeSource.getAttribute(Class), references to all AttributeImpls that this stream uses should be retrieved during instantiation.

To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in TokenStream.incrementToken().

Specified by:
incrementToken in class TokenStream
Returns:
false for end of stream; true otherwise
Throws:
IOException

addAllLowerCase

protected static final void addAllLowerCase(CharArraySet target,
                                            Collection<?> col)

makeLowerCaseCopy

protected static char[] makeLowerCaseCopy(char[] buffer)

createToken

protected final Token createToken(int offset,
                                  int length,
                                  Token prototype)

decompose

protected void decompose(Token token)

decomposeInternal

protected abstract void decomposeInternal(Token token)

reset

public void reset()
           throws IOException
Description copied from class: TokenFilter
Reset the filter as well as the input TokenStream.

Overrides:
reset in class TokenFilter
Throws:
IOException


Copyright © 2000-2011 Apache Software Foundation. All Rights Reserved.