org.apache.lucene.analysis.compound
Class CompoundWordTokenFilterBase

java.lang.Object
  extended by org.apache.lucene.util.AttributeSource
      extended by org.apache.lucene.analysis.TokenStream
          extended by org.apache.lucene.analysis.TokenFilter
              extended by org.apache.lucene.analysis.compound.CompoundWordTokenFilterBase
All Implemented Interfaces:
Closeable
Direct Known Subclasses:
DictionaryCompoundWordTokenFilter, HyphenationCompoundWordTokenFilter

public abstract class CompoundWordTokenFilterBase
extends org.apache.lucene.analysis.TokenFilter

Base class for decomposition token filters.

You must specify the required Version compatibility when creating CompoundWordTokenFilterBase:

If you pass in a CharArraySet as dictionary, it should be case-insensitive unless it contains only lowercased entries and you have LowerCaseFilter before this filter in your analysis chain. For optional performance (as this filter does lots of lookups to the dictionary, you should use the latter analysis chain/CharArraySet). Be aware: If you supply arbitrary Sets to the ctors or String[] dictionaries, they will be automatically transformed to case-insensitive!


Nested Class Summary
protected  class CompoundWordTokenFilterBase.CompoundToken
          Helper class to hold decompounded token information
 
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State
 
Field Summary
static int DEFAULT_MAX_SUBWORD_SIZE
          The default for maximal length of subwords that get propagated to the output of this filter
static int DEFAULT_MIN_SUBWORD_SIZE
          The default for minimal length of subwords that get propagated to the output of this filter
static int DEFAULT_MIN_WORD_SIZE
          The default for minimal word length that gets decomposed
protected  org.apache.lucene.analysis.CharArraySet dictionary
           
protected  int maxSubwordSize
           
protected  int minSubwordSize
           
protected  int minWordSize
           
protected  org.apache.lucene.analysis.tokenattributes.OffsetAttribute offsetAtt
           
protected  boolean onlyLongestMatch
           
protected  org.apache.lucene.analysis.tokenattributes.CharTermAttribute termAtt
           
protected  LinkedList<CompoundWordTokenFilterBase.CompoundToken> tokens
           
 
Fields inherited from class org.apache.lucene.analysis.TokenFilter
input
 
Constructor Summary
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input, Set<?> dictionary)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set) instead
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input, Set<?> dictionary, boolean onlyLongestMatch)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set, boolean) instead
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input, Set<?> dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set, int, int, int, boolean) instead
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input, String[] dictionary)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[]) instead
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input, String[] dictionary, boolean onlyLongestMatch)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[], boolean) instead
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
          Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[], int, int, int, boolean) instead
protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion, org.apache.lucene.analysis.TokenStream input, Set<?> dictionary)
           
protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion, org.apache.lucene.analysis.TokenStream input, Set<?> dictionary, boolean onlyLongestMatch)
           
protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion, org.apache.lucene.analysis.TokenStream input, Set<?> dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
           
protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion, org.apache.lucene.analysis.TokenStream input, String[] dictionary)
           
protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion, org.apache.lucene.analysis.TokenStream input, String[] dictionary, boolean onlyLongestMatch)
           
protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion, org.apache.lucene.analysis.TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
           
 
Method Summary
protected abstract  void decompose()
          Decomposes the current termAtt and places CompoundWordTokenFilterBase.CompoundToken instances in the tokens list.
 boolean incrementToken()
           
static org.apache.lucene.analysis.CharArraySet makeDictionary(org.apache.lucene.util.Version matchVersion, String[] dictionary)
          Deprecated. Only available for backwards compatibility.
 void reset()
           
 
Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end
 
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Field Detail

DEFAULT_MIN_WORD_SIZE

public static final int DEFAULT_MIN_WORD_SIZE
The default for minimal word length that gets decomposed

See Also:
Constant Field Values

DEFAULT_MIN_SUBWORD_SIZE

public static final int DEFAULT_MIN_SUBWORD_SIZE
The default for minimal length of subwords that get propagated to the output of this filter

See Also:
Constant Field Values

DEFAULT_MAX_SUBWORD_SIZE

public static final int DEFAULT_MAX_SUBWORD_SIZE
The default for maximal length of subwords that get propagated to the output of this filter

See Also:
Constant Field Values

dictionary

protected final org.apache.lucene.analysis.CharArraySet dictionary

tokens

protected final LinkedList<CompoundWordTokenFilterBase.CompoundToken> tokens

minWordSize

protected final int minWordSize

minSubwordSize

protected final int minSubwordSize

maxSubwordSize

protected final int maxSubwordSize

onlyLongestMatch

protected final boolean onlyLongestMatch

termAtt

protected final org.apache.lucene.analysis.tokenattributes.CharTermAttribute termAtt

offsetAtt

protected final org.apache.lucene.analysis.tokenattributes.OffsetAttribute offsetAtt
Constructor Detail

CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input,
                                                 String[] dictionary,
                                                 int minWordSize,
                                                 int minSubwordSize,
                                                 int maxSubwordSize,
                                                 boolean onlyLongestMatch)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[], int, int, int, boolean) instead


CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input,
                                                 String[] dictionary,
                                                 boolean onlyLongestMatch)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[], boolean) instead


CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input,
                                                 Set<?> dictionary,
                                                 boolean onlyLongestMatch)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set, boolean) instead


CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input,
                                                 String[] dictionary)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, String[]) instead


CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input,
                                                 Set<?> dictionary)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set) instead


CompoundWordTokenFilterBase

@Deprecated
protected CompoundWordTokenFilterBase(org.apache.lucene.analysis.TokenStream input,
                                                 Set<?> dictionary,
                                                 int minWordSize,
                                                 int minSubwordSize,
                                                 int maxSubwordSize,
                                                 boolean onlyLongestMatch)
Deprecated. use CompoundWordTokenFilterBase(Version, TokenStream, Set, int, int, int, boolean) instead


CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion,
                                      org.apache.lucene.analysis.TokenStream input,
                                      String[] dictionary,
                                      int minWordSize,
                                      int minSubwordSize,
                                      int maxSubwordSize,
                                      boolean onlyLongestMatch)

CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion,
                                      org.apache.lucene.analysis.TokenStream input,
                                      String[] dictionary,
                                      boolean onlyLongestMatch)

CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion,
                                      org.apache.lucene.analysis.TokenStream input,
                                      Set<?> dictionary,
                                      boolean onlyLongestMatch)

CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion,
                                      org.apache.lucene.analysis.TokenStream input,
                                      String[] dictionary)

CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion,
                                      org.apache.lucene.analysis.TokenStream input,
                                      Set<?> dictionary)

CompoundWordTokenFilterBase

protected CompoundWordTokenFilterBase(org.apache.lucene.util.Version matchVersion,
                                      org.apache.lucene.analysis.TokenStream input,
                                      Set<?> dictionary,
                                      int minWordSize,
                                      int minSubwordSize,
                                      int maxSubwordSize,
                                      boolean onlyLongestMatch)
Method Detail

makeDictionary

@Deprecated
public static org.apache.lucene.analysis.CharArraySet makeDictionary(org.apache.lucene.util.Version matchVersion,
                                                                                String[] dictionary)
Deprecated. Only available for backwards compatibility.


incrementToken

public final boolean incrementToken()
                             throws IOException
Specified by:
incrementToken in class org.apache.lucene.analysis.TokenStream
Throws:
IOException

decompose

protected abstract void decompose()
Decomposes the current termAtt and places CompoundWordTokenFilterBase.CompoundToken instances in the tokens list. The original token may not be placed in the list, as it is automatically passed through this filter.


reset

public void reset()
           throws IOException
Overrides:
reset in class org.apache.lucene.analysis.TokenFilter
Throws:
IOException


Copyright © 2000-2011 Apache Software Foundation. All Rights Reserved.