org.apache.lucene.analysis.util

Class TokenizerFactory

java.lang.Object

org.apache.lucene.analysis.util.AbstractAnalysisFactory

org.apache.lucene.analysis.util.TokenizerFactory

Direct Known Subclasses:

ArabicLetterTokenizerFactory, ChineseTokenizerFactory, CJKTokenizerFactory, ClassicTokenizerFactory, EdgeNGramTokenizerFactory, KeywordTokenizerFactory, LetterTokenizerFactory, LowerCaseTokenizerFactory, NGramTokenizerFactory, PathHierarchyTokenizerFactory, PatternTokenizerFactory, RussianLetterTokenizerFactory, StandardTokenizerFactory, UAX29URLEmailTokenizerFactory, WhitespaceTokenizerFactory, WikipediaTokenizerFactory

public abstract class TokenizerFactory
extends AbstractAnalysisFactory

Abstract parent class for analysis factories that create Tokenizer instances.

Field Summary

Fields inherited from class org.apache.lucene.analysis.util.AbstractAnalysisFactory

LUCENE_MATCH_VERSION_PARAM, luceneMatchVersion

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	TokenizerFactory(Map<String,String> args) Initialize this factory via a set of key-value pairs.

Method Summary

Methods

Modifier and Type	Method and Description
static Set<String>	availableTokenizers() returns a list of all available tokenizer names from context classpath
abstract Tokenizer	create(AttributeSource.AttributeFactory factory, Reader input) Creates a TokenStream of the specified input using the given AttributeFactory
Tokenizer	create(Reader input) Creates a TokenStream of the specified input using the default attribute factory.
static TokenizerFactory	forName(String name, Map<String,String> args) looks up a tokenizer by name from context classpath
static Class<? extends TokenizerFactory>	lookupClass(String name) looks up a tokenizer class by name from context classpath
static void	reloadTokenizers(ClassLoader classloader) Reloads the factory list from the given ClassLoader.

Methods inherited from class org.apache.lucene.analysis.util.AbstractAnalysisFactory

assureMatchVersion, get, get, get, get, get, getBoolean, getChar, getClassArg, getFloat, getInt, getLines, getLuceneMatchVersion, getOriginalArgs, getPattern, getSet, getSnowballWordSet, getWordSet, isExplicitLuceneMatchVersion, require, require, require, requireBoolean, requireChar, requireFloat, requireInt, setExplicitLuceneMatchVersion, splitFileNames

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerFactory

protected TokenizerFactory(Map<String,String> args)

Initialize this factory via a set of key-value pairs.

Method Detail

forName

public static TokenizerFactory forName(String name,
                       Map<String,String> args)

looks up a tokenizer by name from context classpath

lookupClass

public static Class<? extends TokenizerFactory> lookupClass(String name)

looks up a tokenizer class by name from context classpath

availableTokenizers

public static Set<String> availableTokenizers()

returns a list of all available tokenizer names from context classpath

reloadTokenizers

public static void reloadTokenizers(ClassLoader classloader)

Reloads the factory list from the given ClassLoader. Changes to the factories are visible after the method ends, all iterators (availableTokenizers(),...) stay consistent.

NOTE: Only new factories are added, existing ones are never removed or replaced.

This method is expensive and should only be called for discovery of new factories on the given classpath/classloader!

create

public final Tokenizer create(Reader input)

Creates a TokenStream of the specified input using the default attribute factory.

create

public abstract Tokenizer create(AttributeSource.AttributeFactory factory,
               Reader input)

Creates a TokenStream of the specified input using the given AttributeFactory