org.apache.lucene.search.suggest.analyzing

Class AnalyzingInfixSuggester

java.lang.Object

org.apache.lucene.search.suggest.Lookup

org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester

All Implemented Interfaces:

Closeable, AutoCloseable, Accountable

Direct Known Subclasses:

BlendedInfixSuggester

public class AnalyzingInfixSuggester
extends Lookup
implements Closeable

Analyzes the input text and then suggests matches based on prefix matches to any tokens in the indexed text. This also highlights the tokens that match.

This suggester supports payloads. Matches are sorted only by the suggest weight; it would be nice to support blended score + weight sort in the future. This means this suggester best applies when there is a strong a-priori ranking of all the suggestions.

This suggester supports contexts, including arbitrary binary terms.

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.search.suggest.Lookup

Lookup.LookupPriorityQueue, Lookup.LookupResult

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	CONTEXTS_FIELD_NAME Field name used for the indexed context, as a StringField and a SortedSetDVField, for filtering.
static boolean	DEFAULT_ALL_TERMS_REQUIRED Default boolean clause option for multiple terms matching (all terms required).
static boolean	DEFAULT_HIGHLIGHT Default higlighting option.
static int	DEFAULT_MIN_PREFIX_CHARS Default minimum number of leading characters before PrefixQuery is used (4).
protected static String	EXACT_TEXT_FIELD_NAME Field name used for the indexed text, as a StringField, for exact lookup.
protected Analyzer	indexAnalyzer Analyzer used at index time
protected Analyzer	queryAnalyzer Analyzer used at search time
protected SearcherManager	searcherMgr IndexSearcher used for lookups.
protected static String	TEXT_FIELD_NAME Field name used for the indexed text.

Fields inherited from class org.apache.lucene.search.suggest.Lookup

CHARSEQUENCE_COMPARATOR

Constructor Summary

Constructors

Constructor and Description
AnalyzingInfixSuggester(Directory dir, Analyzer analyzer) Create a new instance, loading from a previously built AnalyzingInfixSuggester directory, if it exists.
AnalyzingInfixSuggester(Directory dir, Analyzer indexAnalyzer, Analyzer queryAnalyzer, int minPrefixChars, boolean commitOnBuild) Create a new instance, loading from a previously built AnalyzingInfixSuggester directory, if it exists.
AnalyzingInfixSuggester(Directory dir, Analyzer indexAnalyzer, Analyzer queryAnalyzer, int minPrefixChars, boolean commitOnBuild, boolean allTermsRequired, boolean highlight) Create a new instance, loading from a previously built AnalyzingInfixSuggester directory, if it exists.

Method Summary

Modifier and Type	Method and Description
void	add(BytesRef text, Set<BytesRef> contexts, long weight, BytesRef payload) Adds a new suggestion.
void	addContextToQuery(BooleanQuery.Builder query, BytesRef context, BooleanClause.Occur clause) This method is handy as we do not need access to internal fields such as CONTEXTS_FIELD_NAME in order to build queries However, here may not be its best location.
protected void	addNonMatch(StringBuilder sb, String text) Called while highlighting a single result, to append a non-matching chunk of text from the suggestion to the provided fragments list.
protected void	addPrefixMatch(StringBuilder sb, String surface, String analyzed, String prefixToken) Called while highlighting a single result, to append a matched prefix token, to the provided fragments list.
protected void	addWholeMatch(StringBuilder sb, String surface, String analyzed) Called while highlighting a single result, to append the whole matched token to the provided fragments list.
void	build(InputIterator iter) Builds up a new internal Lookup representation based on the given InputIterator.
void	close()
void	commit() Commits all pending changes made to this suggester to disk.
protected List<Lookup.LookupResult>	createResults(IndexSearcher searcher, TopFieldDocs hits, int num, CharSequence charSequence, boolean doHighlight, Set<String> matchedTokens, String prefixToken) Create the results based on the search hits.
protected Query	finishQuery(BooleanQuery.Builder in, boolean allTermsRequired) Subclass can override this to tweak the Query before searching.
Collection<Accountable>	getChildResources()
long	getCount() Get the number of entries the lookup was built with
protected Directory	getDirectory(Path path) Subclass can override to choose a specific Directory implementation.
protected IndexWriterConfig	getIndexWriterConfig(Analyzer indexAnalyzer, IndexWriterConfig.OpenMode openMode) Override this to customize index settings, e.g.
protected Query	getLastTokenQuery(String token) This is called if the last token isn't ended (e.g.
protected FieldType	getTextFieldType() Subclass can override this method to change the field type of the text field e.g.
protected Object	highlight(String text, Set<String> matchedTokens, String prefixToken) Override this method to customize the Object representing a single highlighted suggestions; the result is set on each Lookup.LookupResult.highlightKey member.
boolean	load(DataInput out) Discard current lookup data and load it from a previously saved copy.
List<Lookup.LookupResult>	lookup(CharSequence key, BooleanQuery contextQuery, int num, boolean allTermsRequired, boolean doHighlight) This is an advanced method providing the capability to send down to the suggester any arbitrary lucene query to be used to filter the result of the suggester
List<Lookup.LookupResult>	lookup(CharSequence key, int num, boolean allTermsRequired, boolean doHighlight) Lookup, without any context.
List<Lookup.LookupResult>	lookup(CharSequence key, Map<BytesRef,BooleanClause.Occur> contextInfo, int num, boolean allTermsRequired, boolean doHighlight) Retrieve suggestions, specifying whether all terms must match (allTermsRequired) and whether the hits should be highlighted (doHighlight).
List<Lookup.LookupResult>	lookup(CharSequence key, Set<BytesRef> contexts, boolean onlyMorePopular, int num) Look up a key and return possible completion for this key.
List<Lookup.LookupResult>	lookup(CharSequence key, Set<BytesRef> contexts, int num, boolean allTermsRequired, boolean doHighlight) Lookup, with context but without booleans.
long	ramBytesUsed()
void	refresh() Reopens the underlying searcher; it's best to "batch up" many additions/updates, and then call refresh once in the end.
boolean	store(DataOutput in) Persist the constructed lookup data to a directory.
void	update(BytesRef text, Set<BytesRef> contexts, long weight, BytesRef payload) Updates a previous suggestion, matching the exact same text as before.

Methods inherited from class org.apache.lucene.search.suggest.Lookup

build, load, lookup, store

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEXT_FIELD_NAME

protected static final String TEXT_FIELD_NAME

Field name used for the indexed text.

See Also:

Constant Field Values

EXACT_TEXT_FIELD_NAME

protected static final String EXACT_TEXT_FIELD_NAME

Field name used for the indexed text, as a StringField, for exact lookup.

See Also:

Constant Field Values

CONTEXTS_FIELD_NAME

protected static final String CONTEXTS_FIELD_NAME

Field name used for the indexed context, as a StringField and a SortedSetDVField, for filtering.

See Also:

Constant Field Values

queryAnalyzer

protected final Analyzer queryAnalyzer

Analyzer used at search time

indexAnalyzer

protected final Analyzer indexAnalyzer

Analyzer used at index time

searcherMgr

protected SearcherManager searcherMgr

IndexSearcher used for lookups.

DEFAULT_MIN_PREFIX_CHARS

public static final int DEFAULT_MIN_PREFIX_CHARS

Default minimum number of leading characters before PrefixQuery is used (4).

See Also:

Constant Field Values

DEFAULT_ALL_TERMS_REQUIRED

public static final boolean DEFAULT_ALL_TERMS_REQUIRED

Default boolean clause option for multiple terms matching (all terms required).

See Also:

Constant Field Values

DEFAULT_HIGHLIGHT

public static final boolean DEFAULT_HIGHLIGHT

Default higlighting option.

See Also:

Constant Field Values

Constructor Detail

AnalyzingInfixSuggester

public AnalyzingInfixSuggester(Directory dir,
                               Analyzer analyzer)
                        throws IOException

Create a new instance, loading from a previously built AnalyzingInfixSuggester directory, if it exists. This directory must be private to the infix suggester (i.e., not an external Lucene index). Note that close() will also close the provided directory.

Throws:

IOException

AnalyzingInfixSuggester

public AnalyzingInfixSuggester(Directory dir,
                               Analyzer indexAnalyzer,
                               Analyzer queryAnalyzer,
                               int minPrefixChars,
                               boolean commitOnBuild)
                        throws IOException

Create a new instance, loading from a previously built AnalyzingInfixSuggester directory, if it exists. This directory must be private to the infix suggester (i.e., not an external Lucene index). Note that close() will also close the provided directory.

Parameters:

minPrefixChars - Minimum number of leading characters before PrefixQuery is used (default 4). Prefixes shorter than this are indexed as character ngrams (increasing index size but making lookups faster).

commitOnBuild - Call commit after the index has finished building. This would persist the suggester index to disk and future instances of this suggester can use this pre-built dictionary.

Throws:

IOException

AnalyzingInfixSuggester

public AnalyzingInfixSuggester(Directory dir,
                               Analyzer indexAnalyzer,
                               Analyzer queryAnalyzer,
                               int minPrefixChars,
                               boolean commitOnBuild,
                               boolean allTermsRequired,
                               boolean highlight)
                        throws IOException

Create a new instance, loading from a previously built AnalyzingInfixSuggester directory, if it exists. This directory must be private to the infix suggester (i.e., not an external Lucene index). Note that close() will also close the provided directory.

Parameters:

minPrefixChars - Minimum number of leading characters before PrefixQuery is used (default 4). Prefixes shorter than this are indexed as character ngrams (increasing index size but making lookups faster).

commitOnBuild - Call commit after the index has finished building. This would persist the suggester index to disk and future instances of this suggester can use this pre-built dictionary.

allTermsRequired - All terms in the suggest query must be matched.

highlight - Highlight suggest query in suggestions.

Throws:

IOException

Method Detail

getIndexWriterConfig

protected IndexWriterConfig getIndexWriterConfig(Analyzer indexAnalyzer,
                                                 IndexWriterConfig.OpenMode openMode)

Override this to customize index settings, e.g. which codec to use.

getDirectory

protected Directory getDirectory(Path path)
                          throws IOException

Subclass can override to choose a specific Directory implementation.

Throws:

IOException

build

public void build(InputIterator iter)
           throws IOException

Description copied from class: Lookup

Builds up a new internal Lookup representation based on the given InputIterator. The implementation might re-sort the data internally.

Specified by:

build in class Lookup

Throws:

IOException

commit

public void commit()
            throws IOException

Commits all pending changes made to this suggester to disk.

Throws:

IOException

See Also:

IndexWriter.commit()

add

public void add(BytesRef text,
                Set<BytesRef> contexts,
                long weight,
                BytesRef payload)
         throws IOException

Adds a new suggestion. Be sure to use update(org.apache.lucene.util.BytesRef, java.util.Set<org.apache.lucene.util.BytesRef>, long, org.apache.lucene.util.BytesRef) instead if you want to replace a previous suggestion. After adding or updating a batch of new suggestions, you must call refresh() in the end in order to see the suggestions in lookup(java.lang.CharSequence, java.util.Set<org.apache.lucene.util.BytesRef>, boolean, int)

Throws:

IOException

update

public void update(BytesRef text,
                   Set<BytesRef> contexts,
                   long weight,
                   BytesRef payload)
            throws IOException

Updates a previous suggestion, matching the exact same text as before. Use this to change the weight or payload of an already added suggestion. If you know this text is not already present you can use add(org.apache.lucene.util.BytesRef, java.util.Set<org.apache.lucene.util.BytesRef>, long, org.apache.lucene.util.BytesRef) instead. After adding or updating a batch of new suggestions, you must call refresh() in the end in order to see the suggestions in lookup(java.lang.CharSequence, java.util.Set<org.apache.lucene.util.BytesRef>, boolean, int)

Throws:

IOException

refresh

public void refresh()
             throws IOException

Reopens the underlying searcher; it's best to "batch up" many additions/updates, and then call refresh once in the end.

Throws:

IOException

getTextFieldType

protected FieldType getTextFieldType()

Subclass can override this method to change the field type of the text field e.g. to change the index options

lookup

public List<Lookup.LookupResult> lookup(CharSequence key,
                                        Set<BytesRef> contexts,
                                        boolean onlyMorePopular,
                                        int num)
                                 throws IOException

Description copied from class: Lookup

Look up a key and return possible completion for this key.

Specified by:

lookup in class Lookup

Parameters:

key - lookup key. Depending on the implementation this may be a prefix, misspelling, or even infix.

contexts - contexts to filter the lookup by, or null if all contexts are allowed; if the suggestion contains any of the contexts, it's a match

onlyMorePopular - return only more popular results

num - maximum number of results to return

Returns:

a list of possible completions, with their relative weight (e.g. popularity)

Throws:

IOException

lookup

public List<Lookup.LookupResult> lookup(CharSequence key,
                                        int num,
                                        boolean allTermsRequired,
                                        boolean doHighlight)
                                 throws IOException

Lookup, without any context.

Throws:

IOException

lookup

public List<Lookup.LookupResult> lookup(CharSequence key,
                                        Set<BytesRef> contexts,
                                        int num,
                                        boolean allTermsRequired,
                                        boolean doHighlight)
                                 throws IOException

Lookup, with context but without booleans. Context booleans default to SHOULD, so each suggestion must have at least one of the contexts.

Throws:

IOException

getLastTokenQuery

protected Query getLastTokenQuery(String token)
                           throws IOException

This is called if the last token isn't ended (e.g. user did not type a space after it). Return an appropriate Query clause to add to the BooleanQuery.

Throws:

IOException

lookup

public List<Lookup.LookupResult> lookup(CharSequence key,
                                        Map<BytesRef,BooleanClause.Occur> contextInfo,
                                        int num,
                                        boolean allTermsRequired,
                                        boolean doHighlight)
                                 throws IOException

Retrieve suggestions, specifying whether all terms must match (allTermsRequired) and whether the hits should be highlighted (doHighlight).

Throws:

IOException

addContextToQuery

public void addContextToQuery(BooleanQuery.Builder query,
                              BytesRef context,
                              BooleanClause.Occur clause)

This method is handy as we do not need access to internal fields such as CONTEXTS_FIELD_NAME in order to build queries However, here may not be its best location.

Parameters:

query - an instance of @See BooleanQuery

context - the context

clause - one of BooleanClause.Occur

lookup

public List<Lookup.LookupResult> lookup(CharSequence key,
                                        BooleanQuery contextQuery,
                                        int num,
                                        boolean allTermsRequired,
                                        boolean doHighlight)
                                 throws IOException

This is an advanced method providing the capability to send down to the suggester any arbitrary lucene query to be used to filter the result of the suggester

Overrides:

lookup in class Lookup

Parameters:

key - the keyword being looked for

contextQuery - an arbitrary Lucene query to be used to filter the result of the suggester. addContextToQuery(org.apache.lucene.search.BooleanQuery.Builder, org.apache.lucene.util.BytesRef, org.apache.lucene.search.BooleanClause.Occur) could be used to build this contextQuery.

num - number of items to return

allTermsRequired - all searched terms must match or not

doHighlight - if true, the matching term will be highlighted in the search result

Returns:

the result of the suggester

Throws:

IOException - f the is IO exception while reading data from the index

createResults

protected List<Lookup.LookupResult> createResults(IndexSearcher searcher,
                                                  TopFieldDocs hits,
                                                  int num,
                                                  CharSequence charSequence,
                                                  boolean doHighlight,
                                                  Set<String> matchedTokens,
                                                  String prefixToken)
                                           throws IOException

Create the results based on the search hits. Can be overridden by subclass to add particular behavior (e.g. weight transformation). Note that there is no prefix toke (the prefixToken argument will be null) whenever the final token in the incoming request was in fact finished (had trailing characters, such as white-space).

Throws:

IOException - If there are problems reading fields from the underlying Lucene index.

finishQuery

protected Query finishQuery(BooleanQuery.Builder in,
                            boolean allTermsRequired)

Subclass can override this to tweak the Query before searching.

highlight

protected Object highlight(String text,
                           Set<String> matchedTokens,
                           String prefixToken)
                    throws IOException

Override this method to customize the Object representing a single highlighted suggestions; the result is set on each Lookup.LookupResult.highlightKey member.

Throws:

IOException

addNonMatch

protected void addNonMatch(StringBuilder sb,
                           String text)

Called while highlighting a single result, to append a non-matching chunk of text from the suggestion to the provided fragments list.

Parameters:

sb - The StringBuilder to append to

text - The text chunk to add

addWholeMatch

protected void addWholeMatch(StringBuilder sb,
                             String surface,
                             String analyzed)

Called while highlighting a single result, to append the whole matched token to the provided fragments list.

Parameters:

sb - The StringBuilder to append to

surface - The surface form (original) text

analyzed - The analyzed token corresponding to the surface form text

addPrefixMatch

protected void addPrefixMatch(StringBuilder sb,
                              String surface,
                              String analyzed,
                              String prefixToken)

Called while highlighting a single result, to append a matched prefix token, to the provided fragments list.

Parameters:

sb - The StringBuilder to append to

surface - The fragment of the surface form (indexed during build(org.apache.lucene.search.suggest.InputIterator), corresponding to this match

analyzed - The analyzed token that matched

prefixToken - The prefix of the token that matched

store

public boolean store(DataOutput in)
              throws IOException

Description copied from class: Lookup

Persist the constructed lookup data to a directory. Optional operation.

Specified by:

store in class Lookup

Parameters:

in - DataOutput to write the data to.

Returns:

true if successful, false if unsuccessful or not supported.

Throws:

IOException - when fatal IO error occurs.

load

public boolean load(DataInput out)
             throws IOException

Description copied from class: Lookup

Discard current lookup data and load it from a previously saved copy. Optional operation.

Specified by:

load in class Lookup

Parameters:

out - the DataInput to load the lookup data.

Returns:

true if completed successfully, false if unsuccessful or not supported.

Throws:

IOException - when fatal IO error occurs.

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

ramBytesUsed

public long ramBytesUsed()

Specified by:

ramBytesUsed in interface Accountable

getChildResources

public Collection<Accountable> getChildResources()

Specified by:

getChildResources in interface Accountable

getCount

public long getCount()
              throws IOException

Description copied from class: Lookup

Get the number of entries the lookup was built with

Specified by:

getCount in class Lookup

Returns:

total number of suggester entries

Throws:

IOException