org.apache.lucene.facet.taxonomy.directory

Class DirectoryTaxonomyWriter

java.lang.Object

org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter

All Implemented Interfaces:

Closeable, TaxonomyWriter, TwoPhaseCommit

public class DirectoryTaxonomyWriter
extends Object
implements TaxonomyWriter

TaxonomyWriter which uses a Directory to store the taxonomy information on disk, and keeps an additional in-memory cache of some or all categories.

In addition to the permanently-stored information in the Directory, efficiency dictates that we also keep an in-memory cache of recently seen or all categories, so that we do not need to go back to disk for every category addition to see which ordinal this category already has, if any. A TaxonomyWriterCache object determines the specific caching algorithm used.

This class offers some hooks for extending classes to control the IndexWriter instance that is used. See openIndexWriter(org.apache.lucene.store.Directory, org.apache.lucene.index.IndexWriterConfig).

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DirectoryTaxonomyWriter.DiskOrdinalMap DirectoryTaxonomyWriter.OrdinalMap maintained on file system
static class	DirectoryTaxonomyWriter.MemoryOrdinalMap DirectoryTaxonomyWriter.OrdinalMap maintained in memory
static interface	DirectoryTaxonomyWriter.OrdinalMap Mapping from old ordinal to new ordinals, used when merging indexes wit separate taxonomies.

Field Summary

Fields

Modifier and Type	Field and Description
static String	INDEX_EPOCH Property name of user commit data that contains the index epoch.

Constructor Summary

Constructors

Constructor and Description
DirectoryTaxonomyWriter(Directory d) Create this with OpenMode.CREATE_OR_APPEND.
DirectoryTaxonomyWriter(Directory directory, IndexWriterConfig.OpenMode openMode) Creates a new instance with a default cache as defined by defaultTaxonomyWriterCache().
DirectoryTaxonomyWriter(Directory directory, IndexWriterConfig.OpenMode openMode, TaxonomyWriterCache cache) Construct a Taxonomy writer.

Method Summary

Methods

Modifier and Type	Method and Description
int	addCategory(FacetLabel categoryPath) addCategory() adds a category with a given path name to the taxonomy, and returns its ordinal.
void	addTaxonomy(Directory taxoDir, DirectoryTaxonomyWriter.OrdinalMap map) Takes the categories from the given taxonomy directory, and adds the missing ones to this taxonomy.
void	close() Frees used resources as well as closes the underlying IndexWriter, which commits whatever changes made to it to the underlying Directory.
protected void	closeResources() A hook for extending classes to close additional resources that were used.
void	commit()
protected IndexWriterConfig	createIndexWriterConfig(IndexWriterConfig.OpenMode openMode) Create the IndexWriterConfig that would be used for opening the internal index writer.
static TaxonomyWriterCache	defaultTaxonomyWriterCache() Defines the default TaxonomyWriterCache to use in constructors which do not specify one.
protected void	ensureOpen() Verifies that this instance wasn't closed, or throws AlreadyClosedException if it is.
protected int	findCategory(FacetLabel categoryPath) Look up the given category in the cache and/or the on-disk storage, returning the category's ordinal, or a negative number in case the category does not yet exist in the taxonomy.
Map<String,String>	getCommitData() Returns the commit user data map that was set on TaxonomyWriter.setCommitData(Map).
Directory	getDirectory() Returns the Directory of this taxonomy writer.
int	getParent(int ordinal) getParent() returns the ordinal of the parent category of the category with the given ordinal.
int	getSize() getSize() returns the number of categories in the taxonomy.
long	getTaxonomyEpoch() Expert: returns current index epoch, if this is a near-real-time reader.
protected IndexWriter	openIndexWriter(Directory directory, IndexWriterConfig config) Open internal index writer, which contains the taxonomy data.
void	prepareCommit() prepare most of the work needed for a two-phase commit.
void	replaceTaxonomy(Directory taxoDir) Replaces the current taxonomy with the given one.
void	rollback() Rollback changes to the taxonomy writer and closes the instance.
void	setCacheMissesUntilFill(int i) Set the number of cache misses before an attempt is made to read the entire taxonomy into the in-memory cache.
void	setCommitData(Map<String,String> commitUserData) Sets the commit user data map.
static void	unlock(Directory directory) Forcibly unlocks the taxonomy in the named directory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INDEX_EPOCH

public static final String INDEX_EPOCH

Property name of user commit data that contains the index epoch. The epoch changes whenever the taxonomy is recreated (i.e. opened with IndexWriterConfig.OpenMode.CREATE.

Applications should not use this property in their commit data because it will be overridden by this taxonomy writer.

See Also:

Constant Field Values

Constructor Detail

DirectoryTaxonomyWriter

public DirectoryTaxonomyWriter(Directory directory,
                       IndexWriterConfig.OpenMode openMode,
                       TaxonomyWriterCache cache)
                        throws IOException

Construct a Taxonomy writer.

Parameters:

directory - The Directory in which to store the taxonomy. Note that the taxonomy is written directly to that directory (not to a subdirectory of it).

openMode - Specifies how to open a taxonomy for writing: APPEND means open an existing index for append (failing if the index does not yet exist). CREATE means create a new index (first deleting the old one if it already existed). APPEND_OR_CREATE appends to an existing index if there is one, otherwise it creates a new index.

cache - A TaxonomyWriterCache implementation which determines the in-memory caching policy. See for example LruTaxonomyWriterCache and Cl2oTaxonomyWriterCache. If null or missing, defaultTaxonomyWriterCache() is used.

Throws:

CorruptIndexException - if the taxonomy is corrupted.

LockObtainFailedException - if the taxonomy is locked by another writer. If it is known that no other concurrent writer is active, the lock might have been left around by an old dead process, and should be removed using unlock(Directory).

IOException - if another error occurred.

DirectoryTaxonomyWriter

public DirectoryTaxonomyWriter(Directory directory,
                       IndexWriterConfig.OpenMode openMode)
                        throws IOException

Creates a new instance with a default cache as defined by defaultTaxonomyWriterCache().

Throws:

IOException

DirectoryTaxonomyWriter

public DirectoryTaxonomyWriter(Directory d)
                        throws IOException

Create this with OpenMode.CREATE_OR_APPEND.

Throws:

IOException

Method Detail

unlock

public static void unlock(Directory directory)
                   throws IOException

Forcibly unlocks the taxonomy in the named directory.

Caution: this should only be used by failure recovery code, when it is known that no other process nor thread is in fact currently accessing this taxonomy.

This method is unnecessary if your Directory uses a NativeFSLockFactory instead of the default SimpleFSLockFactory. When the "native" lock is used, a lock does not stay behind forever when the process using it dies.

Throws:

IOException

openIndexWriter

protected IndexWriter openIndexWriter(Directory directory,
                          IndexWriterConfig config)
                               throws IOException

Open internal index writer, which contains the taxonomy data.

Extensions may provide their own IndexWriter implementation or instance.
NOTE: the instance this method returns will be closed upon calling to close().
NOTE: the merge policy in effect must not merge none adjacent segments. See comment in createIndexWriterConfig(IndexWriterConfig.OpenMode) for the logic behind this.

Parameters:

directory - the Directory on top of which an IndexWriter should be opened.

config - configuration for the internal index writer.

Throws:

IOException

See Also:

createIndexWriterConfig(IndexWriterConfig.OpenMode)

createIndexWriterConfig

protected IndexWriterConfig createIndexWriterConfig(IndexWriterConfig.OpenMode openMode)

Create the IndexWriterConfig that would be used for opening the internal index writer.
Extensions can configure the IndexWriter as they see fit, including setting a merge-scheduler, or deletion-policy, different RAM size etc.

NOTE: internal docids of the configured index must not be altered. For that, categories are never deleted from the taxonomy index. In addition, merge policy in effect must not merge none adjacent segments.

Parameters:

openMode - see IndexWriterConfig.OpenMode

See Also:

openIndexWriter(Directory, IndexWriterConfig)

defaultTaxonomyWriterCache

public static TaxonomyWriterCache defaultTaxonomyWriterCache()

Defines the default TaxonomyWriterCache to use in constructors which do not specify one.

The current default is Cl2oTaxonomyWriterCache constructed with the parameters (1024, 0.15f, 3), i.e., the entire taxonomy is cached in memory while building it.

close

public void close()
           throws IOException

Frees used resources as well as closes the underlying IndexWriter, which commits whatever changes made to it to the underlying Directory.

Specified by:

close in interface Closeable

Throws:

IOException

closeResources

protected void closeResources()
                       throws IOException

A hook for extending classes to close additional resources that were used. The default implementation closes the IndexReader as well as the TaxonomyWriterCache instances that were used.
NOTE: if you override this method, you should include a super.closeResources() call in your implementation.

Throws:

IOException

findCategory

protected int findCategory(FacetLabel categoryPath)
                    throws IOException

Look up the given category in the cache and/or the on-disk storage, returning the category's ordinal, or a negative number in case the category does not yet exist in the taxonomy.

Throws:

IOException

addCategory

public int addCategory(FacetLabel categoryPath)
                throws IOException

Description copied from interface: TaxonomyWriter

addCategory() adds a category with a given path name to the taxonomy, and returns its ordinal. If the category was already present in the taxonomy, its existing ordinal is returned.

Before adding a category, addCategory() makes sure that all its ancestor categories exist in the taxonomy as well. As result, the ordinal of a category is guaranteed to be smaller then the ordinal of any of its descendants.

Specified by:

addCategory in interface TaxonomyWriter

Throws:

IOException

ensureOpen

protected final void ensureOpen()

Verifies that this instance wasn't closed, or throws AlreadyClosedException if it is.

commit

public void commit()
            throws IOException

Specified by:

commit in interface TwoPhaseCommit

Throws:

IOException

setCommitData

public void setCommitData(Map<String,String> commitUserData)

Description copied from interface: TaxonomyWriter

Sets the commit user data map. That method is considered a transaction and will be committed even if no other changes were made to the writer instance.

NOTE: the map is cloned internally, therefore altering the map's contents after calling this method has no effect.

Specified by:

setCommitData in interface TaxonomyWriter

getCommitData

public Map<String,String> getCommitData()

Description copied from interface: TaxonomyWriter

Returns the commit user data map that was set on TaxonomyWriter.setCommitData(Map).

Specified by:

getCommitData in interface TaxonomyWriter

prepareCommit

public void prepareCommit()
                   throws IOException

prepare most of the work needed for a two-phase commit. See IndexWriter.prepareCommit().

Specified by:

prepareCommit in interface TwoPhaseCommit

Throws:

IOException

getSize

public int getSize()

Description copied from interface: TaxonomyWriter

getSize() returns the number of categories in the taxonomy.

Because categories are numbered consecutively starting with 0, it means the taxonomy contains ordinals 0 through getSize()-1.

Note that the number returned by getSize() is often slightly higher than the number of categories inserted into the taxonomy; This is because when a category is added to the taxonomy, its ancestors are also added automatically (including the root, which always get ordinal 0).

Specified by:

getSize in interface TaxonomyWriter

setCacheMissesUntilFill

public void setCacheMissesUntilFill(int i)

Set the number of cache misses before an attempt is made to read the entire taxonomy into the in-memory cache.

This taxonomy writer holds an in-memory cache of recently seen categories to speed up operation. On each cache-miss, the on-disk index needs to be consulted. When an existing taxonomy is opened, a lot of slow disk reads like that are needed until the cache is filled, so it is more efficient to read the entire taxonomy into memory at once. We do this complete read after a certain number (defined by this method) of cache misses.

If the number is set to 0, the entire taxonomy is read into the cache on first use, without fetching individual categories first.

NOTE: it is assumed that this method is called immediately after the taxonomy writer has been created.

getParent

public int getParent(int ordinal)
              throws IOException

Description copied from interface: TaxonomyWriter

getParent() returns the ordinal of the parent category of the category with the given ordinal.

When a category is specified as a path name, finding the path of its parent is as trivial as dropping the last component of the path. getParent() is functionally equivalent to calling getPath() on the given ordinal, dropping the last component of the path, and then calling getOrdinal() to get an ordinal back.

If the given ordinal is the ROOT_ORDINAL, an INVALID_ORDINAL is returned. If the given ordinal is a top-level category, the ROOT_ORDINAL is returned. If an invalid ordinal is given (negative or beyond the last available ordinal), an ArrayIndexOutOfBoundsException is thrown. However, it is expected that getParent will only be called for ordinals which are already known to be in the taxonomy. TODO (Facet): instead of a getParent(ordinal) method, consider having a

getCategory(categorypath, prefixlen) which is similar to addCategory except it doesn't add new categories; This method can be used to get the ordinals of all prefixes of the given category, and it can use exactly the same code and cache used by addCategory() so it means less code.

Specified by:

getParent in interface TaxonomyWriter

Throws:

IOException

addTaxonomy

public void addTaxonomy(Directory taxoDir,
               DirectoryTaxonomyWriter.OrdinalMap map)
                 throws IOException

Takes the categories from the given taxonomy directory, and adds the missing ones to this taxonomy. Additionally, it fills the given DirectoryTaxonomyWriter.OrdinalMap with a mapping from the original ordinal to the new ordinal.

Throws:

IOException

rollback

public void rollback()
              throws IOException

Rollback changes to the taxonomy writer and closes the instance. Following this method the instance becomes unusable (calling any of its API methods will yield an AlreadyClosedException).

Specified by:

rollback in interface TwoPhaseCommit

Throws:

IOException

replaceTaxonomy

public void replaceTaxonomy(Directory taxoDir)
                     throws IOException

Replaces the current taxonomy with the given one. This method should generally be called in conjunction with IndexWriter.addIndexes(Directory...) to replace both the taxonomy as well as the search index content.

Throws:

IOException

getDirectory

public Directory getDirectory()

Returns the Directory of this taxonomy writer.

getTaxonomyEpoch

public final long getTaxonomyEpoch()

Expert: returns current index epoch, if this is a near-real-time reader. Used by DirectoryTaxonomyReader to support NRT.

NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.