org.apache.lucene.index
Class LiveIndexWriterConfig

java.lang.Object
  extended by org.apache.lucene.index.LiveIndexWriterConfig
Direct Known Subclasses:
IndexWriterConfig

public class LiveIndexWriterConfig
extends Object

Holds all the configuration used by IndexWriter with few setters for settings that can be changed on an IndexWriter instance "live".

Since:
4.0

Field Summary
protected  Codec codec
          Codec used to write new segments.
protected  IndexCommit commit
          IndexCommit that IndexWriter is opened on.
protected  IndexDeletionPolicy delPolicy
          IndexDeletionPolicy controlling when commit points are deleted.
protected  org.apache.lucene.index.FlushPolicy flushPolicy
          FlushPolicy to control when segments are flushed.
protected  org.apache.lucene.index.DocumentsWriterPerThreadPool indexerThreadPool
          DocumentsWriterPerThreadPool to control how threads are allocated to DocumentsWriterPerThread.
protected  org.apache.lucene.index.DocumentsWriterPerThread.IndexingChain indexingChain
          DocumentsWriterPerThread.IndexingChain that determines how documents are indexed.
protected  InfoStream infoStream
          InfoStream for debugging messages.
protected  Version matchVersion
          Version that IndexWriter should emulate.
protected  MergePolicy mergePolicy
          MergePolicy for selecting merges.
protected  MergeScheduler mergeScheduler
          MergeScheduler to use for running merges.
protected  IndexWriterConfig.OpenMode openMode
          IndexWriterConfig.OpenMode that IndexWriter is opened with.
protected  int perThreadHardLimitMB
          Sets the hard upper bound on RAM usage for a single segment, after which the segment is forced to flush.
protected  boolean readerPooling
          True if readers should be pooled.
protected  Similarity similarity
          Similarity to use when encoding norms.
protected  boolean useCompoundFile
          True if segment flushes should use compound file format
protected  long writeLockTimeout
          Timeout when trying to obtain the write lock on init.
 
Method Summary
 Analyzer getAnalyzer()
          Returns the default analyzer to use for indexing documents.
 Codec getCodec()
          Returns the current Codec.
 IndexCommit getIndexCommit()
          Returns the IndexCommit as specified in IndexWriterConfig.setIndexCommit(IndexCommit) or the default, null which specifies to open the latest index commit point.
 IndexDeletionPolicy getIndexDeletionPolicy()
          Returns the IndexDeletionPolicy specified in IndexWriterConfig.setIndexDeletionPolicy(IndexDeletionPolicy) or the default KeepOnlyLastCommitDeletionPolicy/
 InfoStream getInfoStream()
          Returns InfoStream used for debugging.
 int getMaxBufferedDeleteTerms()
          Returns the number of buffered deleted terms that will trigger a flush of all buffered deletes if enabled.
 int getMaxBufferedDocs()
          Returns the number of buffered added documents that will trigger a flush if enabled.
 int getMaxThreadStates()
          Returns the max number of simultaneous threads that may be indexing documents at once in IndexWriter.
 IndexWriter.IndexReaderWarmer getMergedSegmentWarmer()
          Returns the current merged segment warmer.
 MergePolicy getMergePolicy()
          Returns the current MergePolicy in use by this writer.
 MergeScheduler getMergeScheduler()
          Returns the MergeScheduler that was set by IndexWriterConfig.setMergeScheduler(MergeScheduler).
 IndexWriterConfig.OpenMode getOpenMode()
          Returns the IndexWriterConfig.OpenMode set by IndexWriterConfig.setOpenMode(OpenMode).
 double getRAMBufferSizeMB()
          Returns the value set by setRAMBufferSizeMB(double) if enabled.
 int getRAMPerThreadHardLimitMB()
          Returns the max amount of memory each DocumentsWriterPerThread can consume until forcefully flushed.
 boolean getReaderPooling()
          Returns true if IndexWriter should pool readers even if DirectoryReader.open(IndexWriter, boolean) has not been called.
 int getReaderTermsIndexDivisor()
          Returns the termInfosIndexDivisor.
 Similarity getSimilarity()
          Expert: returns the Similarity implementation used by this IndexWriter.
 int getTermIndexInterval()
          Returns the interval between indexed terms.
 boolean getUseCompoundFile()
          Returns true iff the IndexWriter packs newly written segments in a compound file.
 long getWriteLockTimeout()
          Returns allowed timeout when acquiring the write lock.
 LiveIndexWriterConfig setMaxBufferedDeleteTerms(int maxBufferedDeleteTerms)
          Determines the maximum number of delete-by-term operations that will be buffered before both the buffered in-memory delete terms and queries are applied and flushed.
 LiveIndexWriterConfig setMaxBufferedDocs(int maxBufferedDocs)
          Determines the minimal number of documents required before the buffered in-memory documents are flushed as a new Segment.
 LiveIndexWriterConfig setMergedSegmentWarmer(IndexWriter.IndexReaderWarmer mergeSegmentWarmer)
          Set the merged segment warmer.
 LiveIndexWriterConfig setRAMBufferSizeMB(double ramBufferSizeMB)
          Determines the amount of RAM that may be used for buffering added documents and deletions before they are flushed to the Directory.
 LiveIndexWriterConfig setReaderTermsIndexDivisor(int divisor)
          Sets the termsIndexDivisor passed to any readers that IndexWriter opens, for example when applying deletes or creating a near-real-time reader in DirectoryReader.open(IndexWriter, boolean).
 LiveIndexWriterConfig setTermIndexInterval(int interval)
          Expert: set the interval between indexed terms.
 LiveIndexWriterConfig setUseCompoundFile(boolean useCompoundFile)
          Sets if the IndexWriter should pack newly written segments in a compound file.
 String toString()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

delPolicy

protected volatile IndexDeletionPolicy delPolicy
IndexDeletionPolicy controlling when commit points are deleted.


commit

protected volatile IndexCommit commit
IndexCommit that IndexWriter is opened on.


openMode

protected volatile IndexWriterConfig.OpenMode openMode
IndexWriterConfig.OpenMode that IndexWriter is opened with.


similarity

protected volatile Similarity similarity
Similarity to use when encoding norms.


mergeScheduler

protected volatile MergeScheduler mergeScheduler
MergeScheduler to use for running merges.


writeLockTimeout

protected volatile long writeLockTimeout
Timeout when trying to obtain the write lock on init.


indexingChain

protected volatile org.apache.lucene.index.DocumentsWriterPerThread.IndexingChain indexingChain
DocumentsWriterPerThread.IndexingChain that determines how documents are indexed.


codec

protected volatile Codec codec
Codec used to write new segments.


infoStream

protected volatile InfoStream infoStream
InfoStream for debugging messages.


mergePolicy

protected volatile MergePolicy mergePolicy
MergePolicy for selecting merges.


indexerThreadPool

protected volatile org.apache.lucene.index.DocumentsWriterPerThreadPool indexerThreadPool
DocumentsWriterPerThreadPool to control how threads are allocated to DocumentsWriterPerThread.


readerPooling

protected volatile boolean readerPooling
True if readers should be pooled.


flushPolicy

protected volatile org.apache.lucene.index.FlushPolicy flushPolicy
FlushPolicy to control when segments are flushed.


perThreadHardLimitMB

protected volatile int perThreadHardLimitMB
Sets the hard upper bound on RAM usage for a single segment, after which the segment is forced to flush.


matchVersion

protected final Version matchVersion
Version that IndexWriter should emulate.


useCompoundFile

protected volatile boolean useCompoundFile
True if segment flushes should use compound file format

Method Detail

getAnalyzer

public Analyzer getAnalyzer()
Returns the default analyzer to use for indexing documents.


setTermIndexInterval

public LiveIndexWriterConfig setTermIndexInterval(int interval)
Expert: set the interval between indexed terms. Large values cause less memory to be used by IndexReader, but slow random-access to terms. Small values cause more memory to be used by an IndexReader, and speed random-access to terms.

This parameter determines the amount of computation required per query term, regardless of the number of documents that contain that term. In particular, it is the maximum number of other terms that must be scanned before a term is located and its frequency and position information may be processed. In a large index with user-entered query terms, query processing time is likely to be dominated not by term lookup but rather by the processing of frequency and positional data. In a small index or when many uncommon query terms are generated (e.g., by wildcard queries) term lookup may become a dominant cost.

In particular, numUniqueTerms/interval terms are read into memory by an IndexReader, and, on average, interval/2 terms must be scanned for each random term access.

Takes effect immediately, but only applies to newly flushed/merged segments.

NOTE: This parameter does not apply to all PostingsFormat implementations, including the default one in this release. It only makes sense for term indexes that are implemented as a fixed gap between terms. For example, Lucene41PostingsFormat implements the term index instead based upon how terms share prefixes. To configure its parameters (the minimum and maximum size for a block), you would instead use Lucene41PostingsFormat.Lucene41PostingsFormat(int, int). which can also be configured on a per-field basis:

 //customize Lucene41PostingsFormat, passing minBlockSize=50, maxBlockSize=100
 final PostingsFormat tweakedPostings = new Lucene41PostingsFormat(50, 100);
 iwc.setCodec(new Lucene45Codec() {
   @Override
   public PostingsFormat getPostingsFormatForField(String field) {
     if (field.equals("fieldWithTonsOfTerms"))
       return tweakedPostings;
     else
       return super.getPostingsFormatForField(field);
   }
 });
 
Note that other implementations may have their own parameters, or no parameters at all.

See Also:
IndexWriterConfig.DEFAULT_TERM_INDEX_INTERVAL

getTermIndexInterval

public int getTermIndexInterval()
Returns the interval between indexed terms.

See Also:
setTermIndexInterval(int)

setMaxBufferedDeleteTerms

public LiveIndexWriterConfig setMaxBufferedDeleteTerms(int maxBufferedDeleteTerms)
Determines the maximum number of delete-by-term operations that will be buffered before both the buffered in-memory delete terms and queries are applied and flushed.

Disabled by default (writer flushes by RAM usage).

NOTE: This setting won't trigger a segment flush.

Takes effect immediately, but only the next time a document is added, updated or deleted. Also, if you only delete-by-query, this setting has no effect, i.e. delete queries are buffered until the next segment is flushed.

Throws:
IllegalArgumentException - if maxBufferedDeleteTerms is enabled but smaller than 1
See Also:
setRAMBufferSizeMB(double)

getMaxBufferedDeleteTerms

public int getMaxBufferedDeleteTerms()
Returns the number of buffered deleted terms that will trigger a flush of all buffered deletes if enabled.

See Also:
setMaxBufferedDeleteTerms(int)

setRAMBufferSizeMB

public LiveIndexWriterConfig setRAMBufferSizeMB(double ramBufferSizeMB)
Determines the amount of RAM that may be used for buffering added documents and deletions before they are flushed to the Directory. Generally for faster indexing performance it's best to flush by RAM usage instead of document count and use as large a RAM buffer as you can.

When this is set, the writer will flush whenever buffered documents and deletions use this much RAM. Pass in IndexWriterConfig.DISABLE_AUTO_FLUSH to prevent triggering a flush due to RAM usage. Note that if flushing by document count is also enabled, then the flush will be triggered by whichever comes first.

The maximum RAM limit is inherently determined by the JVMs available memory. Yet, an IndexWriter session can consume a significantly larger amount of memory than the given RAM limit since this limit is just an indicator when to flush memory resident documents to the Directory. Flushes are likely happen concurrently while other threads adding documents to the writer. For application stability the available memory in the JVM should be significantly larger than the RAM buffer used for indexing.

NOTE: the account of RAM usage for pending deletions is only approximate. Specifically, if you delete by Query, Lucene currently has no way to measure the RAM usage of individual Queries so the accounting will under-estimate and you should compensate by either calling commit() periodically yourself, or by using setMaxBufferedDeleteTerms(int) to flush and apply buffered deletes by count instead of RAM usage (for each buffered delete Query a constant number of bytes is used to estimate RAM usage). Note that enabling setMaxBufferedDeleteTerms(int) will not trigger any segment flushes.

NOTE: It's not guaranteed that all memory resident documents are flushed once this limit is exceeded. Depending on the configured FlushPolicy only a subset of the buffered documents are flushed and therefore only parts of the RAM buffer is released.

The default value is IndexWriterConfig.DEFAULT_RAM_BUFFER_SIZE_MB.

Takes effect immediately, but only the next time a document is added, updated or deleted.

Throws:
IllegalArgumentException - if ramBufferSize is enabled but non-positive, or it disables ramBufferSize when maxBufferedDocs is already disabled
See Also:
IndexWriterConfig.setRAMPerThreadHardLimitMB(int)

getRAMBufferSizeMB

public double getRAMBufferSizeMB()
Returns the value set by setRAMBufferSizeMB(double) if enabled.


setMaxBufferedDocs

public LiveIndexWriterConfig setMaxBufferedDocs(int maxBufferedDocs)
Determines the minimal number of documents required before the buffered in-memory documents are flushed as a new Segment. Large values generally give faster indexing.

When this is set, the writer will flush every maxBufferedDocs added documents. Pass in IndexWriterConfig.DISABLE_AUTO_FLUSH to prevent triggering a flush due to number of buffered documents. Note that if flushing by RAM usage is also enabled, then the flush will be triggered by whichever comes first.

Disabled by default (writer flushes by RAM usage).

Takes effect immediately, but only the next time a document is added, updated or deleted.

Throws:
IllegalArgumentException - if maxBufferedDocs is enabled but smaller than 2, or it disables maxBufferedDocs when ramBufferSize is already disabled
See Also:
setRAMBufferSizeMB(double)

getMaxBufferedDocs

public int getMaxBufferedDocs()
Returns the number of buffered added documents that will trigger a flush if enabled.

See Also:
setMaxBufferedDocs(int)

setMergedSegmentWarmer

public LiveIndexWriterConfig setMergedSegmentWarmer(IndexWriter.IndexReaderWarmer mergeSegmentWarmer)
Set the merged segment warmer. See IndexWriter.IndexReaderWarmer.

Takes effect on the next merge.


getMergedSegmentWarmer

public IndexWriter.IndexReaderWarmer getMergedSegmentWarmer()
Returns the current merged segment warmer. See IndexWriter.IndexReaderWarmer.


setReaderTermsIndexDivisor

public LiveIndexWriterConfig setReaderTermsIndexDivisor(int divisor)
Sets the termsIndexDivisor passed to any readers that IndexWriter opens, for example when applying deletes or creating a near-real-time reader in DirectoryReader.open(IndexWriter, boolean). If you pass -1, the terms index won't be loaded by the readers. This is only useful in advanced situations when you will only .next() through all terms; attempts to seek will hit an exception.

Takes effect immediately, but only applies to readers opened after this call

NOTE: divisor settings > 1 do not apply to all PostingsFormat implementations, including the default one in this release. It only makes sense for terms indexes that can efficiently re-sample terms at load time.


getReaderTermsIndexDivisor

public int getReaderTermsIndexDivisor()
Returns the termInfosIndexDivisor.

See Also:
setReaderTermsIndexDivisor(int)

getOpenMode

public IndexWriterConfig.OpenMode getOpenMode()
Returns the IndexWriterConfig.OpenMode set by IndexWriterConfig.setOpenMode(OpenMode).


getIndexDeletionPolicy

public IndexDeletionPolicy getIndexDeletionPolicy()
Returns the IndexDeletionPolicy specified in IndexWriterConfig.setIndexDeletionPolicy(IndexDeletionPolicy) or the default KeepOnlyLastCommitDeletionPolicy/


getIndexCommit

public IndexCommit getIndexCommit()
Returns the IndexCommit as specified in IndexWriterConfig.setIndexCommit(IndexCommit) or the default, null which specifies to open the latest index commit point.


getSimilarity

public Similarity getSimilarity()
Expert: returns the Similarity implementation used by this IndexWriter.


getMergeScheduler

public MergeScheduler getMergeScheduler()
Returns the MergeScheduler that was set by IndexWriterConfig.setMergeScheduler(MergeScheduler).


getWriteLockTimeout

public long getWriteLockTimeout()
Returns allowed timeout when acquiring the write lock.

See Also:
IndexWriterConfig.setWriteLockTimeout(long)

getCodec

public Codec getCodec()
Returns the current Codec.


getMergePolicy

public MergePolicy getMergePolicy()
Returns the current MergePolicy in use by this writer.

See Also:
IndexWriterConfig.setMergePolicy(MergePolicy)

getMaxThreadStates

public int getMaxThreadStates()
Returns the max number of simultaneous threads that may be indexing documents at once in IndexWriter.


getReaderPooling

public boolean getReaderPooling()
Returns true if IndexWriter should pool readers even if DirectoryReader.open(IndexWriter, boolean) has not been called.


getRAMPerThreadHardLimitMB

public int getRAMPerThreadHardLimitMB()
Returns the max amount of memory each DocumentsWriterPerThread can consume until forcefully flushed.

See Also:
IndexWriterConfig.setRAMPerThreadHardLimitMB(int)

getInfoStream

public InfoStream getInfoStream()
Returns InfoStream used for debugging.

See Also:
IndexWriterConfig.setInfoStream(InfoStream)

setUseCompoundFile

public LiveIndexWriterConfig setUseCompoundFile(boolean useCompoundFile)
Sets if the IndexWriter should pack newly written segments in a compound file. Default is true.

Use false for batch indexing with very large ram buffer settings.

Note: To control compound file usage during segment merges see MergePolicy.setNoCFSRatio(double) and MergePolicy.setMaxCFSSegmentSizeMB(double). This setting only applies to newly created segments.


getUseCompoundFile

public boolean getUseCompoundFile()
Returns true iff the IndexWriter packs newly written segments in a compound file. Default is true.


toString

public String toString()
Overrides:
toString in class Object


Copyright © 2000-2014 Apache Software Foundation. All Rights Reserved.