org.apache.lucene.index

Class LogMergePolicy

java.lang.Object

org.apache.lucene.index.MergePolicy

org.apache.lucene.index.LogMergePolicy

All Implemented Interfaces:

Closeable, Cloneable

Direct Known Subclasses:

LogByteSizeMergePolicy, LogDocMergePolicy

public abstract class LogMergePolicy
extends MergePolicy

This class implements a MergePolicy that tries to merge segments into levels of exponentially increasing size, where each level has fewer segments than the value of the merge factor. Whenever extra segments (beyond the merge factor upper bound) are encountered, all segments within the level are merged. You can get or set the merge factor using getMergeFactor() and setMergeFactor(int) respectively.

This class is abstract and requires a subclass to define the size(org.apache.lucene.index.SegmentInfoPerCommit) method which specifies how a segment's size is determined. LogDocMergePolicy is one subclass that measures size by document count in the segment. LogByteSizeMergePolicy is another subclass that measures size as the total byte size of the file(s) for the segment.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.index.MergePolicy

MergePolicy.MergeAbortedException, MergePolicy.MergeException, MergePolicy.MergeSpecification, MergePolicy.MergeTrigger, MergePolicy.OneMerge

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	calibrateSizeByDeletes If true, we pro-rate a segment's size by the percentage of non-deleted documents.
static long	DEFAULT_MAX_CFS_SEGMENT_SIZE Default maxCFSSegmentSize value allows compound file for a segment of any size.
static int	DEFAULT_MAX_MERGE_DOCS Default maximum segment size.
static int	DEFAULT_MERGE_FACTOR Default merge factor, which is how many segments are merged at a time
static double	DEFAULT_NO_CFS_RATIO Default noCFSRatio.
static double	LEVEL_LOG_SPAN Defines the allowed range of log(size) for each level.
protected long	maxCFSSegmentSize If the size of the merged segment exceeds this value then it will not use compound file format.
protected int	maxMergeDocs If a segment has more than this many documents then it will never be merged.
protected long	maxMergeSize If the size of a segment exceeds this value then it will never be merged.
protected long	maxMergeSizeForForcedMerge If the size of a segment exceeds this value then it will never be merged during IndexWriter.forceMerge(int).
protected int	mergeFactor How many segments to merge at a time.
protected long	minMergeSize Any segments whose size is smaller than this value will be rounded up to this value.
protected double	noCFSRatio If the size of the merge segment exceeds this ratio of the total index size then it will remain in non-compound format even if setUseCompoundFile(boolean) is true.
protected boolean	useCompoundFile True if new segments (flushed or merged) should use the compound file format.

Fields inherited from class org.apache.lucene.index.MergePolicy

writer

Constructor Summary

Constructors

Constructor and Description
LogMergePolicy() Sole constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Release all resources for the policy.
MergePolicy.MergeSpecification	findForcedDeletesMerges(SegmentInfos segmentInfos) Finds merges necessary to force-merge all deletes from the index.
MergePolicy.MergeSpecification	findForcedMerges(SegmentInfos infos, int maxNumSegments, Map<SegmentInfoPerCommit,Boolean> segmentsToMerge) Returns the merges necessary to merge the index down to a specified number of segments.
MergePolicy.MergeSpecification	findMerges(MergePolicy.MergeTrigger mergeTrigger, SegmentInfos infos) Checks if any merges are now necessary and returns a MergePolicy.MergeSpecification if so.
boolean	getCalibrateSizeByDeletes() Returns true if the segment size should be calibrated by the number of deletes when choosing segments for merge.
double	getMaxCFSSegmentSizeMB() Returns the largest size allowed for a compound file segment
int	getMaxMergeDocs() Returns the largest segment (measured by document count) that may be merged with other segments.
int	getMergeFactor() Returns the number of segments that are merged at once and also controls the total number of segments allowed to accumulate in the index.
double	getNoCFSRatio() Returns current noCFSRatio.
boolean	getUseCompoundFile() Returns true if newly flushed and newly merge segments are written in compound file format.
protected boolean	isMerged(SegmentInfoPerCommit info) Returns true if this single info is already fully merged (has no pending norms or deletes, is in the same dir as the writer, and matches the current compound file setting
protected boolean	isMerged(SegmentInfos infos, int maxNumSegments, Map<SegmentInfoPerCommit,Boolean> segmentsToMerge) Returns true if the number of segments eligible for merging is less than or equal to the specified maxNumSegments.
protected void	message(String message) Print a debug message to IndexWriter's infoStream.
void	setCalibrateSizeByDeletes(boolean calibrateSizeByDeletes) Sets whether the segment size should be calibrated by the number of deletes when choosing segments for merge.
void	setMaxCFSSegmentSizeMB(double v) If a merged segment will be more than this value, leave the segment as non-compound file even if compound file is enabled.
void	setMaxMergeDocs(int maxMergeDocs) Determines the largest segment (measured by document count) that may be merged with other segments.
void	setMergeFactor(int mergeFactor) Determines how often segment indices are merged by addDocument().
void	setNoCFSRatio(double noCFSRatio) If a merged segment will be more than this percentage of the total size of the index, leave the segment as non-compound file even if compound file is enabled.
void	setUseCompoundFile(boolean useCompoundFile) Sets whether compound file format should be used for newly flushed and newly merged segments.
protected abstract long	size(SegmentInfoPerCommit info) Return the size of the provided SegmentInfoPerCommit.
protected long	sizeBytes(SegmentInfoPerCommit info) Return the byte size of the provided SegmentInfoPerCommit, pro-rated by percentage of non-deleted documents if setCalibrateSizeByDeletes(boolean) is set.
protected long	sizeDocs(SegmentInfoPerCommit info) Return the number of documents in the provided SegmentInfoPerCommit, pro-rated by percentage of non-deleted documents if setCalibrateSizeByDeletes(boolean) is set.
String	toString()
boolean	useCompoundFile(SegmentInfos infos, SegmentInfoPerCommit mergedInfo) Returns true if a new segment (regardless of its origin) should use the compound file format.
protected boolean	verbose() Returns true if LMP is enabled in IndexWriter's infoStream.

Methods inherited from class org.apache.lucene.index.MergePolicy

clone, setIndexWriter

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LEVEL_LOG_SPAN

public static final double LEVEL_LOG_SPAN

Defines the allowed range of log(size) for each level. A level is computed by taking the max segment log size, minus LEVEL_LOG_SPAN, and finding all segments falling within that range.

See Also:

Constant Field Values

DEFAULT_MERGE_FACTOR

public static final int DEFAULT_MERGE_FACTOR

Default merge factor, which is how many segments are merged at a time

See Also:

Constant Field Values

DEFAULT_MAX_MERGE_DOCS

public static final int DEFAULT_MAX_MERGE_DOCS

Default maximum segment size. A segment of this size or larger will never be merged. @see setMaxMergeDocs

See Also:

Constant Field Values

DEFAULT_NO_CFS_RATIO

public static final double DEFAULT_NO_CFS_RATIO

Default noCFSRatio. If a merge's size is >= 10% of the index, then we disable compound file for it.

See Also:

setNoCFSRatio(double), Constant Field Values

DEFAULT_MAX_CFS_SEGMENT_SIZE

public static final long DEFAULT_MAX_CFS_SEGMENT_SIZE

Default maxCFSSegmentSize value allows compound file for a segment of any size. The actual file format is still subject to noCFSRatio.

See Also:

setMaxCFSSegmentSizeMB(double), Constant Field Values

mergeFactor

protected int mergeFactor

How many segments to merge at a time.

minMergeSize

protected long minMergeSize

Any segments whose size is smaller than this value will be rounded up to this value. This ensures that tiny segments are aggressively merged.

maxMergeSize

protected long maxMergeSize

If the size of a segment exceeds this value then it will never be merged.

maxMergeSizeForForcedMerge

protected long maxMergeSizeForForcedMerge

If the size of a segment exceeds this value then it will never be merged during IndexWriter.forceMerge(int).

maxMergeDocs

protected int maxMergeDocs

If a segment has more than this many documents then it will never be merged.

noCFSRatio

protected double noCFSRatio

If the size of the merge segment exceeds this ratio of the total index size then it will remain in non-compound format even if setUseCompoundFile(boolean) is true.

maxCFSSegmentSize

protected long maxCFSSegmentSize

If the size of the merged segment exceeds this value then it will not use compound file format.

calibrateSizeByDeletes

protected boolean calibrateSizeByDeletes

If true, we pro-rate a segment's size by the percentage of non-deleted documents.

useCompoundFile

protected boolean useCompoundFile

True if new segments (flushed or merged) should use the compound file format. Note that large segments may sometimes still use non-compound format (see setNoCFSRatio(double).

Constructor Detail

LogMergePolicy

public LogMergePolicy()

Sole constructor. (For invocation by subclass constructors, typically implicit.)

Method Detail

verbose

protected boolean verbose()

Returns true if LMP is enabled in IndexWriter's infoStream.

getNoCFSRatio

public double getNoCFSRatio()

Returns current noCFSRatio.

See Also:

setNoCFSRatio(double)

setNoCFSRatio

public void setNoCFSRatio(double noCFSRatio)

If a merged segment will be more than this percentage of the total size of the index, leave the segment as non-compound file even if compound file is enabled. Set to 1.0 to always use CFS regardless of merge size.

message

protected void message(String message)

Print a debug message to IndexWriter's infoStream.

getMergeFactor

public int getMergeFactor()

Returns the number of segments that are merged at once and also controls the total number of segments allowed to accumulate in the index.

setMergeFactor

public void setMergeFactor(int mergeFactor)

Determines how often segment indices are merged by addDocument(). With smaller values, less RAM is used while indexing, and searches are faster, but indexing speed is slower. With larger values, more RAM is used during indexing, and while searches is slower, indexing is faster. Thus larger values (> 10) are best for batch index creation, and smaller values (< 10) for indices that are interactively maintained.

useCompoundFile

public boolean useCompoundFile(SegmentInfos infos,
                      SegmentInfoPerCommit mergedInfo)
                        throws IOException

Description copied from class: MergePolicy

Returns true if a new segment (regardless of its origin) should use the compound file format.

Specified by:

useCompoundFile in class MergePolicy

Throws:

IOException

setUseCompoundFile

public void setUseCompoundFile(boolean useCompoundFile)

Sets whether compound file format should be used for newly flushed and newly merged segments.

getUseCompoundFile

public boolean getUseCompoundFile()

Returns true if newly flushed and newly merge segments are written in compound file format. @see #setUseCompoundFile

setCalibrateSizeByDeletes

public void setCalibrateSizeByDeletes(boolean calibrateSizeByDeletes)

Sets whether the segment size should be calibrated by the number of deletes when choosing segments for merge.

getCalibrateSizeByDeletes

public boolean getCalibrateSizeByDeletes()

Returns true if the segment size should be calibrated by the number of deletes when choosing segments for merge.

close

public void close()

Description copied from class: MergePolicy

Release all resources for the policy.

Specified by:

close in interface Closeable

Specified by:

close in class MergePolicy

size

protected abstract long size(SegmentInfoPerCommit info)
                      throws IOException

Return the size of the provided SegmentInfoPerCommit.

Throws:

IOException

sizeDocs

protected long sizeDocs(SegmentInfoPerCommit info)
                 throws IOException

Return the number of documents in the provided SegmentInfoPerCommit, pro-rated by percentage of non-deleted documents if setCalibrateSizeByDeletes(boolean) is set.

Throws:

IOException

sizeBytes

protected long sizeBytes(SegmentInfoPerCommit info)
                  throws IOException

Return the byte size of the provided SegmentInfoPerCommit, pro-rated by percentage of non-deleted documents if setCalibrateSizeByDeletes(boolean) is set.

Throws:

IOException

isMerged

protected boolean isMerged(SegmentInfos infos,
               int maxNumSegments,
               Map<SegmentInfoPerCommit,Boolean> segmentsToMerge)
                    throws IOException

Returns true if the number of segments eligible for merging is less than or equal to the specified maxNumSegments.

Throws:

IOException

isMerged

protected boolean isMerged(SegmentInfoPerCommit info)
                    throws IOException

Returns true if this single info is already fully merged (has no pending norms or deletes, is in the same dir as the writer, and matches the current compound file setting

Throws:

IOException

findForcedMerges

public MergePolicy.MergeSpecification findForcedMerges(SegmentInfos infos,
                                              int maxNumSegments,
                                              Map<SegmentInfoPerCommit,Boolean> segmentsToMerge)
                                                throws IOException

Returns the merges necessary to merge the index down to a specified number of segments. This respects the maxMergeSizeForForcedMerge setting. By default, and assuming maxNumSegments=1, only one segment will be left in the index, where that segment has no deletions pending nor separate norms, and it is in compound file format if the current useCompoundFile setting is true. This method returns multiple merges (mergeFactor at a time) so the MergeScheduler in use may make use of concurrency.

Specified by:

findForcedMerges in class MergePolicy

Parameters:

infos - the total set of segments in the index

maxNumSegments - requested maximum number of segments in the index (currently this is always 1)

segmentsToMerge - contains the specific SegmentInfo instances that must be merged away. This may be a subset of all SegmentInfos. If the value is True for a given SegmentInfo, that means this segment was an original segment present in the to-be-merged index; else, it was a segment produced by a cascaded merge.

Throws:

IOException

findForcedDeletesMerges

public MergePolicy.MergeSpecification findForcedDeletesMerges(SegmentInfos segmentInfos)
                                                       throws IOException

Finds merges necessary to force-merge all deletes from the index. We simply merge adjacent segments that have deletes, up to mergeFactor at a time.

Specified by:

findForcedDeletesMerges in class MergePolicy

Parameters:

segmentInfos - the total set of segments in the index

Throws:

IOException

findMerges

public MergePolicy.MergeSpecification findMerges(MergePolicy.MergeTrigger mergeTrigger,
                                        SegmentInfos infos)
                                          throws IOException

Checks if any merges are now necessary and returns a MergePolicy.MergeSpecification if so. A merge is necessary when there are more than setMergeFactor(int) segments at a given level. When multiple levels have too many segments, this method will return multiple merges, allowing the MergeScheduler to use concurrency.

Specified by:

findMerges in class MergePolicy

Parameters:

mergeTrigger - the event that triggered the merge

infos - the total set of segments in the index

Throws:

IOException

setMaxMergeDocs

public void setMaxMergeDocs(int maxMergeDocs)

Determines the largest segment (measured by document count) that may be merged with other segments. Small values (e.g., less than 10,000) are best for interactive indexing, as this limits the length of pauses while indexing to a few seconds. Larger values are best for batched indexing and speedier searches.

The default value is Integer.MAX_VALUE.

The default merge policy (LogByteSizeMergePolicy) also allows you to set this limit by net size (in MB) of the segment, using LogByteSizeMergePolicy.setMaxMergeMB(double).

getMaxMergeDocs

public int getMaxMergeDocs()

Returns the largest segment (measured by document count) that may be merged with other segments.

See Also:

setMaxMergeDocs(int)

toString

public String toString()

Overrides:

toString in class Object

getMaxCFSSegmentSizeMB

public final double getMaxCFSSegmentSizeMB()

Returns the largest size allowed for a compound file segment

setMaxCFSSegmentSizeMB

public final void setMaxCFSSegmentSizeMB(double v)

If a merged segment will be more than this value, leave the segment as non-compound file even if compound file is enabled. Set this to Double.POSITIVE_INFINITY (default) and noCFSRatio to 1.0 to always use CFS regardless of merge size.