org.apache.lucene.index

Class TieredMergePolicy

java.lang.Object

org.apache.lucene.index.MergePolicy

org.apache.lucene.index.TieredMergePolicy

All Implemented Interfaces:

Closeable, Cloneable

public class TieredMergePolicy
extends MergePolicy

Merges segments of approximately equal size, subject to an allowed number of segments per tier. This is similar to LogByteSizeMergePolicy, except this merge policy is able to merge non-adjacent segment, and separates how many segments are merged at once (setMaxMergeAtOnce(int)) from how many segments are allowed per tier (setSegmentsPerTier(double)). This merge policy also does not over-merge (i.e. cascade merges).

For normal merging, this policy first computes a "budget" of how many segments are allowed to be in the index. If the index is over-budget, then the policy sorts segments by decreasing size (pro-rating by percent deletes), and then finds the least-cost merge. Merge cost is measured by a combination of the "skew" of the merge (size of largest segment divided by smallest segment), total merge size and percent deletes reclaimed, so that merges with lower skew, smaller size and those reclaiming more deletes, are favored.

If a merge will produce a segment that's larger than setMaxMergedSegmentMB(double), then the policy will merge fewer segments (down to 1 at once, if that one has deletions) to keep the segment size under budget.

NOTE: this policy freely merges non-adjacent segments; if this is a problem, use LogMergePolicy.

NOTE: This policy always merges by byte size of the segments, always pro-rates by percent deletes, and does not apply any maximum segment size during forceMerge (unlike LogByteSizeMergePolicy).

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	TieredMergePolicy.MergeScore Holds score and explanation for a single candidate merge.

Nested classes/interfaces inherited from class org.apache.lucene.index.MergePolicy

MergePolicy.MergeAbortedException, MergePolicy.MergeException, MergePolicy.MergeSpecification, MergePolicy.MergeTrigger, MergePolicy.OneMerge

Field Summary

Fields inherited from class org.apache.lucene.index.MergePolicy

writer

Constructor Summary

Constructors

Constructor and Description
TieredMergePolicy() Sole constructor, setting all settings to their defaults.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Release all resources for the policy.
MergePolicy.MergeSpecification	findForcedDeletesMerges(SegmentInfos infos) Determine what set of merge operations is necessary in order to expunge all deletes from the index.
MergePolicy.MergeSpecification	findForcedMerges(SegmentInfos infos, int maxSegmentCount, Map<SegmentInfoPerCommit,Boolean> segmentsToMerge) Determine what set of merge operations is necessary in order to merge to <= the specified segment count.
MergePolicy.MergeSpecification	findMerges(MergePolicy.MergeTrigger mergeTrigger, SegmentInfos infos) Determine what set of merge operations are now necessary on the index.
double	getFloorSegmentMB() Returns the current floorSegmentMB.
double	getForceMergeDeletesPctAllowed() Returns the current forceMergeDeletesPctAllowed setting.
double	getMaxCFSSegmentSizeMB() Returns the largest size allowed for a compound file segment
int	getMaxMergeAtOnce() Returns the current maxMergeAtOnce setting.
int	getMaxMergeAtOnceExplicit() Returns the current maxMergeAtOnceExplicit setting.
double	getMaxMergedSegmentMB() Returns the current maxMergedSegmentMB setting.
double	getNoCFSRatio() Returns the current noCFSRatio setting.
double	getReclaimDeletesWeight() See setReclaimDeletesWeight(double).
double	getSegmentsPerTier() Returns the current segmentsPerTier setting.
boolean	getUseCompoundFile() Returns the current useCompoundFile setting.
protected TieredMergePolicy.MergeScore	score(List<SegmentInfoPerCommit> candidate, boolean hitTooLarge, long mergingBytes) Expert: scores one merge; subclasses can override.
TieredMergePolicy	setFloorSegmentMB(double v) Segments smaller than this are "rounded up" to this size, ie treated as equal (floor) size for merge selection.
TieredMergePolicy	setForceMergeDeletesPctAllowed(double v) When forceMergeDeletes is called, we only merge away a segment if its delete percentage is over this threshold.
TieredMergePolicy	setMaxCFSSegmentSizeMB(double v) If a merged segment will be more than this value, leave the segment as non-compound file even if compound file is enabled.
TieredMergePolicy	setMaxMergeAtOnce(int v) Maximum number of segments to be merged at a time during "normal" merging.
TieredMergePolicy	setMaxMergeAtOnceExplicit(int v) Maximum number of segments to be merged at a time, during forceMerge or forceMergeDeletes.
TieredMergePolicy	setMaxMergedSegmentMB(double v) Maximum sized segment to produce during normal merging.
TieredMergePolicy	setNoCFSRatio(double noCFSRatio) If a merged segment will be more than this percentage of the total size of the index, leave the segment as non-compound file even if compound file is enabled.
TieredMergePolicy	setReclaimDeletesWeight(double v) Controls how aggressively merges that reclaim more deletions are favored.
TieredMergePolicy	setSegmentsPerTier(double v) Sets the allowed number of segments per tier.
TieredMergePolicy	setUseCompoundFile(boolean useCompoundFile) Sets whether compound file format should be used for newly flushed and newly merged segments.
String	toString()
boolean	useCompoundFile(SegmentInfos infos, SegmentInfoPerCommit mergedInfo) Returns true if a new segment (regardless of its origin) should use the compound file format.

Methods inherited from class org.apache.lucene.index.MergePolicy

clone, setIndexWriter

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

TieredMergePolicy

public TieredMergePolicy()

Sole constructor, setting all settings to their defaults.

Method Detail

setMaxMergeAtOnce

public TieredMergePolicy setMaxMergeAtOnce(int v)

Maximum number of segments to be merged at a time during "normal" merging. For explicit merging (eg, forceMerge or forceMergeDeletes was called), see setMaxMergeAtOnceExplicit(int). Default is 10.

getMaxMergeAtOnce

public int getMaxMergeAtOnce()

Returns the current maxMergeAtOnce setting.

See Also:

setMaxMergeAtOnce(int)

setMaxMergeAtOnceExplicit

public TieredMergePolicy setMaxMergeAtOnceExplicit(int v)

Maximum number of segments to be merged at a time, during forceMerge or forceMergeDeletes. Default is 30.

getMaxMergeAtOnceExplicit

public int getMaxMergeAtOnceExplicit()

Returns the current maxMergeAtOnceExplicit setting.

See Also:

setMaxMergeAtOnceExplicit(int)

setMaxMergedSegmentMB

public TieredMergePolicy setMaxMergedSegmentMB(double v)

Maximum sized segment to produce during normal merging. This setting is approximate: the estimate of the merged segment size is made by summing sizes of to-be-merged segments (compensating for percent deleted docs). Default is 5 GB.

getMaxMergedSegmentMB

public double getMaxMergedSegmentMB()

Returns the current maxMergedSegmentMB setting.

See Also:

getMaxMergedSegmentMB()

setReclaimDeletesWeight

public TieredMergePolicy setReclaimDeletesWeight(double v)

Controls how aggressively merges that reclaim more deletions are favored. Higher values favor selecting merges that reclaim deletions. A value of 0.0 means deletions don't impact merge selection.

getReclaimDeletesWeight

public double getReclaimDeletesWeight()

See setReclaimDeletesWeight(double).

setFloorSegmentMB

public TieredMergePolicy setFloorSegmentMB(double v)

Segments smaller than this are "rounded up" to this size, ie treated as equal (floor) size for merge selection. This is to prevent frequent flushing of tiny segments from allowing a long tail in the index. Default is 2 MB.

getFloorSegmentMB

public double getFloorSegmentMB()

Returns the current floorSegmentMB.

See Also:

setFloorSegmentMB(double)

setForceMergeDeletesPctAllowed

public TieredMergePolicy setForceMergeDeletesPctAllowed(double v)

When forceMergeDeletes is called, we only merge away a segment if its delete percentage is over this threshold. Default is 10%.

getForceMergeDeletesPctAllowed

public double getForceMergeDeletesPctAllowed()

Returns the current forceMergeDeletesPctAllowed setting.

See Also:

setForceMergeDeletesPctAllowed(double)

setSegmentsPerTier

public TieredMergePolicy setSegmentsPerTier(double v)

Sets the allowed number of segments per tier. Smaller values mean more merging but fewer segments.

NOTE: this value should be >= the setMaxMergeAtOnce(int) otherwise you'll force too much merging to occur.

Default is 10.0.

getSegmentsPerTier

public double getSegmentsPerTier()

Returns the current segmentsPerTier setting.

See Also:

setSegmentsPerTier(double)

setUseCompoundFile

public TieredMergePolicy setUseCompoundFile(boolean useCompoundFile)

Sets whether compound file format should be used for newly flushed and newly merged segments. Default true.

getUseCompoundFile

public boolean getUseCompoundFile()

Returns the current useCompoundFile setting.

See Also:

setUseCompoundFile(boolean)

setNoCFSRatio

public TieredMergePolicy setNoCFSRatio(double noCFSRatio)

If a merged segment will be more than this percentage of the total size of the index, leave the segment as non-compound file even if compound file is enabled. Set to 1.0 to always use CFS regardless of merge size. Default is 0.1.

getNoCFSRatio

public double getNoCFSRatio()

Returns the current noCFSRatio setting.

See Also:

setNoCFSRatio(double)

findMerges

public MergePolicy.MergeSpecification findMerges(MergePolicy.MergeTrigger mergeTrigger,
                                        SegmentInfos infos)
                                          throws IOException

Description copied from class: MergePolicy

Determine what set of merge operations are now necessary on the index. IndexWriter calls this whenever there is a change to the segments. This call is always synchronized on the IndexWriter instance so only one thread at a time will call this method.

Specified by:

findMerges in class MergePolicy

Parameters:

mergeTrigger - the event that triggered the merge

infos - the total set of segments in the index

Throws:

IOException

score

protected TieredMergePolicy.MergeScore score(List<SegmentInfoPerCommit> candidate,
                                 boolean hitTooLarge,
                                 long mergingBytes)
                                      throws IOException

Expert: scores one merge; subclasses can override.

Throws:

IOException

findForcedMerges

public MergePolicy.MergeSpecification findForcedMerges(SegmentInfos infos,
                                              int maxSegmentCount,
                                              Map<SegmentInfoPerCommit,Boolean> segmentsToMerge)
                                                throws IOException

Description copied from class: MergePolicy

Determine what set of merge operations is necessary in order to merge to <= the specified segment count. IndexWriter calls this when its IndexWriter.forceMerge(int) method is called. This call is always synchronized on the IndexWriter instance so only one thread at a time will call this method.

Specified by:

findForcedMerges in class MergePolicy

Parameters:

infos - the total set of segments in the index

maxSegmentCount - requested maximum number of segments in the index (currently this is always 1)

segmentsToMerge - contains the specific SegmentInfo instances that must be merged away. This may be a subset of all SegmentInfos. If the value is True for a given SegmentInfo, that means this segment was an original segment present in the to-be-merged index; else, it was a segment produced by a cascaded merge.

Throws:

IOException

findForcedDeletesMerges

public MergePolicy.MergeSpecification findForcedDeletesMerges(SegmentInfos infos)
                                                       throws IOException

Description copied from class: MergePolicy

Determine what set of merge operations is necessary in order to expunge all deletes from the index.

Specified by:

findForcedDeletesMerges in class MergePolicy

Parameters:

infos - the total set of segments in the index

Throws:

IOException

useCompoundFile

public boolean useCompoundFile(SegmentInfos infos,
                      SegmentInfoPerCommit mergedInfo)
                        throws IOException

Description copied from class: MergePolicy

Returns true if a new segment (regardless of its origin) should use the compound file format.

Specified by:

useCompoundFile in class MergePolicy

Throws:

IOException

close

public void close()

Description copied from class: MergePolicy

Release all resources for the policy.

Specified by:

close in interface Closeable

Specified by:

close in class MergePolicy

toString

public String toString()

Overrides:

toString in class Object

getMaxCFSSegmentSizeMB

public final double getMaxCFSSegmentSizeMB()

Returns the largest size allowed for a compound file segment

setMaxCFSSegmentSizeMB

public final TieredMergePolicy setMaxCFSSegmentSizeMB(double v)

If a merged segment will be more than this value, leave the segment as non-compound file even if compound file is enabled. Set this to Double.POSITIVE_INFINITY (default) and noCFSRatio to 1.0 to always use CFS regardless of merge size.