Class NRTCachingDirectory
- All Implemented Interfaces:
Closeable
,AutoCloseable
,Accountable
This class is likely only useful in a near-real-time context, where indexing rate is lowish but reopen rate is highish, resulting in many tiny files being written. This directory keeps such segments (as well as the segments produced by merging them, as long as they are small enough), in RAM.
This is safe to use: when your app calls {IndexWriter#commit}, all cached files will be flushed from the cached and sync'd.
Here's a simple example usage:
Directory fsDir = FSDirectory.open(new File("/path/to/index").toPath()); NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(fsDir, 5.0, 60.0); IndexWriterConfig conf = new IndexWriterConfig(analyzer); IndexWriter writer = new IndexWriter(cachedFSDir, conf);
This will cache all newly flushed segments, all merges whose expected segment size is
<= 5 MB
, unless the net cached bytes exceeds 60 MB at which point all writes will not be cached
(until the net bytes falls below 60 MB).
- WARNING: This API is experimental and might change in incompatible ways in the next release.
-
Field Summary
Fields inherited from class org.apache.lucene.store.FilterDirectory
in
Fields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE
-
Constructor Summary
ConstructorsConstructorDescriptionNRTCachingDirectory
(Directory delegate, double maxMergeSizeMB, double maxCachedMB) We will cache a newly created output if 1) it's a flush or a merge and the estimated size of the merged segment is<= maxMergeSizeMB
, and 2) the total cached bytes is<= maxCachedMB
-
Method Summary
Modifier and TypeMethodDescriptionvoid
close()
Close this directory, which flushes any cached files to the delegate and then closes the delegate.createOutput
(String name, IOContext context) Creates a new, empty file in the directory and returns anIndexOutput
instance for appending data to this file.createTempOutput
(String prefix, String suffix, IOContext context) Creates a new, empty, temporary file in the directory and returns anIndexOutput
instance for appending data to this file.void
deleteFile
(String name) Removes an existing file in the directory.protected boolean
doCacheWrite
(String name, IOContext context) Subclass can override this to customize logic; return true if this file should be written to the RAM-based cache first.long
fileLength
(String name) Returns the byte length of a file in the directory.String[]
listAll()
Returns names of all files stored in this directory.String[]
Opens a stream for reading an existing file.long
Return the memory usage of this object in bytes.void
Renamessource
file todest
file wheredest
must not already exist in the directory.void
sync
(Collection<String> fileNames) Ensures that any writes to these files are moved to stable storage (made durable).toString()
Methods inherited from class org.apache.lucene.store.FilterDirectory
ensureOpen, getDelegate, getPendingDeletions, obtainLock, syncMetaData, unwrap
Methods inherited from class org.apache.lucene.store.Directory
copyFrom, getTempFileName, openChecksumInput
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
Constructor Details
-
NRTCachingDirectory
We will cache a newly created output if 1) it's a flush or a merge and the estimated size of the merged segment is<= maxMergeSizeMB
, and 2) the total cached bytes is<= maxCachedMB
-
-
Method Details
-
toString
- Overrides:
toString
in classFilterDirectory
-
listAll
Description copied from class:Directory
Returns names of all files stored in this directory. The output must be in sorted (UTF-16, java'sString.compareTo(java.lang.String)
) order.- Overrides:
listAll
in classFilterDirectory
- Throws:
IOException
- in case of I/O error
-
deleteFile
Description copied from class:Directory
Removes an existing file in the directory.This method must throw either
NoSuchFileException
orFileNotFoundException
ifname
points to a non-existing file.- Overrides:
deleteFile
in classFilterDirectory
- Parameters:
name
- the name of an existing file.- Throws:
IOException
- in case of I/O error
-
fileLength
Description copied from class:Directory
Returns the byte length of a file in the directory.This method must throw either
NoSuchFileException
orFileNotFoundException
ifname
points to a non-existing file.- Overrides:
fileLength
in classFilterDirectory
- Parameters:
name
- the name of an existing file.- Throws:
IOException
- in case of I/O error
-
listCachedFiles
-
createOutput
Description copied from class:Directory
Creates a new, empty file in the directory and returns anIndexOutput
instance for appending data to this file.This method must throw
FileAlreadyExistsException
if the file already exists.- Overrides:
createOutput
in classFilterDirectory
- Parameters:
name
- the name of the file to create.- Throws:
IOException
- in case of I/O error
-
sync
Description copied from class:Directory
Ensures that any writes to these files are moved to stable storage (made durable).Lucene uses this to properly commit changes to the index, to prevent a machine/OS crash from corrupting the index.
- Overrides:
sync
in classFilterDirectory
- Throws:
IOException
- See Also:
-
rename
Description copied from class:Directory
Renamessource
file todest
file wheredest
must not already exist in the directory.It is permitted for this operation to not be truly atomic, for example both
source
anddest
can be visible temporarily inDirectory.listAll()
. However, the implementation of this method must ensure the content ofdest
appears as the entiresource
atomically. So oncedest
is visible for readers, the entire content of previoussource
is visible.This method is used by IndexWriter to publish commits.
- Overrides:
rename
in classFilterDirectory
- Throws:
IOException
-
openInput
Description copied from class:Directory
Opens a stream for reading an existing file.This method must throw either
NoSuchFileException
orFileNotFoundException
ifname
points to a non-existing file.- Overrides:
openInput
in classFilterDirectory
- Parameters:
name
- the name of an existing file.- Throws:
IOException
- in case of I/O error
-
close
Close this directory, which flushes any cached files to the delegate and then closes the delegate.- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Overrides:
close
in classFilterDirectory
- Throws:
IOException
-
doCacheWrite
Subclass can override this to customize logic; return true if this file should be written to the RAM-based cache first. -
createTempOutput
public IndexOutput createTempOutput(String prefix, String suffix, IOContext context) throws IOException Description copied from class:Directory
Creates a new, empty, temporary file in the directory and returns anIndexOutput
instance for appending data to this file.The temporary file name (accessible via
IndexOutput.getName()
) will start withprefix
, end withsuffix
and have a reserved file extension.tmp
.- Overrides:
createTempOutput
in classFilterDirectory
- Throws:
IOException
-
ramBytesUsed
public long ramBytesUsed()Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.- Specified by:
ramBytesUsed
in interfaceAccountable
-