public class NRTCachingDirectory extends FilterDirectory implements Accountable
RAMDirectory
around any provided delegate directory, to
be used during NRT search.
This class is likely only useful in a near-real-time context, where indexing rate is lowish but reopen rate is highish, resulting in many tiny files being written. This directory keeps such segments (as well as the segments produced by merging them, as long as they are small enough), in RAM.
This is safe to use: when your app calls {IndexWriter#commit}, all cached files will be flushed from the cached and sync'd.
Here's a simple example usage:
Directory fsDir = FSDirectory.open(new File("/path/to/index").toPath());
NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(fsDir, 5.0, 60.0);
IndexWriterConfig conf = new IndexWriterConfig(analyzer);
IndexWriter writer = new IndexWriter(cachedFSDir, conf);
This will cache all newly flushed segments, all merges
whose expected segment size is <= 5 MB, unless the net
cached bytes exceeds 60 MB at which point all writes will
not be cached (until the net bytes falls below 60 MB).
in| Constructor and Description |
|---|
NRTCachingDirectory(Directory delegate,
double maxMergeSizeMB,
double maxCachedMB)
We will cache a newly created output if 1) it's a
flush or a merge and the estimated size of the merged segment is
<= maxMergeSizeMB, and 2) the total cached bytes is
<= maxCachedMB |
| Modifier and Type | Method and Description |
|---|---|
void |
close()
Close this directory, which flushes any cached files
to the delegate and then closes the delegate.
|
IndexOutput |
createOutput(String name,
IOContext context)
Creates a new, empty file in the directory with the given name.
|
IndexOutput |
createTempOutput(String prefix,
String suffix,
IOContext context)
Creates a new, empty file for writing in the directory, with a
temporary file name including prefix and suffix, ending with the
reserved extension
.tmp. |
void |
deleteFile(String name)
Removes an existing file in the directory.
|
protected boolean |
doCacheWrite(String name,
IOContext context)
Subclass can override this to customize logic; return
true if this file should be written to the RAMDirectory.
|
long |
fileLength(String name)
Returns the length of a file in the directory.
|
Collection<Accountable> |
getChildResources()
Returns nested resources of this class.
|
String[] |
listAll()
Returns an array of strings, one for each entry in the directory, in sorted (UTF16, java's String.compare) order.
|
String[] |
listCachedFiles() |
IndexInput |
openInput(String name,
IOContext context)
Returns a stream reading an existing file.
|
long |
ramBytesUsed()
Return the memory usage of this object in bytes.
|
void |
renameFile(String source,
String dest)
Renames
source to dest as an atomic operation,
where dest does not yet exist in the directory. |
void |
sync(Collection<String> fileNames)
Ensure that any writes to these files are moved to
stable storage.
|
String |
toString() |
getDelegate, obtainLock, unwrapcopyFrom, ensureOpen, openChecksumInputpublic NRTCachingDirectory(Directory delegate, double maxMergeSizeMB, double maxCachedMB)
<= maxMergeSizeMB, and 2) the total cached bytes is
<= maxCachedMBpublic String toString()
toString in class FilterDirectorypublic String[] listAll() throws IOException
DirectorylistAll in class FilterDirectoryIOException - in case of IO errorpublic void deleteFile(String name) throws IOException
DirectorydeleteFile in class FilterDirectoryIOExceptionpublic long fileLength(String name) throws IOException
DirectoryFileNotFoundException or NoSuchFileException
if the file does not exist.
fileLength in class FilterDirectoryname - the name of the file for which to return the length.IOException - if there was an IO error while retrieving the file's
length.public String[] listCachedFiles()
public IndexOutput createOutput(String name, IOContext context) throws IOException
DirectorycreateOutput in class FilterDirectoryIOExceptionpublic void sync(Collection<String> fileNames) throws IOException
Directorysync in class FilterDirectoryIOExceptionpublic void renameFile(String source, String dest) throws IOException
Directorysource to dest as an atomic operation,
where dest does not yet exist in the directory.
Notes: This method is used by IndexWriter to publish commits.
It is ok if this operation is not truly atomic, for example
both source and dest can be visible temporarily.
It is just important that the contents of dest appear
atomically, or an exception is thrown.
renameFile in class FilterDirectoryIOExceptionpublic IndexInput openInput(String name, IOContext context) throws IOException
DirectoryThrows FileNotFoundException or NoSuchFileException
if the file does not exist.
openInput in class FilterDirectoryIOExceptionpublic void close()
throws IOException
close in interface Closeableclose in interface AutoCloseableclose in class FilterDirectoryIOExceptionprotected boolean doCacheWrite(String name, IOContext context)
public IndexOutput createTempOutput(String prefix, String suffix, IOContext context) throws IOException
Directory.tmp. Use
IndexOutput.getName() to see what name was used.createTempOutput in class FilterDirectoryIOExceptionpublic long ramBytesUsed()
AccountableramBytesUsed in interface Accountablepublic Collection<Accountable> getChildResources()
AccountablegetChildResources in interface AccountableAccountablesCopyright © 2000-2016 Apache Software Foundation. All Rights Reserved.