public class MMapDirectory extends FSDirectory
Directory
implementation that uses
mmap for reading, and FSDirectory.FSIndexOutput
for writing.
NOTE: memory mapping uses up a portion of the
virtual memory address space in your process equal to the
size of the file being mapped. Before using this class,
be sure your have plenty of virtual address space, e.g. by
using a 64 bit JRE, or a 32 bit JRE with indexes that are
guaranteed to fit within the address space.
On 32 bit platforms also consult MMapDirectory(Path, LockFactory, int)
if you have problems with mmap failing because of fragmented
address space. If you get an OutOfMemoryException, it is recommended
to reduce the chunk size, until it works.
Due to
this bug in Sun's JRE, MMapDirectory's IndexInput.close()
is unable to close the underlying OS file handle. Only when GC
finally collects the underlying objects, which could be quite
some time later, will the file handle be closed.
This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of.
This class supplies the workaround mentioned in the bug report
(see setUseUnmap(boolean)
), which may fail on
non-Oracle/OpenJDK JVMs. It forcefully unmaps the buffer on close by using
an undocumented internal cleanup functionality. If
UNMAP_SUPPORTED
is true
, the workaround
will be automatically enabled (with no guarantees; if you discover
any problems, you can disable it).
NOTE: Accessing this class either directly or
indirectly from a thread while it's interrupted can close the
underlying channel immediately if at the same time the thread is
blocked on IO. The channel will remain closed and subsequent access
to MMapDirectory
will throw a ClosedChannelException
. If
your application uses either Thread.interrupt()
or
Future.cancel(boolean)
you should use the legacy RAFDirectory
from the Lucene misc
module in favor of MMapDirectory
.
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_MAX_CHUNK_SIZE
Default max chunk size.
|
static String |
UNMAP_NOT_SUPPORTED_REASON
if
UNMAP_SUPPORTED is false , this contains the reason why unmapping is not supported. |
static boolean |
UNMAP_SUPPORTED
true , if this platform supports unmapping mmapped files. |
directory
isOpen, lockFactory
Constructor and Description |
---|
MMapDirectory(Path path)
Create a new MMapDirectory for the named location and
FSLockFactory.getDefault() . |
MMapDirectory(Path path,
int maxChunkSize)
Create a new MMapDirectory for the named location and
FSLockFactory.getDefault() . |
MMapDirectory(Path path,
LockFactory lockFactory)
Create a new MMapDirectory for the named location.
|
MMapDirectory(Path path,
LockFactory lockFactory,
int maxChunkSize)
Create a new MMapDirectory for the named location, specifying the
maximum chunk size used for memory mapping.
|
Modifier and Type | Method and Description |
---|---|
int |
getMaxChunkSize()
Returns the current mmap chunk size.
|
boolean |
getPreload()
Returns
true if mapped pages should be loaded. |
boolean |
getUseUnmap()
Returns
true , if the unmap workaround is enabled. |
IndexInput |
openInput(String name,
IOContext context)
Creates an IndexInput for the file with the given name.
|
void |
setPreload(boolean preload)
Set to
true to ask mapped pages to be loaded
into physical memory on init. |
void |
setUseUnmap(boolean useUnmapHack)
This method enables the workaround for unmapping the buffers
from address space after closing
IndexInput , that is
mentioned in the bug report. |
checkPendingDeletions, close, createOutput, createTempOutput, deleteFile, deletePendingFiles, ensureCanRead, fileLength, fsync, getDirectory, listAll, listAll, open, open, rename, sync, syncMetaData, toString
ensureOpen, obtainLock
copyFrom, openChecksumInput, renameFile
public static final int DEFAULT_MAX_CHUNK_SIZE
MMapDirectory(Path, LockFactory, int)
public static final boolean UNMAP_SUPPORTED
true
, if this platform supports unmapping mmapped files.public static final String UNMAP_NOT_SUPPORTED_REASON
UNMAP_SUPPORTED
is false
, this contains the reason why unmapping is not supported.public MMapDirectory(Path path, LockFactory lockFactory) throws IOException
path
- the path of the directorylockFactory
- the lock factory to useIOException
- if there is a low-level I/O errorpublic MMapDirectory(Path path) throws IOException
FSLockFactory.getDefault()
.
The directory is created at the named location if it does not yet exist.path
- the path of the directoryIOException
- if there is a low-level I/O errorpublic MMapDirectory(Path path, int maxChunkSize) throws IOException
FSLockFactory.getDefault()
.
The directory is created at the named location if it does not yet exist.path
- the path of the directorymaxChunkSize
- maximum chunk size (default is 1 GiBytes for
64 bit JVMs and 256 MiBytes for 32 bit JVMs) used for memory mapping.IOException
- if there is a low-level I/O errorpublic MMapDirectory(Path path, LockFactory lockFactory, int maxChunkSize) throws IOException
Especially on 32 bit platform, the address space can be very fragmented,
so large index files cannot be mapped. Using a lower chunk size makes
the directory implementation a little bit slower (as the correct chunk
may be resolved on lots of seeks) but the chance is higher that mmap
does not fail. On 64 bit Java platforms, this parameter should always
be 1 << 30
, as the address space is big enough.
Please note: The chunk size is always rounded down to a power of 2.
path
- the path of the directorylockFactory
- the lock factory to use, or null for the default
(NativeFSLockFactory
);maxChunkSize
- maximum chunk size (default is 1 GiBytes for
64 bit JVMs and 256 MiBytes for 32 bit JVMs) used for memory mapping.IOException
- if there is a low-level I/O errorpublic void setUseUnmap(boolean useUnmapHack)
IndexInput
, that is
mentioned in the bug report. This hack may fail on non-Oracle/OpenJDK JVMs.
It forcefully unmaps the buffer on close by using
an undocumented internal cleanup functionality.
NOTE: Enabling this is completely unsupported
by Java and may lead to JVM crashes if IndexInput
is closed while another thread is still accessing it (SIGSEGV).
To enable the hack, the following requirements need to be
fulfilled: The used JVM must be Oracle Java / OpenJDK 8
(preliminary support for Java 9 EA build 150+ was added with Lucene 6.4).
In addition, the following permissions need to be granted
to lucene-core.jar
in your
policy file:
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
permission java.lang.RuntimePermission "accessClassInPackage.sun.misc";
IllegalArgumentException
- if UNMAP_SUPPORTED
is false
and the workaround cannot be enabled.
The exception message also contains an explanation why the hack
cannot be enabled (e.g., missing permissions).public boolean getUseUnmap()
true
, if the unmap workaround is enabled.setUseUnmap(boolean)
public void setPreload(boolean preload)
true
to ask mapped pages to be loaded
into physical memory on init. The behavior is best-effort
and operating system dependent.MappedByteBuffer.load()
public boolean getPreload()
true
if mapped pages should be loaded.setPreload(boolean)
public final int getMaxChunkSize()
MMapDirectory(Path, LockFactory, int)
public IndexInput openInput(String name, IOContext context) throws IOException
openInput
in class Directory
IOException
Copyright © 2000-2017 Apache Software Foundation. All Rights Reserved.