Class MMapDirectory
- java.lang.Object
-
- org.apache.lucene.store.Directory
-
- org.apache.lucene.store.BaseDirectory
-
- org.apache.lucene.store.FSDirectory
-
- org.apache.lucene.store.MMapDirectory
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
public class MMapDirectory extends FSDirectory
File-basedDirectory
implementation that uses mmap for reading, andFSDirectory.FSIndexOutput
for writing.NOTE: memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure your have plenty of virtual address space, e.g. by using a 64 bit JRE, or a 32 bit JRE with indexes that are guaranteed to fit within the address space. On 32 bit platforms also consult
MMapDirectory(Path, LockFactory, long)
if you have problems with mmap failing because of fragmented address space. If you get an OutOfMemoryException, it is recommended to reduce the chunk size, until it works.Due to this bug in Sun's JRE, MMapDirectory's
IndexInput.close()
is unable to close the underlying OS file handle. Only when GC finally collects the underlying objects, which could be quite some time later, will the file handle be closed.This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of.
This class supplies the workaround mentioned in the bug report (see
setUseUnmap(boolean)
), which may fail on non-Oracle/OpenJDK JVMs. It forcefully unmaps the buffer on close by using an undocumented internal cleanup functionality. IfUNMAP_SUPPORTED
istrue
, the workaround will be automatically enabled (with no guarantees; if you discover any problems, you can disable it).On Java 19 with
--enable-preview
command line setting, this class will use the modernMemorySegment
API which allows to safely unmap.NOTE: Accessing this class either directly or indirectly from a thread while it's interrupted can close the underlying channel immediately if at the same time the thread is blocked on IO. The channel will remain closed and subsequent access to
MMapDirectory
will throw aClosedChannelException
. If your application uses eitherThread.interrupt()
orFuture.cancel(boolean)
you should use the legacyRAFDirectory
from the Lucenemisc
module in favor ofMMapDirectory
.NOTE: If your application requires external synchronization, you should not synchronize on the
MMapDirectory
instance as this may cause deadlock; use your own (non-Lucene) objects instead.- See Also:
- Blog post about MMapDirectory
-
-
Field Summary
Fields Modifier and Type Field Description static long
DEFAULT_MAX_CHUNK_SIZE
Default max chunk size: 16 GiBytes for 64 bit Java 19 JVMs running with--enable-preview
as command line parameter 1 GiBytes for other 64 bit JVMs 256 MiBytes for 32 bit JVMsstatic String
UNMAP_NOT_SUPPORTED_REASON
ifUNMAP_SUPPORTED
isfalse
, this contains the reason why unmapping is not supported.static boolean
UNMAP_SUPPORTED
true
, if this platform supports unmapping mmapped files.-
Fields inherited from class org.apache.lucene.store.FSDirectory
directory
-
Fields inherited from class org.apache.lucene.store.BaseDirectory
isOpen, lockFactory
-
-
Constructor Summary
Constructors Constructor Description MMapDirectory(Path path)
Create a new MMapDirectory for the named location andFSLockFactory.getDefault()
.MMapDirectory(Path path, int maxChunkSize)
Deprecated.useMMapDirectory(Path, long)
instead.MMapDirectory(Path path, long maxChunkSize)
Create a new MMapDirectory for the named location andFSLockFactory.getDefault()
.MMapDirectory(Path path, LockFactory lockFactory)
Create a new MMapDirectory for the named location.MMapDirectory(Path path, LockFactory lockFactory, int maxChunkSize)
Deprecated.useMMapDirectory(Path, LockFactory, long)
instead.MMapDirectory(Path path, LockFactory lockFactory, long maxChunkSize)
Create a new MMapDirectory for the named location, specifying the maximum chunk size used for memory mapping.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description long
getMaxChunkSize()
Returns the current mmap chunk size.boolean
getPreload()
Returnstrue
if mapped pages should be loaded.boolean
getUseUnmap()
Returnstrue
, if the unmap workaround is enabled.IndexInput
openInput(String name, IOContext context)
Creates an IndexInput for the file with the given name.void
setPreload(boolean preload)
Set totrue
to ask mapped pages to be loaded into physical memory on init.void
setUseUnmap(boolean useUnmapHack)
This method enables the workaround for unmapping the buffers from address space after closingIndexInput
, that is mentioned in the bug report.-
Methods inherited from class org.apache.lucene.store.FSDirectory
close, createOutput, createTempOutput, deleteFile, deletePendingFiles, ensureCanRead, fileLength, fsync, getDirectory, getPendingDeletions, listAll, listAll, open, open, rename, sync, syncMetaData, toString
-
Methods inherited from class org.apache.lucene.store.BaseDirectory
ensureOpen, obtainLock
-
Methods inherited from class org.apache.lucene.store.Directory
copyFrom, getTempFileName, openChecksumInput
-
-
-
-
Field Detail
-
DEFAULT_MAX_CHUNK_SIZE
public static final long DEFAULT_MAX_CHUNK_SIZE
Default max chunk size:- 16 GiBytes for 64 bit Java 19 JVMs running with
--enable-preview
as command line parameter - 1 GiBytes for other 64 bit JVMs
- 256 MiBytes for 32 bit JVMs
- 16 GiBytes for 64 bit Java 19 JVMs running with
-
UNMAP_SUPPORTED
public static final boolean UNMAP_SUPPORTED
true
, if this platform supports unmapping mmapped files.
-
UNMAP_NOT_SUPPORTED_REASON
public static final String UNMAP_NOT_SUPPORTED_REASON
ifUNMAP_SUPPORTED
isfalse
, this contains the reason why unmapping is not supported.
-
-
Constructor Detail
-
MMapDirectory
public MMapDirectory(Path path, LockFactory lockFactory) throws IOException
Create a new MMapDirectory for the named location. The directory is created at the named location if it does not yet exist.- Parameters:
path
- the path of the directorylockFactory
- the lock factory to use- Throws:
IOException
- if there is a low-level I/O error
-
MMapDirectory
public MMapDirectory(Path path) throws IOException
Create a new MMapDirectory for the named location andFSLockFactory.getDefault()
. The directory is created at the named location if it does not yet exist.- Parameters:
path
- the path of the directory- Throws:
IOException
- if there is a low-level I/O error
-
MMapDirectory
@Deprecated public MMapDirectory(Path path, int maxChunkSize) throws IOException
Deprecated.useMMapDirectory(Path, long)
instead.Create a new MMapDirectory for the named location andFSLockFactory.getDefault()
. The directory is created at the named location if it does not yet exist.- Throws:
IOException
-
MMapDirectory
public MMapDirectory(Path path, long maxChunkSize) throws IOException
Create a new MMapDirectory for the named location andFSLockFactory.getDefault()
. The directory is created at the named location if it does not yet exist.- Parameters:
path
- the path of the directorymaxChunkSize
- maximum chunk size (for default seeDEFAULT_MAX_CHUNK_SIZE
) used for memory mapping.- Throws:
IOException
- if there is a low-level I/O error
-
MMapDirectory
@Deprecated public MMapDirectory(Path path, LockFactory lockFactory, int maxChunkSize) throws IOException
Deprecated.useMMapDirectory(Path, LockFactory, long)
instead.Create a new MMapDirectory for the named location andFSLockFactory.getDefault()
. The directory is created at the named location if it does not yet exist.- Throws:
IOException
-
MMapDirectory
public MMapDirectory(Path path, LockFactory lockFactory, long maxChunkSize) throws IOException
Create a new MMapDirectory for the named location, specifying the maximum chunk size used for memory mapping. The directory is created at the named location if it does not yet exist.Especially on 32 bit platform, the address space can be very fragmented, so large index files cannot be mapped. Using a lower chunk size makes the directory implementation a little bit slower (as the correct chunk may be resolved on lots of seeks) but the chance is higher that mmap does not fail. On 64 bit Java platforms, this parameter should always be large (like 1 GiBytes, or even larger with Java 19), as the address space is big enough. If it is larger, fragmentation of address space increases, but number of file handles and mappings is lower for huge installations with many open indexes.
Please note: The chunk size is always rounded down to a power of 2.
- Parameters:
path
- the path of the directorylockFactory
- the lock factory to use, or null for the default (NativeFSLockFactory
);maxChunkSize
- maximum chunk size (for default seeDEFAULT_MAX_CHUNK_SIZE
) used for memory mapping.- Throws:
IOException
- if there is a low-level I/O error
-
-
Method Detail
-
setUseUnmap
public void setUseUnmap(boolean useUnmapHack)
This method enables the workaround for unmapping the buffers from address space after closingIndexInput
, that is mentioned in the bug report. This hack may fail on non-Oracle/OpenJDK JVMs. It forcefully unmaps the buffer on close by using an undocumented internal cleanup functionality.On Java 19 with
--enable-preview
command line setting, this class will use the modernMemorySegment
API which allows to safely unmap. The following warnings no longer apply in that case!NOTE: Enabling this is completely unsupported by Java and may lead to JVM crashes if
IndexInput
is closed while another thread is still accessing it (SIGSEGV).To enable the hack, the following requirements need to be fulfilled: The used JVM must be Oracle Java / OpenJDK 8 (preliminary support for Java 9 EA build 150+ was added with Lucene 6.4). In addition, the following permissions need to be granted to
lucene-core.jar
in your policy file:permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
permission java.lang.RuntimePermission "accessClassInPackage.sun.misc";
- Throws:
IllegalArgumentException
- ifUNMAP_SUPPORTED
isfalse
and the workaround cannot be enabled. The exception message also contains an explanation why the hack cannot be enabled (e.g., missing permissions).
-
getUseUnmap
public boolean getUseUnmap()
Returnstrue
, if the unmap workaround is enabled.- See Also:
setUseUnmap(boolean)
-
setPreload
public void setPreload(boolean preload)
Set totrue
to ask mapped pages to be loaded into physical memory on init. The behavior is best-effort and operating system dependent.
-
getPreload
public boolean getPreload()
Returnstrue
if mapped pages should be loaded.- See Also:
setPreload(boolean)
-
getMaxChunkSize
public final long getMaxChunkSize()
Returns the current mmap chunk size.- See Also:
MMapDirectory(Path, LockFactory, long)
-
openInput
public IndexInput openInput(String name, IOContext context) throws IOException
Creates an IndexInput for the file with the given name.- Specified by:
openInput
in classDirectory
- Parameters:
name
- the name of an existing file.- Throws:
IOException
- in case of I/O error
-
-