Package org.apache.lucene.store

Class MMapDirectory

java.lang.Object

org.apache.lucene.store.Directory

org.apache.lucene.store.BaseDirectory

org.apache.lucene.store.FSDirectory

org.apache.lucene.store.MMapDirectory

All Implemented Interfaces:

Closeable, AutoCloseable

public class MMapDirectory
extends FSDirectory

File-based Directory implementation that uses mmap for reading, and FSDirectory.FSIndexOutput for writing.

NOTE: memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure your have plenty of virtual address space, e.g. by using a 64 bit JRE, or a 32 bit JRE with indexes that are guaranteed to fit within the address space. On 32 bit platforms also consult MMapDirectory(Path, LockFactory, long) if you have problems with mmap failing because of fragmented address space. If you get an IOException about mapping failed, it is recommended to reduce the chunk size, until it works.

This class supports preloading files into physical memory upon opening. This can help improve performance of searches on a cold page cache at the expense of slowing down opening an index. See setPreload(BiPredicate) for more details.

This class supports grouping of files that are part of the same logical group. This is a hint that allows for better handling of resources. For example, individual files that are part of the same segment can be considered part of the same logical group. See setGroupingFunction(Function) for more details.

This class will use the modern MemorySegment^PREVIEW API available since Java 21 which allows to safely unmap previously mmapped files after closing the IndexInputs. There is no need to enable the "preview feature" of your Java version; it works out of box with some compilation tricks. For more information about the foreign memory API read documentation of the java.lang.foreign package.

On some platforms like Linux and MacOS X, this class will invoke the syscall madvise() to advise how OS kernel should handle paging after opening a file. For this to work, Java code must be able to call native code. If this is not allowed, a warning is logged. To enable native access for Lucene in a modularized application, pass --enable-native-access=org.apache.lucene.core to the Java command line. If Lucene is running in a classpath-based application, use --enable-native-access=ALL-UNNAMED.

NOTE: Accessing this class either directly or indirectly from a thread while it's interrupted can close the underlying channel immediately if at the same time the thread is blocked on IO. The channel will remain closed and subsequent access to MMapDirectory will throw a ClosedChannelException. If your application uses either Thread.interrupt() or Future.cancel(boolean) you should use the legacy RAFDirectory from the Lucene misc module in favor of MMapDirectory.

NOTE: If your application requires external synchronization, you should not synchronize on the MMapDirectory instance as this may cause deadlock; use your own (non-Lucene) objects instead.

See Also:

• Blog post about MMapDirectory

Field Summary

Fields

Modifier and Type	Field	Description
static final BiPredicate<String,IOContext>	ALL_FILES	Argument for setPreload(BiPredicate) that configures all files to be preloaded upon opening them.
static final BiPredicate<String,IOContext>	BASED_ON_LOAD_IO_CONTEXT	Argument for setPreload(BiPredicate) that configures files to be preloaded upon opening them if they use the ReadAdvice.RANDOM_PRELOAD advice.
static final long	DEFAULT_MAX_CHUNK_SIZE	Default max chunk size: 16 GiBytes for 64 bit JVMs 256 MiBytes for 32 bit JVMs
static final Function<String,Optional<String>>	GROUP_BY_SEGMENT	Argument for setGroupingFunction(Function) that configures grouping by segment.
static final BiPredicate<String,IOContext>	NO_FILES	Argument for setPreload(BiPredicate) that configures no files to be preloaded upon opening them.
static final Function<String,Optional<String>>	NO_GROUPING	Argument for setGroupingFunction(Function) that configures no grouping.
static final String	SHARED_ARENA_MAX_PERMITS_SYSPROP	This sysprop allows to control the total maximum number of mmapped files that can be associated with a single shared foreign ArenaPREVIEW.

Fields inherited from class org.apache.lucene.store.FSDirectory

directory

Fields inherited from class org.apache.lucene.store.BaseDirectory

isOpen, lockFactory

Constructor Summary

Constructors

Constructor	Description
MMapDirectory(Path path)	Create a new MMapDirectory for the named location and FSLockFactory.getDefault().
MMapDirectory(Path path, long maxChunkSize)	Create a new MMapDirectory for the named location and FSLockFactory.getDefault().
MMapDirectory(Path path, LockFactory lockFactory)	Create a new MMapDirectory for the named location.
MMapDirectory(Path path, LockFactory lockFactory, long maxChunkSize)	Create a new MMapDirectory for the named location, specifying the maximum chunk size used for memory mapping.

Method Summary

Modifier and Type	Method	Description
final long	getMaxChunkSize()	Returns the current mmap chunk size.
IndexInput	openInput(String name, IOContext context)	Creates an IndexInput for the file with the given name.
void	setGroupingFunction(Function<String,Optional<String>> groupingFunction)	Configures a grouping function for files that are part of the same logical group.
void	setPreload(BiPredicate<String,IOContext> preload)	Configure which files to preload in physical memory upon opening.
static boolean	supportsMadvise()	Returns true, if MMapDirectory uses the platform's madvise() syscall to advise how OS kernel should handle paging after opening a file.

Methods inherited from class org.apache.lucene.store.FSDirectory

close, createOutput, createTempOutput, deleteFile, deletePendingFiles, ensureCanRead, fileLength, fsync, getDirectory, getPendingDeletions, listAll, listAll, open, open, rename, sync, syncMetaData, toString

Methods inherited from class org.apache.lucene.store.BaseDirectory

ensureOpen, obtainLock

Methods inherited from class org.apache.lucene.store.Directory

copyFrom, getTempFileName, openChecksumInput

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

ALL_FILES

public static final BiPredicate<String,IOContext> ALL_FILES

Argument for setPreload(BiPredicate) that configures all files to be preloaded upon opening them.

NO_FILES

public static final BiPredicate<String,IOContext> NO_FILES

Argument for setPreload(BiPredicate) that configures no files to be preloaded upon opening them.

SHARED_ARENA_MAX_PERMITS_SYSPROP

public static final String SHARED_ARENA_MAX_PERMITS_SYSPROP

This sysprop allows to control the total maximum number of mmapped files that can be associated with a single shared foreign Arena^PREVIEW. For example, to set the max number of permits to 256, pass the following on the command line pass -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=256. Setting a value of 1 associates one file to one shared arena.

See Also:

• Constant Field Values

NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

NO_GROUPING

public static final Function<String,Optional<String>> NO_GROUPING

Argument for setGroupingFunction(Function) that configures no grouping.

GROUP_BY_SEGMENT

public static final Function<String,Optional<String>> GROUP_BY_SEGMENT

Argument for setGroupingFunction(Function) that configures grouping by segment.

BASED_ON_LOAD_IO_CONTEXT

public static final BiPredicate<String,IOContext> BASED_ON_LOAD_IO_CONTEXT

Argument for setPreload(BiPredicate) that configures files to be preloaded upon opening them if they use the ReadAdvice.RANDOM_PRELOAD advice.

DEFAULT_MAX_CHUNK_SIZE

public static final long DEFAULT_MAX_CHUNK_SIZE

Default max chunk size:

• 16 GiBytes for 64 bit JVMs
• 256 MiBytes for 32 bit JVMs

Constructor Details

MMapDirectory

public MMapDirectory(Path path,
 LockFactory lockFactory)
              throws IOException

Create a new MMapDirectory for the named location. The directory is created at the named location if it does not yet exist.

Parameters:

path - the path of the directory

lockFactory - the lock factory to use

Throws:

IOException - if there is a low-level I/O error

MMapDirectory

public MMapDirectory(Path path)
              throws IOException

Create a new MMapDirectory for the named location and FSLockFactory.getDefault(). The directory is created at the named location if it does not yet exist.

Parameters:

path - the path of the directory

Throws:

IOException - if there is a low-level I/O error

MMapDirectory

public MMapDirectory(Path path,
 long maxChunkSize)
              throws IOException

Create a new MMapDirectory for the named location and FSLockFactory.getDefault(). The directory is created at the named location if it does not yet exist.

Parameters:

path - the path of the directory

maxChunkSize - maximum chunk size (for default see DEFAULT_MAX_CHUNK_SIZE) used for memory mapping.

Throws:

IOException - if there is a low-level I/O error

MMapDirectory

public MMapDirectory(Path path,
 LockFactory lockFactory,
 long maxChunkSize)
              throws IOException

Create a new MMapDirectory for the named location, specifying the maximum chunk size used for memory mapping. The directory is created at the named location if it does not yet exist.

Especially on 32 bit platform, the address space can be very fragmented, so large index files cannot be mapped. Using a lower chunk size makes the directory implementation a little bit slower (as the correct chunk may be resolved on lots of seeks) but the chance is higher that mmap does not fail. On 64 bit Java platforms, this parameter should always be large (like 1 GiBytes, or even larger with recent Java versions), as the address space is big enough. If it is larger, fragmentation of address space increases, but number of file handles and mappings is lower for huge installations with many open indexes.

Please note: The chunk size is always rounded down to a power of 2.

Parameters:

path - the path of the directory

lockFactory - the lock factory to use, or null for the default (NativeFSLockFactory);

maxChunkSize - maximum chunk size (for default see DEFAULT_MAX_CHUNK_SIZE) used for memory mapping.

Throws:

IOException - if there is a low-level I/O error

Method Details

setPreload

public void setPreload(BiPredicate<String,IOContext> preload)

Configure which files to preload in physical memory upon opening. The default implementation does not preload anything. The behavior is best effort and operating system-dependent.

Parameters:

preload - a BiPredicate whose first argument is the file name, and second argument is the IOContext used to open the file

See Also:

• ALL_FILES
• NO_FILES

setGroupingFunction

public void setGroupingFunction(Function<String,Optional<String>> groupingFunction)

Configures a grouping function for files that are part of the same logical group. The gathering of files into a logical group is a hint that allows for better handling of resources.

By default, grouping is GROUP_BY_SEGMENT. To disable, invoke this method with NO_GROUPING.

Parameters:

groupingFunction - a function that accepts a file name and returns an optional group key. If the optional is present, then its value is the logical group to which the file belongs. Otherwise, the file name if not associated with any logical group.

getMaxChunkSize

public final long getMaxChunkSize()

Returns the current mmap chunk size.

See Also:

• MMapDirectory(Path, LockFactory, long)

openInput

public IndexInput openInput(String name,
 IOContext context)
                     throws IOException

Creates an IndexInput for the file with the given name.

Specified by:

openInput in class Directory

Parameters:

name - the name of an existing file.

Throws:

IOException - in case of I/O error

supportsMadvise

public static boolean supportsMadvise()

Returns true, if MMapDirectory uses the platform's madvise() syscall to advise how OS kernel should handle paging after opening a file.