public final class Lucene41StoredFieldsFormat extends CompressingStoredFieldsFormat
StoredFieldsFormat compresses blocks of 16KB of documents in
order to improve the compression ratio compared to document-level
compression. It uses the LZ4
compression algorithm, which is fast to compress and very fast to decompress
data. Although the compression method that is used focuses more on speed
than on compression ratio, it should provide interesting compression ratios
for redundant inputs (such as log files, HTML or plain text).
Stored fields are represented by two files:
A fields data file (extension .fdt). This file stores a compact representation of documents in compressed blocks of 16KB or more. When writing a segment, documents are appended to an in-memory byte buffer. When its size reaches 16KB or more, some metadata about the documents is flushed to disk, immediately followed by a compressed representation of the buffer using the LZ4 compression format.
Here is a more detailed description of the field data file format:
VInt(let's call it bitsRequired)
VLong, whose 3 last bits are Type and other bits are FieldNum
String| BinaryValue | Int | Float | Long | Double depending on Type
A fields index file (extension .fdx). The data stored in this file is read to load an in-memory data-structure that can be used to locate the start offset of a block containing any document in the fields data file.
In order to have a compact in-memory representation, for every block of 1024 chunks, this stored fields index computes the average number of bytes per chunk and for every chunk, only stores the difference between
Data is written as follows:
VInt, this marks the end of blocks since blocks are not allowed to start with 0
VIntwhich is the number of chunks encoded in the block
packedarray of BlockChunks elements of BitsPerDocBaseDelta bits each, representing the deltas from the average doc base using ZigZag encoding.
packedarray of BlockChunks elements of BitsPerStartPointerDelta bits each, representing the deltas from the average start pointer using ZigZag encoding
DocBase + AvgChunkDocs * n + DocBaseDeltas[n].
StartPointerBase + AvgChunkSize * n + StartPointerDeltas[n].
|Constructor and Description|
Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.