Package org.apache.lucene.codecs.memory
Class FSTOrdTermsWriter
- java.lang.Object
-
- org.apache.lucene.codecs.FieldsConsumer
-
- org.apache.lucene.codecs.memory.FSTOrdTermsWriter
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
public class FSTOrdTermsWriter extends FieldsConsumer
FST-based term dict, using ord as FST output. The FST holds the mapping between <term, ord>, and term's metadata is delta encoded into a single byte block. Typically the byte block consists of four parts: 1. term statistics: docFreq, totalTermFreq; 2. monotonic long[], e.g. the pointer to the postings list for that term; 3. generic byte[], e.g. other information customized by postings base. 4. single-level skip list to speed up metadata decoding by ord.Files:
- .tix: Term Index
- .tbk: Term Block
Term Index
The .tix contains a list of FSTs, one for each field. The FST maps a term to its corresponding order in current field.
- TermIndex(.tix) --> Header, TermFSTNumFields, Footer
- TermFST -->
FST<long>
- Header -->
IndexHeader
- Footer -->
CodecFooter
Notes:
- Since terms are already sorted before writing to Term Block, their ords can directly used to seek term metadata from term block.
Term Block
The .tbk contains all the statistics and metadata for terms, along with field summary (e.g. per-field data like number of documents in current field). For each field, there are four blocks:
- statistics bytes block: contains term statistics;
- metadata longs block: delta-encodes monotonic part of metadata;
- metadata bytes block: encodes other parts of metadata;
- skip block: contains skip data, to speed up metadata seeking and decoding
File Format:
- TermBlock(.tbk) --> Header, PostingsHeader, FieldSummary, DirOffset
- FieldSummary --> NumFields, <FieldNumber, NumTerms, SumTotalTermFreq?, SumDocFreq, DocCount, LongsSize, DataBlock > NumFields, Footer
- DataBlock --> StatsBlockLength, MetaLongsBlockLength, MetaBytesBlockLength, SkipBlock, StatsBlock, MetaLongsBlock, MetaBytesBlock
- SkipBlock --> < StatsFPDelta, MetaLongsSkipFPDelta, MetaBytesSkipFPDelta, MetaLongsSkipDeltaLongsSize >NumTerms
- StatsBlock --> < DocFreq[Same?], (TotalTermFreq-DocFreq) ? > NumTerms
- MetaLongsBlock --> < LongDeltaLongsSize, BytesSize > NumTerms
- MetaBytesBlock --> Byte MetaBytesBlockLength
- Header -->
IndexHeader
- DirOffset -->
Uint64
- NumFields, FieldNumber, DocCount, DocFreq, LongsSize,
FieldNumber, DocCount -->
VInt
- NumTerms, SumTotalTermFreq, SumDocFreq, StatsBlockLength, MetaLongsBlockLength, MetaBytesBlockLength,
StatsFPDelta, MetaLongsSkipFPDelta, MetaBytesSkipFPDelta, MetaLongsSkipStart, TotalTermFreq,
LongDelta,-->
VLong
- Footer -->
CodecFooter
Notes:
- The format of PostingsHeader and MetaBytes are customized by the specific postings implementation: they contain arbitrary per-file data (such as parameters or versioning information), and per-term data (non-monotonic ones like pulsed postings data).
- During initialization the reader will load all the blocks into memory. SkipBlock will be decoded, so that during seek term dict can lookup file pointers directly. StatsFPDelta, MetaLongsSkipFPDelta, etc. are file offset for every SkipInterval's term. MetaLongsSkipDelta is the difference from previous one, which indicates the value of preceding metadata longs for every SkipInterval's term.
- DocFreq is the count of documents which contain the term. TotalTermFreq is the total number of occurrences of the term. Usually these two values are the same for long tail terms, therefore one bit is stole from DocFreq to check this case, so that encoding of TotalTermFreq may be omitted.
- WARNING: This API is experimental and might change in incompatible ways in the next release.
-
-
Field Summary
Fields Modifier and Type Field Description static int
SKIP_INTERVAL
static int
VERSION_CURRENT
static int
VERSION_START
-
Constructor Summary
Constructors Constructor Description FSTOrdTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
void
write(Fields fields)
-
Methods inherited from class org.apache.lucene.codecs.FieldsConsumer
merge
-
-
-
-
Field Detail
-
VERSION_START
public static final int VERSION_START
- See Also:
- Constant Field Values
-
VERSION_CURRENT
public static final int VERSION_CURRENT
- See Also:
- Constant Field Values
-
SKIP_INTERVAL
public static final int SKIP_INTERVAL
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
FSTOrdTermsWriter
public FSTOrdTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter) throws IOException
- Throws:
IOException
-
-
Method Detail
-
write
public void write(Fields fields) throws IOException
- Specified by:
write
in classFieldsConsumer
- Throws:
IOException
-
close
public void close() throws IOException
- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Specified by:
close
in classFieldsConsumer
- Throws:
IOException
-
-