public class DocTermOrds extends Object implements Accountable
getOrdTermsEnum(org.apache.lucene.index.AtomicReader)
method, and then seek-by-ord to get the term's bytes.
While normally term ords are type long, in this API they are
int as the internal representation here cannot address
more than MAX_INT unique terms. Also, typically this
class is used on fields with relatively few unique terms
vs the number of documents. In addition, there is an
internal limit (16 MB) on how many bytes each chunk of
documents may consume. If you trip this limit you'll hit
an IllegalStateException.
Deleted documents are skipped during uninversion, and if
you look them up you'll get 0 ords.
The returned per-document ords do not retain their
original order in the document. Instead they are returned
in sorted (by ord, ie term's BytesRef comparator) order. They
are also de-dup'd (ie if doc has same term more than once
in this field, you'll only get that ord back once).
This class tests whether the provided reader is able to
retrieve terms by ord (ie, it's single segment, and it
uses an ord-capable terms index). If not, this class
will create its own term index internally, allowing to
create a wrapped TermsEnum that can handle ord. The
getOrdTermsEnum(org.apache.lucene.index.AtomicReader)
method then provides this
wrapped enum, if necessary.
The RAM consumption of this class can be high!Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_INDEX_INTERVAL_BITS
Every 128th term is indexed, by default.
|
protected DocsEnum |
docsEnum
Used while uninverting.
|
protected String |
field
Field we are uninverting.
|
protected int[] |
index
Holds the per-document ords or a pointer to the ords.
|
protected BytesRef[] |
indexedTermsArray
Holds the indexed (by default every 128th) terms.
|
protected int |
maxTermDocFreq
Don't uninvert terms that exceed this count.
|
protected int |
numTermsInField
Number of terms in the field.
|
protected int |
ordBase
Ordinal of the first term in the field, or 0 if the
PostingsFormat does not implement TermsEnum.ord() . |
protected int |
phase1_time
Time for phase1 of the uninvert process.
|
protected BytesRef |
prefix
If non-null, only terms matching this prefix were
indexed.
|
protected long |
sizeOfIndexedStrings
Total bytes (sum of term lengths) for all indexed terms.
|
protected long |
termInstances
Total number of references to term numbers.
|
protected byte[][] |
tnums
Holds term ords for documents.
|
protected int |
total_time
Total time to uninvert the field.
|
Modifier | Constructor and Description |
---|---|
|
DocTermOrds(AtomicReader reader,
Bits liveDocs,
String field)
Inverts all terms
|
|
DocTermOrds(AtomicReader reader,
Bits liveDocs,
String field,
BytesRef termPrefix)
Inverts only terms starting w/ prefix
|
|
DocTermOrds(AtomicReader reader,
Bits liveDocs,
String field,
BytesRef termPrefix,
int maxTermDocFreq)
Inverts only terms starting w/ prefix, and only terms
whose docFreq (not taking deletions into account) is
<= maxTermDocFreq
|
|
DocTermOrds(AtomicReader reader,
Bits liveDocs,
String field,
BytesRef termPrefix,
int maxTermDocFreq,
int indexIntervalBits)
Inverts only terms starting w/ prefix, and only terms
whose docFreq (not taking deletions into account) is
<= maxTermDocFreq, with a custom indexing interval
(default is every 128nd term).
|
protected |
DocTermOrds(String field,
int maxTermDocFreq,
int indexIntervalBits)
Subclass inits w/ this, but be sure you then call
uninvert, only once
|
Modifier and Type | Method and Description |
---|---|
TermsEnum |
getOrdTermsEnum(AtomicReader reader)
Returns a TermsEnum that implements ord.
|
boolean |
isEmpty()
Returns
true if no terms were indexed. |
SortedSetDocValues |
iterator(AtomicReader reader)
Returns a SortedSetDocValues view of this instance
|
BytesRef |
lookupTerm(TermsEnum termsEnum,
int ord)
Returns the term (
BytesRef ) corresponding to
the provided ordinal. |
int |
numTerms()
Returns the number of terms in this field
|
long |
ramBytesUsed()
Returns total bytes used.
|
protected void |
setActualDocFreq(int termNum,
int df)
Invoked during
uninvert(AtomicReader,Bits,BytesRef)
to record the document frequency for each uninverted
term. |
protected void |
uninvert(AtomicReader reader,
Bits liveDocs,
BytesRef termPrefix)
Call this only once (if you subclass!)
|
protected void |
visitTerm(TermsEnum te,
int termNum)
Subclass can override this
|
public static final int DEFAULT_INDEX_INTERVAL_BITS
protected final int maxTermDocFreq
protected final String field
protected int numTermsInField
protected long termInstances
protected int total_time
protected int phase1_time
protected int[] index
protected byte[][] tnums
protected long sizeOfIndexedStrings
protected BytesRef[] indexedTermsArray
protected BytesRef prefix
protected int ordBase
PostingsFormat
does not implement TermsEnum.ord()
.protected DocsEnum docsEnum
public DocTermOrds(AtomicReader reader, Bits liveDocs, String field) throws IOException
IOException
public DocTermOrds(AtomicReader reader, Bits liveDocs, String field, BytesRef termPrefix) throws IOException
IOException
public DocTermOrds(AtomicReader reader, Bits liveDocs, String field, BytesRef termPrefix, int maxTermDocFreq) throws IOException
IOException
public DocTermOrds(AtomicReader reader, Bits liveDocs, String field, BytesRef termPrefix, int maxTermDocFreq, int indexIntervalBits) throws IOException
IOException
protected DocTermOrds(String field, int maxTermDocFreq, int indexIntervalBits)
public long ramBytesUsed()
ramBytesUsed
in interface Accountable
public TermsEnum getOrdTermsEnum(AtomicReader reader) throws IOException
NOTE: you must pass the same reader that was used when creating this class
IOException
public int numTerms()
public boolean isEmpty()
true
if no terms were indexed.protected void visitTerm(TermsEnum te, int termNum) throws IOException
IOException
protected void setActualDocFreq(int termNum, int df) throws IOException
uninvert(AtomicReader,Bits,BytesRef)
to record the document frequency for each uninverted
term.IOException
protected void uninvert(AtomicReader reader, Bits liveDocs, BytesRef termPrefix) throws IOException
IOException
public BytesRef lookupTerm(TermsEnum termsEnum, int ord) throws IOException
BytesRef
) corresponding to
the provided ordinal.IOException
public SortedSetDocValues iterator(AtomicReader reader) throws IOException
IOException
Copyright © 2000-2014 Apache Software Foundation. All Rights Reserved.