public class Builder<T> extends Object
NOTE: The algorithm is described at http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.24.3698
The parameterized type T is the output type. See the
subclasses of Outputs
.
FSTs larger than 2.1GB are now possible (as of Lucene 4.2). FSTs containing more than 2.1B nodes are also now possible, however they cannot be packed.
Modifier and Type | Class and Description |
---|---|
static class |
Builder.Arc<T>
Expert: holds a pending (seen but not yet serialized) arc.
|
static class |
Builder.UnCompiledNode<T>
Expert: holds a pending (seen but not yet serialized) Node.
|
Constructor and Description |
---|
Builder(FST.INPUT_TYPE inputType,
int minSuffixCount1,
int minSuffixCount2,
boolean doShareSuffix,
boolean doShareNonSingletonNodes,
int shareMaxTailLength,
Outputs<T> outputs,
boolean allowArrayArcs,
int bytesPageBits)
Instantiates an FST/FSA builder with all the possible tuning and construction
tweaks.
|
Builder(FST.INPUT_TYPE inputType,
Outputs<T> outputs)
Instantiates an FST/FSA builder without any pruning.
|
Modifier and Type | Method and Description |
---|---|
void |
add(IntsRef input,
T output)
Add the next input/output pair.
|
FST<T> |
finish()
Returns final FST.
|
long |
fstRamBytesUsed() |
long |
getArcCount() |
long |
getMappedStateCount() |
long |
getNodeCount() |
long |
getTermCount() |
public Builder(FST.INPUT_TYPE inputType, Outputs<T> outputs)
Builder(FST.INPUT_TYPE, int, int, boolean,
boolean, int, Outputs, boolean, int)
with pruning options turned off.public Builder(FST.INPUT_TYPE inputType, int minSuffixCount1, int minSuffixCount2, boolean doShareSuffix, boolean doShareNonSingletonNodes, int shareMaxTailLength, Outputs<T> outputs, boolean allowArrayArcs, int bytesPageBits)
inputType
- The input type (transition labels). Can be anything from FST.INPUT_TYPE
enumeration. Shorter types will consume less memory. Strings (character sequences) are
represented as FST.INPUT_TYPE.BYTE4
(full unicode codepoints).minSuffixCount1
- If pruning the input graph during construction, this threshold is used for telling
if a node is kept or pruned. If transition_count(node) >= minSuffixCount1, the node
is kept.minSuffixCount2
- (Note: only Mike McCandless knows what this one is really doing...)doShareSuffix
- If true
, the shared suffixes will be compacted into unique paths.
This requires an additional RAM-intensive hash map for lookups in memory. Setting this parameter to
false
creates a single suffix path for all input sequences. This will result in a larger
FST, but requires substantially less memory and CPU during building.doShareNonSingletonNodes
- Only used if doShareSuffix is true. Set this to
true to ensure FST is fully minimal, at cost of more
CPU and more RAM during building.shareMaxTailLength
- Only used if doShareSuffix is true. Set this to
Integer.MAX_VALUE to ensure FST is fully minimal, at cost of more
CPU and more RAM during building.outputs
- The output type for each input sequence. Applies only if building an FST. For
FSA, use NoOutputs.getSingleton()
and NoOutputs.getNoOutput()
as the
singleton output object.allowArrayArcs
- Pass false to disable the array arc optimization
while building the FST; this will make the resulting
FST smaller but slower to traverse.bytesPageBits
- How many bits wide to make each
byte[] block in the BytesStore; if you know the FST
will be large then make this larger. For example 15
bits = 32768 byte pages.public long getTermCount()
public long getNodeCount()
public long getArcCount()
public long getMappedStateCount()
public void add(IntsRef input, T output) throws IOException
IntsRef.compareTo(org.apache.lucene.util.IntsRef)
. It's also OK to add the same
input twice in a row with different outputs, as long
as Outputs
implements the Outputs.merge(T, T)
method. Note that input is fully consumed after this
method is returned (so caller is free to reuse), but
output is not. So if your outputs are changeable (eg
ByteSequenceOutputs
or IntSequenceOutputs
) then you cannot reuse across
calls.IOException
public FST<T> finish() throws IOException
IOException
public long fstRamBytesUsed()
Copyright © 2000-2018 Apache Software Foundation. All Rights Reserved.