Class TokenSources
- java.lang.Object
-
- org.apache.lucene.search.highlight.TokenSources
-
public class TokenSources extends Object
Convenience methods for obtaining aTokenStream
for use with theHighlighter
- can obtain from term vectors with offsets and positions or from an Analyzer re-parsing the stored content.- See Also:
TokenStreamFromTermVector
-
-
Method Summary
All Methods Static Methods Concrete Methods Deprecated Methods Modifier and Type Method Description static TokenStream
getAnyTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer)
Deprecated.static TokenStream
getAnyTokenStream(IndexReader reader, int docId, String field, Document document, Analyzer analyzer)
Deprecated.static TokenStream
getTermVectorTokenStreamOrNull(String field, Fields tvFields, int maxStartOffset)
Get a token stream by un-inverting the term vector.static TokenStream
getTokenStream(String field, String contents, Analyzer analyzer)
Deprecated.static TokenStream
getTokenStream(String field, Fields tvFields, String text, Analyzer analyzer, int maxStartOffset)
Get a token stream from either un-inverting a term vector if possible, or by analyzing the text.static TokenStream
getTokenStream(Document doc, String field, Analyzer analyzer)
Deprecated.static TokenStream
getTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer)
Deprecated.static TokenStream
getTokenStream(Terms tpv)
Deprecated.static TokenStream
getTokenStream(Terms vector, boolean tokenPositionsGuaranteedContiguous)
Deprecated.static TokenStream
getTokenStreamWithOffsets(IndexReader reader, int docId, String field)
Deprecated.
-
-
-
Method Detail
-
getTokenStream
public static TokenStream getTokenStream(String field, Fields tvFields, String text, Analyzer analyzer, int maxStartOffset) throws IOException
Get a token stream from either un-inverting a term vector if possible, or by analyzing the text.WARNING: Don't call this if there is more than one value for this field. If there are, and if there are term vectors, then there is a single tokenstream with offsets suggesting all the field values were concatenated.
- Parameters:
field
- The field to either get term vectors from or to analyze the text from.tvFields
- fromTermVectors.get(int)
. Possibly null. For performance, this instance should be re-used for the same document (e.g. when highlighting multiple fields).text
- the text to analyze, failing term vector un-inversionanalyzer
- the analyzer to analyzetext
with, failing term vector un-inversionmaxStartOffset
- Terms with a startOffset greater than this aren't returned. Use -1 for no limit. Suggest usingHighlighter.getMaxDocCharsToAnalyze()
- 1.- Returns:
- a token stream from either term vectors, or from analyzing the text. Never null.
- Throws:
IOException
-
getTermVectorTokenStreamOrNull
public static TokenStream getTermVectorTokenStreamOrNull(String field, Fields tvFields, int maxStartOffset) throws IOException
Get a token stream by un-inverting the term vector. This method returns null iftvFields
is null or if the field has no term vector, or if the term vector doesn't have offsets. Positions are recommended on the term vector but it isn't strictly required.- Parameters:
field
- The field to get term vectors from.tvFields
- fromTermVectors.get(int)
. Possibly null. For performance, this instance should be re-used for the same document (e.g. when highlighting multiple fields).maxStartOffset
- Terms with a startOffset greater than this aren't returned. Use -1 for no limit. Suggest usingHighlighter.getMaxDocCharsToAnalyze()
- 1- Returns:
- a token stream from term vectors. Null if no term vectors with the right options.
- Throws:
IOException
-
getAnyTokenStream
@Deprecated public static TokenStream getAnyTokenStream(IndexReader reader, int docId, String field, Document document, Analyzer analyzer) throws IOException
Deprecated.A convenience method that tries to first get aTokenStreamFromTermVector
for the specified docId, then, falls back to using the passed inDocument
to retrieve the TokenStream. This is useful when you already have the document, but would prefer to use the vector first.- Parameters:
reader
- TheIndexReader
to use to try and get the vector fromdocId
- The docId to retrieve.field
- The field to retrieve on the documentdocument
- The document to fall back onanalyzer
- The analyzer to use for creating the TokenStream if the vector doesn't exist- Returns:
- The
TokenStream
for theIndexableField
on theDocument
- Throws:
IOException
- if there was an error loading
-
getAnyTokenStream
@Deprecated public static TokenStream getAnyTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer) throws IOException
Deprecated.A convenience method that tries a number of approaches to getting a token stream. The cost of finding there are no termVectors in the index is minimal (1000 invocations still registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable- Returns:
- null if field not stored correctly
- Throws:
IOException
- If there is a low-level I/O error
-
getTokenStream
@Deprecated public static TokenStream getTokenStream(Terms vector, boolean tokenPositionsGuaranteedContiguous) throws IOException
Deprecated.Simply callsgetTokenStream(org.apache.lucene.index.Terms)
now.- Throws:
IOException
-
getTokenStream
@Deprecated public static TokenStream getTokenStream(Terms tpv) throws IOException
Deprecated.Returns a token stream generated from aTerms
. This can be used to feed the highlighter with a pre-parsed token stream. TheTerms
must have offsets available. If there are no positions available, all tokens will have position increments reflecting adjacent tokens, or coincident when terms share a start offset. If there are stopwords filtered from the index, you probably want to ensure term vectors have positions so that phrase queries won't match across stopwords.- Throws:
IllegalArgumentException
- if no offsets are availableIOException
-
getTokenStreamWithOffsets
@Deprecated public static TokenStream getTokenStreamWithOffsets(IndexReader reader, int docId, String field) throws IOException
Deprecated.Returns aTokenStream
with positions and offsets constructed from field termvectors. If the field has no termvectors or offsets are not included in the termvector, return null. SeegetTokenStream(org.apache.lucene.index.Terms)
for an explanation of what happens when positions aren't present.- Parameters:
reader
- theIndexReader
to retrieve term vectors fromdocId
- the document to retrieve termvectors forfield
- the field to retrieve termvectors for- Returns:
- a
TokenStream
, or null if offsets are not available - Throws:
IOException
- If there is a low-level I/O error- See Also:
getTokenStream(org.apache.lucene.index.Terms)
-
getTokenStream
@Deprecated public static TokenStream getTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer) throws IOException
Deprecated.- Throws:
IOException
-
getTokenStream
@Deprecated public static TokenStream getTokenStream(Document doc, String field, Analyzer analyzer)
Deprecated.
-
getTokenStream
@Deprecated public static TokenStream getTokenStream(String field, String contents, Analyzer analyzer)
Deprecated.
-
-