Package org.apache.lucene.search.highlight

Class TokenSources

    • Method Detail

      • getTokenStream

        public static TokenStream getTokenStream​(String field,
                                         Fields tvFields,
                                         String text,
                                         Analyzer analyzer,
                                         int maxStartOffset)
                                  throws IOException
        Get a token stream from either un-inverting a term vector if possible, or by analyzing the text.

        WARNING: Don't call this if there is more than one value for this field. If there are, and if there are term vectors, then there is a single tokenstream with offsets suggesting all the field values were concatenated.

        Parameters:
        field - The field to either get term vectors from or to analyze the text from.
        tvFields - from IndexReader.getTermVectors(int). Possibly null. For performance, this instance should be re-used for the same document (e.g. when highlighting multiple fields).
        text - the text to analyze, failing term vector un-inversion
        analyzer - the analyzer to analyze text with, failing term vector un-inversion
        maxStartOffset - Terms with a startOffset greater than this aren't returned. Use -1 for no limit. Suggest using Highlighter.getMaxDocCharsToAnalyze() - 1.
        Returns:
        a token stream from either term vectors, or from analyzing the text. Never null.
        Throws:
        IOException

      • getTermVectorTokenStreamOrNull

        public static TokenStream getTermVectorTokenStreamOrNull​(String field,
                                                         Fields tvFields,
                                                         int maxStartOffset)
                                                  throws IOException
        Get a token stream by un-inverting the term vector. This method returns null if tvFields is null or if the field has no term vector, or if the term vector doesn't have offsets. Positions are recommended on the term vector but it isn't strictly required.
        Parameters:
        field - The field to get term vectors from.
        tvFields - from IndexReader.getTermVectors(int). Possibly null. For performance, this instance should be re-used for the same document (e.g. when highlighting multiple fields).
        maxStartOffset - Terms with a startOffset greater than this aren't returned. Use -1 for no limit. Suggest using Highlighter.getMaxDocCharsToAnalyze() - 1
        Returns:
        a token stream from term vectors. Null if no term vectors with the right options.
        Throws:
        IOException

      • getAnyTokenStream

        @Deprecated
public static TokenStream getAnyTokenStream​(IndexReader reader,
                                            int docId,
                                            String field,
                                            Document document,
                                            Analyzer analyzer)
                                     throws IOException
        Deprecated.
        A convenience method that tries to first get a TokenStreamFromTermVector for the specified docId, then, falls back to using the passed in Document to retrieve the TokenStream. This is useful when you already have the document, but would prefer to use the vector first.
        Parameters:
        reader - The IndexReader to use to try and get the vector from
        docId - The docId to retrieve.
        field - The field to retrieve on the document
        document - The document to fall back on
        analyzer - The analyzer to use for creating the TokenStream if the vector doesn't exist
        Returns:
        The TokenStream for the IndexableField on the Document
        Throws:
        IOException - if there was an error loading

      • getAnyTokenStream

        @Deprecated
public static TokenStream getAnyTokenStream​(IndexReader reader,
                                            int docId,
                                            String field,
                                            Analyzer analyzer)
                                     throws IOException
        Deprecated.
        A convenience method that tries a number of approaches to getting a token stream. The cost of finding there are no termVectors in the index is minimal (1000 invocations still registers 0 ms). So this "lazy" (flexible?) approach to coding is probably acceptable
        Returns:
        null if field not stored correctly
        Throws:
        IOException - If there is a low-level I/O error

      • getTokenStream

        @Deprecated
public static TokenStream getTokenStream​(Terms tpv)
                                  throws IOException
        Deprecated.
        Returns a token stream generated from a Terms. This can be used to feed the highlighter with a pre-parsed token stream. The Terms must have offsets available. If there are no positions available, all tokens will have position increments reflecting adjacent tokens, or coincident when terms share a start offset. If there are stopwords filtered from the index, you probably want to ensure term vectors have positions so that phrase queries won't match across stopwords.
        Throws:
        IllegalArgumentException - if no offsets are available
        IOException