org.apache.lucene.analysis

Class BaseTokenStreamTestCase

java.lang.Object

org.junit.Assert

org.apache.lucene.util.LuceneTestCase

org.apache.lucene.analysis.BaseTokenStreamTestCase

Direct Known Subclasses:

WordBreakTestUnicode_6_3_0

public abstract class BaseTokenStreamTestCase
extends LuceneTestCase

Base class for all Lucene unit tests that use TokenStreams.

When writing unit tests for analysis components, it's highly recommended to use the helper methods here (especially in conjunction with MockAnalyzer or MockTokenizer), as they contain many assertions and checks to catch bugs.

See Also:

MockAnalyzer, MockTokenizer

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	BaseTokenStreamTestCase.CheckClearAttributesAttribute Attribute that records if it was cleared or not.
static class	BaseTokenStreamTestCase.CheckClearAttributesAttributeImpl Attribute that records if it was cleared or not.

Nested classes/interfaces inherited from class org.apache.lucene.util.LuceneTestCase

LuceneTestCase.AwaitsFix, LuceneTestCase.BadApple, LuceneTestCase.Monster, LuceneTestCase.Nightly, LuceneTestCase.Slow, LuceneTestCase.SuppressCodecs, LuceneTestCase.SuppressFileSystems, LuceneTestCase.SuppressFsync, LuceneTestCase.SuppressReproduceLine, LuceneTestCase.SuppressSysoutChecks, LuceneTestCase.SuppressTempFileChecks, LuceneTestCase.ThrowingRunnable, LuceneTestCase.Weekly

Field Summary

Fields inherited from class org.apache.lucene.util.LuceneTestCase

assertsAreEnabled, classRules, DEFAULT_LINE_DOCS_FILE, INFOSTREAM, JENKINS_LARGE_LINE_DOCS_FILE, LEAVE_TEMPORARY, MAYBE_CACHE_POLICY, RANDOM_MULTIPLIER, ruleChain, suiteFailureMarker, SYSPROP_AWAITSFIX, SYSPROP_BADAPPLES, SYSPROP_FAILFAST, SYSPROP_MAXFAILURES, SYSPROP_MONSTER, SYSPROP_NIGHTLY, SYSPROP_SLOW, SYSPROP_WEEKLY, TEST_ASSERTS_ENABLED, TEST_AWAITSFIX, TEST_BADAPPLES, TEST_CODEC, TEST_DIRECTORY, TEST_DOCVALUESFORMAT, TEST_LINE_DOCS_FILE, TEST_MONSTER, TEST_NIGHTLY, TEST_POSTINGSFORMAT, TEST_SLOW, TEST_THROTTLING, TEST_WEEKLY, VERBOSE

Constructor Summary

Constructors

Constructor and Description
BaseTokenStreamTestCase()

Method Summary

Modifier and Type	Method and Description
static void	assertAnalyzesTo(Analyzer a, String input, String[] output)
static void	assertAnalyzesTo(Analyzer a, String input, String[] output, int[] posIncrements)
static void	assertAnalyzesTo(Analyzer a, String input, String[] output, int[] startOffsets, int[] endOffsets)
static void	assertAnalyzesTo(Analyzer a, String input, String[] output, int[] startOffsets, int[] endOffsets, int[] posIncrements)
static void	assertAnalyzesTo(Analyzer a, String input, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements)
static void	assertAnalyzesTo(Analyzer a, String input, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements, int[] posLengths)
static void	assertAnalyzesTo(Analyzer a, String input, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements, int[] posLengths, boolean graphOffsetsAreCorrect)
static void	assertAnalyzesTo(Analyzer a, String input, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements, int[] posLengths, boolean graphOffsetsAreCorrect, byte[][] payloads)
static void	assertAnalyzesTo(Analyzer a, String input, String[] output, String[] types)
static void	assertAnalyzesToPositions(Analyzer a, String input, String[] output, int[] posIncrements, int[] posLengths)
static void	assertAnalyzesToPositions(Analyzer a, String input, String[] output, String[] types, int[] posIncrements, int[] posLengths)
static void	assertGraphStrings(Analyzer analyzer, String text, String... expectedStrings) Enumerates all accepted strings in the token graph created by the analyzer on the provided text, and then asserts that it's equal to the expected strings.
static void	assertGraphStrings(TokenStream tokenStream, String... expectedStrings) Enumerates all accepted strings in the token graph created by the already initialized TokenStream.
static void	assertStreamHasNumberOfTokens(TokenStream ts, int expectedCount) Asserts that the given stream has expected number of tokens.
static void	assertTokenStreamContents(TokenStream ts, String[] output)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] posIncrements)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, int[] posIncrements)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, int[] posIncrements, int[] posLengths, Integer finalOffset)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, int[] posIncrements, Integer finalOffset)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, Integer finalOffset)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements, int[] posLengths)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements, int[] posLengths, Integer finalOffset)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements, int[] posLengths, Integer finalOffset, boolean graphOffsetsAreCorrect)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements, int[] posLengths, Integer finalOffset, boolean[] keywordAtts, boolean graphOffsetsAreCorrect)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements, int[] posLengths, Integer finalOffset, Integer finalPosInc, boolean[] keywordAtts, boolean graphOffsetsAreCorrect, byte[][] payloads)
static void	assertTokenStreamContents(TokenStream ts, String[] output, int[] startOffsets, int[] endOffsets, String[] types, int[] posIncrements, Integer finalOffset)
static void	assertTokenStreamContents(TokenStream ts, String[] output, String[] types)
static void	checkAnalysisConsistency(Random random, Analyzer a, boolean useCharFilter, String text)
static void	checkAnalysisConsistency(Random random, Analyzer a, boolean useCharFilter, String text, boolean graphOffsetsAreCorrect)
static void	checkOneTerm(Analyzer a, String input, String expected)
static void	checkRandomData(Random random, Analyzer a, int iterations) utility method for blasting tokenstreams with data to make sure they don't do anything crazy
static void	checkRandomData(Random random, Analyzer a, int iterations, boolean simple) utility method for blasting tokenstreams with data to make sure they don't do anything crazy
static void	checkRandomData(Random random, Analyzer a, int iterations, int maxWordLength) utility method for blasting tokenstreams with data to make sure they don't do anything crazy
static void	checkRandomData(Random random, Analyzer a, int iterations, int maxWordLength, boolean simple)
static void	checkRandomData(Random random, Analyzer a, int iterations, int maxWordLength, boolean simple, boolean graphOffsetsAreCorrect)
static String	escape(String s)
static Set<String>	getGraphStrings(Analyzer analyzer, String text) Returns all paths accepted by the token stream graph produced by analyzing text with the provided analyzer.
static Set<String>	getGraphStrings(TokenStream tokenStream) Returns all paths accepted by the token stream graph produced by the already initialized TokenStream.
protected static MockTokenizer	keywordMockTokenizer(Reader input)
protected static MockTokenizer	keywordMockTokenizer(String input)
static AttributeFactory	newAttributeFactory() Returns a random AttributeFactory impl
static AttributeFactory	newAttributeFactory(Random random) Returns a random AttributeFactory impl
protected String	toDot(Analyzer a, String inputText)
protected void	toDotFile(Analyzer a, String inputText, String localFileName)
static String	toString(Analyzer analyzer, String text) Returns a String summary of the tokens this analyzer produces on this text
protected static MockTokenizer	whitespaceMockTokenizer(Reader input)
protected static MockTokenizer	whitespaceMockTokenizer(String input)

Methods inherited from class org.apache.lucene.util.LuceneTestCase

addVirusChecker, assertDeletedDocsEquals, assertDocsAndPositionsEnumEquals, assertDocsEnumEquals, assertDocsSkippingEquals, assertDocValuesEquals, assertDocValuesEquals, assertFieldInfosEquals, assertFieldsEquals, assertFieldStatisticsEquals, assertNormsEquals, assertPointsEquals, assertPositionsSkippingEquals, assertReaderEquals, assertReaderStatisticsEquals, assertStoredFieldEquals, assertStoredFieldsEquals, assertTermsEnumEquals, assertTermsEquals, assertTermsStatisticsEquals, assertTermStatsEquals, assertTermVectorsEquals, asSet, assumeFalse, assumeNoException, assumeTrue, assumeWorkingMMapOnWindows, atLeast, atLeast, closeAfterSuite, closeAfterTest, collate, createTempDir, createTempDir, createTempFile, createTempFile, dumpArray, dumpIterator, expectThrows, expectThrows, expectThrows, expectThrowsAnyOf, expectThrowsAnyOf, getBaseTempDirForTestClass, getDataInputStream, getDataPath, getOnlyLeafReader, getTestClass, getTestName, hasWorkingMMapOnWindows, isTestThread, localeForLanguageTag, maybeChangeLiveIndexWriterConfig, maybeWrapReader, newAlcoholicMergePolicy, newAlcoholicMergePolicy, newDirectory, newDirectory, newDirectory, newDirectory, newDirectory, newField, newField, newFSDirectory, newFSDirectory, newIndexWriterConfig, newIndexWriterConfig, newIndexWriterConfig, newIOContext, newIOContext, newLogMergePolicy, newLogMergePolicy, newLogMergePolicy, newLogMergePolicy, newLogMergePolicy, newMaybeVirusCheckingDirectory, newMaybeVirusCheckingFSDirectory, newMergePolicy, newMergePolicy, newMergePolicy, newMockDirectory, newMockDirectory, newMockDirectory, newMockFSDirectory, newMockFSDirectory, newSearcher, newSearcher, newSearcher, newStringField, newStringField, newStringField, newStringField, newTextField, newTextField, newTieredMergePolicy, newTieredMergePolicy, overrideDefaultQueryCache, overrideTestDefaultQueryCache, random, randomLocale, randomTimeZone, rarely, rarely, replaceMaxFailureRule, resetDefaultQueryCache, restoreCPUCoreCount, restoreIndexWriterMaxDocs, restoreSpins, runWithRestrictedPermissions, setIndexWriterMaxDocs, setUp, setupCPUCoreCount, setupSpins, slowFileExists, tearDown, usually, usually, wrapReader

Methods inherited from class org.junit.Assert

assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertFalse, assertFalse, assertNotNull, assertNotNull, assertNotSame, assertNotSame, assertNull, assertNull, assertSame, assertSame, assertThat, assertThat, assertTrue, assertTrue, fail, fail

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BaseTokenStreamTestCase

public BaseTokenStreamTestCase()

Method Detail

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             String[] types,
                                             int[] posIncrements,
                                             int[] posLengths,
                                             Integer finalOffset,
                                             Integer finalPosInc,
                                             boolean[] keywordAtts,
                                             boolean graphOffsetsAreCorrect,
                                             byte[][] payloads)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             String[] types,
                                             int[] posIncrements,
                                             int[] posLengths,
                                             Integer finalOffset,
                                             boolean[] keywordAtts,
                                             boolean graphOffsetsAreCorrect)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             String[] types,
                                             int[] posIncrements,
                                             int[] posLengths,
                                             Integer finalOffset,
                                             boolean graphOffsetsAreCorrect)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             String[] types,
                                             int[] posIncrements,
                                             int[] posLengths,
                                             Integer finalOffset)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             String[] types,
                                             int[] posIncrements,
                                             Integer finalOffset)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             String[] types,
                                             int[] posIncrements)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             String[] types,
                                             int[] posIncrements,
                                             int[] posLengths)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             String[] types)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] posIncrements)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             Integer finalOffset)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             int[] posIncrements)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             int[] posIncrements,
                                             Integer finalOffset)
                                      throws IOException

Throws:

IOException

assertTokenStreamContents

public static void assertTokenStreamContents(TokenStream ts,
                                             String[] output,
                                             int[] startOffsets,
                                             int[] endOffsets,
                                             int[] posIncrements,
                                             int[] posLengths,
                                             Integer finalOffset)
                                      throws IOException

Throws:

IOException

assertAnalyzesTo

public static void assertAnalyzesTo(Analyzer a,
                                    String input,
                                    String[] output,
                                    int[] startOffsets,
                                    int[] endOffsets,
                                    String[] types,
                                    int[] posIncrements)
                             throws IOException

Throws:

IOException

assertAnalyzesTo

public static void assertAnalyzesTo(Analyzer a,
                                    String input,
                                    String[] output,
                                    int[] startOffsets,
                                    int[] endOffsets,
                                    String[] types,
                                    int[] posIncrements,
                                    int[] posLengths)
                             throws IOException

Throws:

IOException

assertAnalyzesTo

public static void assertAnalyzesTo(Analyzer a,
                                    String input,
                                    String[] output,
                                    int[] startOffsets,
                                    int[] endOffsets,
                                    String[] types,
                                    int[] posIncrements,
                                    int[] posLengths,
                                    boolean graphOffsetsAreCorrect)
                             throws IOException

Throws:

IOException

assertAnalyzesTo

public static void assertAnalyzesTo(Analyzer a,
                                    String input,
                                    String[] output,
                                    int[] startOffsets,
                                    int[] endOffsets,
                                    String[] types,
                                    int[] posIncrements,
                                    int[] posLengths,
                                    boolean graphOffsetsAreCorrect,
                                    byte[][] payloads)
                             throws IOException

Throws:

IOException

assertAnalyzesTo

public static void assertAnalyzesTo(Analyzer a,
                                    String input,
                                    String[] output)
                             throws IOException

Throws:

IOException

assertAnalyzesTo

public static void assertAnalyzesTo(Analyzer a,
                                    String input,
                                    String[] output,
                                    String[] types)
                             throws IOException

Throws:

IOException

assertAnalyzesTo

public static void assertAnalyzesTo(Analyzer a,
                                    String input,
                                    String[] output,
                                    int[] posIncrements)
                             throws IOException

Throws:

IOException

assertAnalyzesToPositions

public static void assertAnalyzesToPositions(Analyzer a,
                                             String input,
                                             String[] output,
                                             int[] posIncrements,
                                             int[] posLengths)
                                      throws IOException

Throws:

IOException

assertAnalyzesToPositions

public static void assertAnalyzesToPositions(Analyzer a,
                                             String input,
                                             String[] output,
                                             String[] types,
                                             int[] posIncrements,
                                             int[] posLengths)
                                      throws IOException

Throws:

IOException

assertAnalyzesTo

public static void assertAnalyzesTo(Analyzer a,
                                    String input,
                                    String[] output,
                                    int[] startOffsets,
                                    int[] endOffsets)
                             throws IOException

Throws:

IOException

assertAnalyzesTo

public static void assertAnalyzesTo(Analyzer a,
                                    String input,
                                    String[] output,
                                    int[] startOffsets,
                                    int[] endOffsets,
                                    int[] posIncrements)
                             throws IOException

Throws:

IOException

checkOneTerm

public static void checkOneTerm(Analyzer a,
                                String input,
                                String expected)
                         throws IOException

Throws:

IOException

checkRandomData

public static void checkRandomData(Random random,
                                   Analyzer a,
                                   int iterations)
                            throws IOException

utility method for blasting tokenstreams with data to make sure they don't do anything crazy

Throws:

IOException

checkRandomData

public static void checkRandomData(Random random,
                                   Analyzer a,
                                   int iterations,
                                   int maxWordLength)
                            throws IOException

utility method for blasting tokenstreams with data to make sure they don't do anything crazy

Throws:

IOException

checkRandomData

public static void checkRandomData(Random random,
                                   Analyzer a,
                                   int iterations,
                                   boolean simple)
                            throws IOException

utility method for blasting tokenstreams with data to make sure they don't do anything crazy

Parameters:

simple - true if only ascii strings will be used (try to avoid)

Throws:

IOException

assertStreamHasNumberOfTokens

public static void assertStreamHasNumberOfTokens(TokenStream ts,
                                                 int expectedCount)
                                          throws IOException

Asserts that the given stream has expected number of tokens.

Throws:

IOException

checkRandomData

public static void checkRandomData(Random random,
                                   Analyzer a,
                                   int iterations,
                                   int maxWordLength,
                                   boolean simple)
                            throws IOException

Throws:

IOException

checkRandomData

public static void checkRandomData(Random random,
                                   Analyzer a,
                                   int iterations,
                                   int maxWordLength,
                                   boolean simple,
                                   boolean graphOffsetsAreCorrect)
                            throws IOException

Throws:

IOException

escape

public static String escape(String s)

checkAnalysisConsistency

public static void checkAnalysisConsistency(Random random,
                                            Analyzer a,
                                            boolean useCharFilter,
                                            String text)
                                     throws IOException

Throws:

IOException

checkAnalysisConsistency

public static void checkAnalysisConsistency(Random random,
                                            Analyzer a,
                                            boolean useCharFilter,
                                            String text,
                                            boolean graphOffsetsAreCorrect)
                                     throws IOException

Throws:

IOException

toDot

protected String toDot(Analyzer a,
                       String inputText)
                throws IOException

Throws:

IOException

toDotFile

protected void toDotFile(Analyzer a,
                         String inputText,
                         String localFileName)
                  throws IOException

Throws:

IOException

whitespaceMockTokenizer

protected static MockTokenizer whitespaceMockTokenizer(Reader input)
                                                throws IOException

Throws:

IOException

whitespaceMockTokenizer

protected static MockTokenizer whitespaceMockTokenizer(String input)
                                                throws IOException

Throws:

IOException

keywordMockTokenizer

protected static MockTokenizer keywordMockTokenizer(Reader input)
                                             throws IOException

Throws:

IOException

keywordMockTokenizer

protected static MockTokenizer keywordMockTokenizer(String input)
                                             throws IOException

Throws:

IOException

newAttributeFactory

public static AttributeFactory newAttributeFactory(Random random)

Returns a random AttributeFactory impl

newAttributeFactory

public static AttributeFactory newAttributeFactory()

Returns a random AttributeFactory impl

assertGraphStrings

public static void assertGraphStrings(Analyzer analyzer,
                                      String text,
                                      String... expectedStrings)
                               throws IOException

Enumerates all accepted strings in the token graph created by the analyzer on the provided text, and then asserts that it's equal to the expected strings. Uses TokenStreamToAutomaton to create an automaton. Asserts the finite strings of the automaton are all and only the given valid strings.

Parameters:

analyzer - analyzer containing the SynonymFilter under test.

text - text to be analyzed.

expectedStrings - all expected finite strings.

Throws:

IOException

assertGraphStrings

public static void assertGraphStrings(TokenStream tokenStream,
                                      String... expectedStrings)
                               throws IOException

Enumerates all accepted strings in the token graph created by the already initialized TokenStream.

Throws:

IOException

getGraphStrings

public static Set<String> getGraphStrings(Analyzer analyzer,
                                          String text)
                                   throws IOException

Returns all paths accepted by the token stream graph produced by analyzing text with the provided analyzer. The tokens CharTermAttribute values are concatenated, and separated with space.

Throws:

IOException

getGraphStrings

public static Set<String> getGraphStrings(TokenStream tokenStream)
                                   throws IOException

Returns all paths accepted by the token stream graph produced by the already initialized TokenStream.

Throws:

IOException

toString

public static String toString(Analyzer analyzer,
                              String text)
                       throws IOException

Returns a String summary of the tokens this analyzer produces on this text

Throws:

IOException