Class MockTokenizer

  • All Implemented Interfaces:
    Closeable, AutoCloseable

    public class MockTokenizer
    extends Tokenizer
    Tokenizer for testing.

    This tokenizer is a replacement for WHITESPACE, SIMPLE, and KEYWORD tokenizers. If you are writing a component such as a TokenFilter, it's a great idea to test it wrapping this tokenizer instead for extra checks. This tokenizer has the following behavior:

    • An internal state-machine is used for checking consumer consistency. These checks can be disabled with setEnableChecks(boolean).
    • For convenience, optionally lowercases terms that it outputs.