Class LowerCaseTokenizer

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable

    public final class LowerCaseTokenizer
    extends LetterTokenizer
    LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.

    Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

    You must specify the required Version compatibility when creating LowerCaseTokenizer:

    • Constructor Detail

      • LowerCaseTokenizer

        public LowerCaseTokenizer​(Version matchVersion,
                                  java.io.Reader in)
        Construct a new LowerCaseTokenizer.
        Parameters:
        matchVersion - Lucene version to match See {@link above}
        in - the input to split up into tokens