Package org.apache.commons.lang.text

Class StrTokenizer

  • All Implemented Interfaces:
    java.lang.Cloneable, java.util.Iterator, java.util.ListIterator
    @Deprecated(since="2021-04-30")
public class StrTokenizer
extends java.lang.Object
implements java.util.ListIterator, java.lang.Cloneable
    Deprecated.
    Commons Lang 2 is in maintenance mode. Commons Lang 3 should be used instead.
    Tokenizes a string based based on delimiters (separators) and supporting quoting and ignored character concepts.

    This class can split a String into many smaller strings. It aims to do a similar job to StringTokenizer, however it offers much more control and flexibility including implementing the ListIterator interface. By default, it is set up like StringTokenizer.

    The input String is split into a number of tokens. Each token is separated from the next String by a delimiter. One or more delimiter characters must be specified.

    Each token may be surrounded by quotes. The quote matcher specifies the quote character(s). A quote may be escaped within a quoted section by duplicating itself.

    Between each token and the delimiter are potentially characters that need trimming. The trimmer matcher specifies these characters. One usage might be to trim whitespace characters.

    At any point outside the quotes there might potentially be invalid characters. The ignored matcher specifies these characters to be removed. One usage might be to remove new line characters.

    Empty tokens may be removed or returned as null. 

      "a,b,c"         - Three tokens "a","b","c"   (comma delimiter)
  " a, b , c "    - Three tokens "a","b","c"   (default CSV processing trims whitespace)
  "a, ", b ,", c" - Three tokens "a, " , " b ", ", c" (quoted text untouched)

    This tokenizer has the following properties and options:

    PropertyTypeDefault
    delimCharSetMatcher{ \t\n\r\f}
    quoteNoneMatcher{}
    ignoreNoneMatcher{}
    emptyTokenAsNullbooleanfalse
    ignoreEmptyTokensbooleantrue
    Since:
    2.2

    • Constructor Summary

      Constructors 
      Constructor Description
      StrTokenizer()
      Deprecated.
      Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer, but with no text to tokenize.
      StrTokenizer​(char[] input)
      Deprecated.
      Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer.
      StrTokenizer​(char[] input, char delim)
      Deprecated.
      Constructs a tokenizer splitting on the specified character.
      StrTokenizer​(char[] input, char delim, char quote)
      Deprecated.
      Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
      StrTokenizer​(char[] input, java.lang.String delim)
      Deprecated.
      Constructs a tokenizer splitting on the specified string.
      StrTokenizer​(char[] input, StrMatcher delim)
      Deprecated.
      Constructs a tokenizer splitting using the specified delimiter matcher.
      StrTokenizer​(char[] input, StrMatcher delim, StrMatcher quote)
      Deprecated.
      Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.
      StrTokenizer​(java.lang.String input)
      Deprecated.
      Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer.
      StrTokenizer​(java.lang.String input, char delim)
      Deprecated.
      Constructs a tokenizer splitting on the specified delimiter character.
      StrTokenizer​(java.lang.String input, char delim, char quote)
      Deprecated.
      Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
      StrTokenizer​(java.lang.String input, java.lang.String delim)
      Deprecated.
      Constructs a tokenizer splitting on the specified delimiter string.
      StrTokenizer​(java.lang.String input, StrMatcher delim)
      Deprecated.
      Constructs a tokenizer splitting using the specified delimiter matcher.
      StrTokenizer​(java.lang.String input, StrMatcher delim, StrMatcher quote)
      Deprecated.
      Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods 
      Modifier and Type Method Description
      void add​(java.lang.Object obj)
      Deprecated.
      Unsupported ListIterator operation.
      java.lang.Object clone()
      Deprecated.
      Creates a new instance of this Tokenizer.
      java.lang.String getContent()
      Deprecated.
      Gets the String content that the tokenizer is parsing.
      static StrTokenizer getCSVInstance()
      Deprecated.
      Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
      static StrTokenizer getCSVInstance​(char[] input)
      Deprecated.
      Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
      static StrTokenizer getCSVInstance​(java.lang.String input)
      Deprecated.
      Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
      StrMatcher getDelimiterMatcher()
      Deprecated.
      Gets the field delimiter matcher.
      StrMatcher getIgnoredMatcher()
      Deprecated.
      Gets the ignored character matcher.
      StrMatcher getQuoteMatcher()
      Deprecated.
      Gets the quote matcher currently in use.
      java.lang.String[] getTokenArray()
      Deprecated.
      Gets a copy of the full token list as an independent modifiable array.
      java.util.List getTokenList()
      Deprecated.
      Gets a copy of the full token list as an independent modifiable list.
      StrMatcher getTrimmerMatcher()
      Deprecated.
      Gets the trimmer character matcher.
      static StrTokenizer getTSVInstance()
      Deprecated.
      Gets a new tokenizer instance which parses Tab Separated Value strings.
      static StrTokenizer getTSVInstance​(char[] input)
      Deprecated.
      Gets a new tokenizer instance which parses Tab Separated Value strings.
      static StrTokenizer getTSVInstance​(java.lang.String input)
      Deprecated.
      Gets a new tokenizer instance which parses Tab Separated Value strings.
      boolean hasNext()
      Deprecated.
      Checks whether there are any more tokens.
      boolean hasPrevious()
      Deprecated.
      Checks whether there are any previous tokens that can be iterated to.
      boolean isEmptyTokenAsNull()
      Deprecated.
      Gets whether the tokenizer currently returns empty tokens as null.
      boolean isIgnoreEmptyTokens()
      Deprecated.
      Gets whether the tokenizer currently ignores empty tokens.
      java.lang.Object next()
      Deprecated.
      Gets the next token.
      int nextIndex()
      Deprecated.
      Gets the index of the next token to return.
      java.lang.String nextToken()
      Deprecated.
      Gets the next token from the String.
      java.lang.Object previous()
      Deprecated.
      Gets the token previous to the last returned token.
      int previousIndex()
      Deprecated.
      Gets the index of the previous token.
      java.lang.String previousToken()
      Deprecated.
      Gets the previous token from the String.
      void remove()
      Deprecated.
      Unsupported ListIterator operation.
      StrTokenizer reset()
      Deprecated.
      Resets this tokenizer, forgetting all parsing and iteration already completed.
      StrTokenizer reset​(char[] input)
      Deprecated.
      Reset this tokenizer, giving it a new input string to parse.
      StrTokenizer reset​(java.lang.String input)
      Deprecated.
      Reset this tokenizer, giving it a new input string to parse.
      void set​(java.lang.Object obj)
      Deprecated.
      Unsupported ListIterator operation.
      StrTokenizer setDelimiterChar​(char delim)
      Deprecated.
      Sets the field delimiter character.
      StrTokenizer setDelimiterMatcher​(StrMatcher delim)
      Deprecated.
      Sets the field delimiter matcher.
      StrTokenizer setDelimiterString​(java.lang.String delim)
      Deprecated.
      Sets the field delimiter string.
      StrTokenizer setEmptyTokenAsNull​(boolean emptyAsNull)
      Deprecated.
      Sets whether the tokenizer should return empty tokens as null.
      StrTokenizer setIgnoredChar​(char ignored)
      Deprecated.
      Set the character to ignore.
      StrTokenizer setIgnoredMatcher​(StrMatcher ignored)
      Deprecated.
      Set the matcher for characters to ignore.
      StrTokenizer setIgnoreEmptyTokens​(boolean ignoreEmptyTokens)
      Deprecated.
      Sets whether the tokenizer should ignore and not return empty tokens.
      StrTokenizer setQuoteChar​(char quote)
      Deprecated.
      Sets the quote character to use.
      StrTokenizer setQuoteMatcher​(StrMatcher quote)
      Deprecated.
      Set the quote matcher to use.
      StrTokenizer setTrimmerMatcher​(StrMatcher trimmer)
      Deprecated.
      Sets the matcher for characters to trim.
      int size()
      Deprecated.
      Gets the number of tokens found in the String.
      java.lang.String toString()
      Deprecated.
      Gets the String content that the tokenizer is parsing.

      • Methods inherited from class java.lang.Object

        equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

      • Methods inherited from interface java.util.Iterator

        forEachRemaining

    • Constructor Detail

      • StrTokenizer

        public StrTokenizer()
        Deprecated.
        Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer, but with no text to tokenize.

        This constructor is normally used with reset(String).

      • StrTokenizer

        public StrTokenizer​(java.lang.String input)
        Deprecated.
        Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer.
        Parameters:
        input - the string which is to be parsed

      • StrTokenizer

        public StrTokenizer​(java.lang.String input,
                    char delim)
        Deprecated.
        Constructs a tokenizer splitting on the specified delimiter character.
        Parameters:
        input - the string which is to be parsed
        delim - the field delimiter character

      • StrTokenizer

        public StrTokenizer​(java.lang.String input,
                    java.lang.String delim)
        Deprecated.
        Constructs a tokenizer splitting on the specified delimiter string.
        Parameters:
        input - the string which is to be parsed
        delim - the field delimiter string

      • StrTokenizer

        public StrTokenizer​(java.lang.String input,
                    StrMatcher delim)
        Deprecated.
        Constructs a tokenizer splitting using the specified delimiter matcher.
        Parameters:
        input - the string which is to be parsed
        delim - the field delimiter matcher

      • StrTokenizer

        public StrTokenizer​(java.lang.String input,
                    char delim,
                    char quote)
        Deprecated.
        Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
        Parameters:
        input - the string which is to be parsed
        delim - the field delimiter character
        quote - the field quoted string character

      • StrTokenizer

        public StrTokenizer​(java.lang.String input,
                    StrMatcher delim,
                    StrMatcher quote)
        Deprecated.
        Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.
        Parameters:
        input - the string which is to be parsed
        delim - the field delimiter matcher
        quote - the field quoted string matcher

      • StrTokenizer

        public StrTokenizer​(char[] input)
        Deprecated.
        Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer.

        The input character array is not cloned, and must not be altered after passing in to this method.

        Parameters:
        input - the string which is to be parsed, not cloned

      • StrTokenizer

        public StrTokenizer​(char[] input,
                    char delim)
        Deprecated.
        Constructs a tokenizer splitting on the specified character.

        The input character array is not cloned, and must not be altered after passing in to this method.

        Parameters:
        input - the string which is to be parsed, not cloned
        delim - the field delimiter character

      • StrTokenizer

        public StrTokenizer​(char[] input,
                    java.lang.String delim)
        Deprecated.
        Constructs a tokenizer splitting on the specified string.

        The input character array is not cloned, and must not be altered after passing in to this method.

        Parameters:
        input - the string which is to be parsed, not cloned
        delim - the field delimiter string

      • StrTokenizer

        public StrTokenizer​(char[] input,
                    StrMatcher delim)
        Deprecated.
        Constructs a tokenizer splitting using the specified delimiter matcher.

        The input character array is not cloned, and must not be altered after passing in to this method.

        Parameters:
        input - the string which is to be parsed, not cloned
        delim - the field delimiter matcher

      • StrTokenizer

        public StrTokenizer​(char[] input,
                    char delim,
                    char quote)
        Deprecated.
        Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.

        The input character array is not cloned, and must not be altered after passing in to this method.

        Parameters:
        input - the string which is to be parsed, not cloned
        delim - the field delimiter character
        quote - the field quoted string character

      • StrTokenizer

        public StrTokenizer​(char[] input,
                    StrMatcher delim,
                    StrMatcher quote)
        Deprecated.
        Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.

        The input character array is not cloned, and must not be altered after passing in to this method.

        Parameters:
        input - the string which is to be parsed, not cloned
        delim - the field delimiter character
        quote - the field quoted string character

    • Method Detail

      • getCSVInstance

        public static StrTokenizer getCSVInstance()
        Deprecated.
        Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

        You must call a "reset" method to set the string which you want to parse.

        Returns:
        a new tokenizer instance which parses Comma Separated Value strings

      • getCSVInstance

        public static StrTokenizer getCSVInstance​(java.lang.String input)
        Deprecated.
        Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).
        Parameters:
        input - the text to parse
        Returns:
        a new tokenizer instance which parses Comma Separated Value strings

      • getCSVInstance

        public static StrTokenizer getCSVInstance​(char[] input)
        Deprecated.
        Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).
        Parameters:
        input - the text to parse
        Returns:
        a new tokenizer instance which parses Comma Separated Value strings

      • getTSVInstance

        public static StrTokenizer getTSVInstance()
        Deprecated.
        Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

        You must call a "reset" method to set the string which you want to parse.

        Returns:
        a new tokenizer instance which parses Tab Separated Value strings.

      • getTSVInstance

        public static StrTokenizer getTSVInstance​(java.lang.String input)
        Deprecated.
        Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).
        Parameters:
        input - the string to parse
        Returns:
        a new tokenizer instance which parses Tab Separated Value strings.

      • getTSVInstance

        public static StrTokenizer getTSVInstance​(char[] input)
        Deprecated.
        Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).
        Parameters:
        input - the string to parse
        Returns:
        a new tokenizer instance which parses Tab Separated Value strings.

      • size

        public int size()
        Deprecated.
        Gets the number of tokens found in the String.
        Returns:
        the number of matched tokens

      • nextToken

        public java.lang.String nextToken()
        Deprecated.
        Gets the next token from the String. Equivalent to next() except it returns null rather than throwing NoSuchElementException when no tokens remain.
        Returns:
        the next sequential token, or null when no more tokens are found

      • previousToken

        public java.lang.String previousToken()
        Deprecated.
        Gets the previous token from the String.
        Returns:
        the previous sequential token, or null when no more tokens are found

      • getTokenArray

        public java.lang.String[] getTokenArray()
        Deprecated.
        Gets a copy of the full token list as an independent modifiable array.
        Returns:
        the tokens as a String array

      • getTokenList

        public java.util.List getTokenList()
        Deprecated.
        Gets a copy of the full token list as an independent modifiable list.
        Returns:
        the tokens as a String array

      • reset

        public StrTokenizer reset()
        Deprecated.
        Resets this tokenizer, forgetting all parsing and iteration already completed.

        This method allows the same tokenizer to be reused for the same String.

        Returns:
        this, to enable chaining

      • reset

        public StrTokenizer reset​(java.lang.String input)
        Deprecated.
        Reset this tokenizer, giving it a new input string to parse. In this manner you can re-use a tokenizer with the same settings on multiple input lines.
        Parameters:
        input - the new string to tokenize, null sets no text to parse
        Returns:
        this, to enable chaining

      • reset

        public StrTokenizer reset​(char[] input)
        Deprecated.
        Reset this tokenizer, giving it a new input string to parse. In this manner you can re-use a tokenizer with the same settings on multiple input lines.

        The input character array is not cloned, and must not be altered after passing in to this method.

        Parameters:
        input - the new character array to tokenize, not cloned, null sets no text to parse
        Returns:
        this, to enable chaining

      • hasNext

        public boolean hasNext()
        Deprecated.
        Checks whether there are any more tokens.
        Specified by:
        hasNext in interface java.util.Iterator
        Specified by:
        hasNext in interface java.util.ListIterator
        Returns:
        true if there are more tokens

      • next

        public java.lang.Object next()
        Deprecated.
        Gets the next token.
        Specified by:
        next in interface java.util.Iterator
        Specified by:
        next in interface java.util.ListIterator
        Returns:
        the next String token
        Throws:
        java.util.NoSuchElementException - if there are no more elements

      • nextIndex

        public int nextIndex()
        Deprecated.
        Gets the index of the next token to return.
        Specified by:
        nextIndex in interface java.util.ListIterator
        Returns:
        the next token index

      • hasPrevious

        public boolean hasPrevious()
        Deprecated.
        Checks whether there are any previous tokens that can be iterated to.
        Specified by:
        hasPrevious in interface java.util.ListIterator
        Returns:
        true if there are previous tokens

      • previous

        public java.lang.Object previous()
        Deprecated.
        Gets the token previous to the last returned token.
        Specified by:
        previous in interface java.util.ListIterator
        Returns:
        the previous token

      • previousIndex

        public int previousIndex()
        Deprecated.
        Gets the index of the previous token.
        Specified by:
        previousIndex in interface java.util.ListIterator
        Returns:
        the previous token index

      • remove

        public void remove()
        Deprecated.
        Unsupported ListIterator operation.
        Specified by:
        remove in interface java.util.Iterator
        Specified by:
        remove in interface java.util.ListIterator
        Throws:
        java.lang.UnsupportedOperationException - always

      • set

        public void set​(java.lang.Object obj)
        Deprecated.
        Unsupported ListIterator operation.
        Specified by:
        set in interface java.util.ListIterator
        Parameters:
        obj - this parameter ignored.
        Throws:
        java.lang.UnsupportedOperationException - always

      • add

        public void add​(java.lang.Object obj)
        Deprecated.
        Unsupported ListIterator operation.
        Specified by:
        add in interface java.util.ListIterator
        Parameters:
        obj - this parameter ignored.
        Throws:
        java.lang.UnsupportedOperationException - always

      • getDelimiterMatcher

        public StrMatcher getDelimiterMatcher()
        Deprecated.
        Gets the field delimiter matcher.
        Returns:
        the delimiter matcher in use

      • setDelimiterMatcher

        public StrTokenizer setDelimiterMatcher​(StrMatcher delim)
        Deprecated.
        Sets the field delimiter matcher.

        The delimitier is used to separate one token from another.

        Parameters:
        delim - the delimiter matcher to use
        Returns:
        this, to enable chaining

      • setDelimiterChar

        public StrTokenizer setDelimiterChar​(char delim)
        Deprecated.
        Sets the field delimiter character.
        Parameters:
        delim - the delimiter character to use
        Returns:
        this, to enable chaining

      • setDelimiterString

        public StrTokenizer setDelimiterString​(java.lang.String delim)
        Deprecated.
        Sets the field delimiter string.
        Parameters:
        delim - the delimiter string to use
        Returns:
        this, to enable chaining

      • getQuoteMatcher

        public StrMatcher getQuoteMatcher()
        Deprecated.
        Gets the quote matcher currently in use.

        The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data. The default value is '"' (double quote).

        Returns:
        the quote matcher in use

      • setQuoteMatcher

        public StrTokenizer setQuoteMatcher​(StrMatcher quote)
        Deprecated.
        Set the quote matcher to use.

        The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data.

        Parameters:
        quote - the quote matcher to use, null ignored
        Returns:
        this, to enable chaining

      • setQuoteChar

        public StrTokenizer setQuoteChar​(char quote)
        Deprecated.
        Sets the quote character to use.

        The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data.

        Parameters:
        quote - the quote character to use
        Returns:
        this, to enable chaining

      • getIgnoredMatcher

        public StrMatcher getIgnoredMatcher()
        Deprecated.
        Gets the ignored character matcher.

        These characters are ignored when parsing the String, unless they are within a quoted region. The default value is not to ignore anything.

        Returns:
        the ignored matcher in use

      • setIgnoredMatcher

        public StrTokenizer setIgnoredMatcher​(StrMatcher ignored)
        Deprecated.
        Set the matcher for characters to ignore.

        These characters are ignored when parsing the String, unless they are within a quoted region.

        Parameters:
        ignored - the ignored matcher to use, null ignored
        Returns:
        this, to enable chaining

      • setIgnoredChar

        public StrTokenizer setIgnoredChar​(char ignored)
        Deprecated.
        Set the character to ignore.

        This character is ignored when parsing the String, unless it is within a quoted region.

        Parameters:
        ignored - the ignored character to use
        Returns:
        this, to enable chaining

      • getTrimmerMatcher

        public StrMatcher getTrimmerMatcher()
        Deprecated.
        Gets the trimmer character matcher.

        These characters are trimmed off on each side of the delimiter until the token or quote is found. The default value is not to trim anything.

        Returns:
        the trimmer matcher in use

      • setTrimmerMatcher

        public StrTokenizer setTrimmerMatcher​(StrMatcher trimmer)
        Deprecated.
        Sets the matcher for characters to trim.

        These characters are trimmed off on each side of the delimiter until the token or quote is found.

        Parameters:
        trimmer - the trimmer matcher to use, null ignored
        Returns:
        this, to enable chaining

      • isEmptyTokenAsNull

        public boolean isEmptyTokenAsNull()
        Deprecated.
        Gets whether the tokenizer currently returns empty tokens as null. The default for this property is false.
        Returns:
        true if empty tokens are returned as null

      • setEmptyTokenAsNull

        public StrTokenizer setEmptyTokenAsNull​(boolean emptyAsNull)
        Deprecated.
        Sets whether the tokenizer should return empty tokens as null. The default for this property is false.
        Parameters:
        emptyAsNull - whether empty tokens are returned as null
        Returns:
        this, to enable chaining

      • isIgnoreEmptyTokens

        public boolean isIgnoreEmptyTokens()
        Deprecated.
        Gets whether the tokenizer currently ignores empty tokens. The default for this property is true.
        Returns:
        true if empty tokens are not returned

      • setIgnoreEmptyTokens

        public StrTokenizer setIgnoreEmptyTokens​(boolean ignoreEmptyTokens)
        Deprecated.
        Sets whether the tokenizer should ignore and not return empty tokens. The default for this property is true.
        Parameters:
        ignoreEmptyTokens - whether empty tokens are not returned
        Returns:
        this, to enable chaining

      • getContent

        public java.lang.String getContent()
        Deprecated.
        Gets the String content that the tokenizer is parsing.
        Returns:
        the string content being parsed

      • clone

        public java.lang.Object clone()
        Deprecated.
        Creates a new instance of this Tokenizer. The new instance is reset so that it will be at the start of the token list. If a CloneNotSupportedException is caught, return null.
        Returns:
        a new instance of this Tokenizer which has been reset.

      • toString

        public java.lang.String toString()
        Deprecated.
        Gets the String content that the tokenizer is parsing.
        Overrides:
        toString in class java.lang.Object
        Returns:
        the string content being parsed