StringTokenizer Tokenizing String

StringTokenizer Tokenizing String

After knowing the data structures from java.util package, let us discuss about the remaining classes of java.util package. The classes include Date and GregorianCalendar to manipulate dates, Random class to generate random numbers and StringTokenizer to separate the string into individual words (known as tokens). The other classes less used are Observer, Observable, Timer and TimerTask etc. Now let us go through these classes one after another.

class StringTokenizer

We know earlier, StreamTokenizer from package which tokenizes a stream. Now let us discuss a similar class StringTokenizer that tokenizes a string. It is from java.util package. Each word in a file/string is called as a token. The StringTokenizer class tokenizes the string either on the basis of default delimiter, whitespace, or the delimiters supplied by the programmer.

Following is the class signature as defined in java.util pakage.

public class StringTokenizer extends Object implements Enumeration

Following program on "StringTokenizer Tokenizing String" uses default whitespace as delimiter. The next program tokenizes using our own delimiters.

StringTokenizer Tokenizing String

StringTokenizer st1 = new StringTokenizer(str1);

The string to be tokenized, str1, is passed as parameter to the StringTokenizer constructor. The countTokens() method of StringTokenizer returns the number of tokens (words) available in the string str1. The StringTokenizer tokenizes taking, by default, whitespace as delimiter.


To print the tokens available with StringTokenizer object, without knowing how many tokens exist, the designers followed a simple programming technique; the same technique followed with java.util.Enumeration and java.sql.ResultSet . The hasMoreElements() method returns a boolean value of true as long as tokens exist with the StringTokenizer object to return. If all the tokens are returned, the method returns false and the loop terminates. The nextToken() returns each token one-by-one for each iteration. The technique involves the usage of two methods; one that returns a boolean value used to iterate and the other that returns the value.

2 thoughts on “StringTokenizer Tokenizing String

Leave a Reply

Your email address will not be published. Required fields are marked *