String Tokenization Java

Breaking a string into independent words is known as string tokenization. To do the job, Java comes with class StringTokenizer from java.util package. It is so simple to use, just pass the string to the constructor of StringTokenizer. Over, everything is done and StringTokenizer places all the words (known as tokens) in your hand.

Following is an Example on String Tokenization.

import java.util.StringTokenizer;
public class STDemo
{
  public static void main(String args[])
  {
    String str = "Hello World How Do You Do";

    StringTokenizer st = new StringTokenizer(str);
 
    int num = st.countTokens();
    System.out.println("Number of tokens in str: " + num);
                                               // to print each token
    System.out.println("The tokesn in str are:");
    while(st.hasMoreTokens())
    {
      System.out.print(st.nextToken() + ", ");
    }

    String str1 = "Ijamdokayqyoudarejokayqalljaredokay";
    StringTokenizer st1 = new StringTokenizer(str1, "j, d, q");
                                               // to print each token
    System.out.println("\n\nThe tokesn in str1 are:");
    while(st1.hasMoreTokens())
    {
      System.out.print(st1.nextToken() + ", ");
    }
  }
}

Output Screenshot on String Tokenization Java

StringTokenizer st = new StringTokenizer(str);

The string str to be tokenized is passed as parameter to StringTokenizer constructor.

countTokens() returns the number of tokens in string str;

while(st.hasMoreTokens())
{
System.out.print(st.nextToken() + “, “);
}

StringTokenizer comes with two methods hasMoreTokens() and nextToken(). hasMoreTokens() iterate all the tokens in a loop and nextToken() returns each token.

StringTokenizer st1 = new StringTokenizer(str1, “j, d, q”);

The StringTokenizer constructor is overloaded. In the above statement, the constructor takes two parameters of string to be tokenized and the second parameter is delimiters. A delimiter is a separator which separates each token in the string. In the previous statement the delimiter is whitespace. Here in this string str1, the delimiters are j, d and q.

Another example on String Tokenization is available at class StringTokenizer.

For more indepth study of Java see Java Differences on 80 Topics