Breaking the data into independent pieces is known as tokenization. For example, breaking a sentence into different words is known as tokenization.
To do the job, Java comes with two tokenizers – StringTokenizer to tokenize a string and StreamTokenizer to tokenize a stream of data coming from I/O stream classes. Both are for very different in their jobs.
Both are discussed with examples in StringTokenizer and StreamTokenizer.
