Tag: ttype

StreamTokenizer Tokenize Stream

April 18, 2011 · 3 min read
Breaking a string or stream into meaningful independent words is known as **tokenization**. Tokenization is a common practice to tool developers. **java.util** package includes **[StringTokenizer](http://way2java.com/java-util/class-stringtokenizer/)** which tokenizes a string into independent words. For StringTokenizer, the source is a string. There comes a similar tokenizer, **StreamTokenize** with **java.io** package for which source is a stream. Here, the StreamTokenizer tokenizes a whole stream into independent words.