This week's book giveaway is in the Java in General forum.
We're giving away four copies of Event Streams in Action and have Alexander Dean & Valentin Crettaz on-line!
See this thread for details.
Win a copy of Event Streams in Action this week in the Java in General forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Devaka Cooray
  • Liutauras Vilda
  • Jeanne Boyarsky
  • Bear Bibeault
Sheriffs:
  • Paul Clapham
  • Knute Snortum
  • Rob Spoor
Saloon Keepers:
  • Tim Moores
  • Ron McLeod
  • Piet Souris
  • Stephan van Hulst
  • Carey Brown
Bartenders:
  • Tim Holloway
  • Frits Walraven
  • Ganesh Patekar

Issue with StreamTokenizer?

 
Ranch Hand
Posts: 242
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I was fooling around with StreamTokenizer to create a SubClass, which can return comments as tokens. In the process, I stumbled into something very puzzling. See the code.



Executing the above code displays:



While parsing the "hello" and "hello, why didn't it return the hello world token?

Any idea?

[ June 28, 2008: Message edited by: Santhosh Kumar ]
[ June 28, 2008: Message edited by: Santhosh Kumar ]
 
Ranch Hand
Posts: 262
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
In the case of "hello", it's because the token is a quoted string, not a word. The API doc of the ttype field says

For a quoted string token, its value is the quote character.

As for "hello, I'd say the output is meaningless since the input is malformed (I'd say the same about hello", too).
 
Ranch Hand
Posts: 457
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
To restate Alan's reply, the documentation for the ttype instance variable gives the information.

This value is also what is being returned and you are setting to token.

There are two values that the tokenizer doesn't define a special code for. The character is your fall-through catch, but the other is the quoted string (which they should have defined another value for - i did it in my modification to your code...).

Here's the code:
 
Santhosh Kumar
Ranch Hand
Posts: 242
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you. That was really helpful.
 
That's a very big dog. I think I want to go home now and hug this tiny ad:
Java Code Review and Psychology
https://coderanch.com/t/714798/java/Java-Code-Review-Psychology
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!