Win a copy of Functional Reactive Programming this week in the Other Languages forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

tokenizer

 
henry wu
Greenhorn
Posts: 23
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi everyone,
I made a StreamTokenizer object to capture keyboard input from the user. I want to be able to input numbers like "001" and "002" but the in tokenizer.nval it only captures "1" and "2". Is there any way around this?
StreamTokenizer tokenizer = new StreamTokenizer(new InputStreamReader(System.in));
if (tokenizer.nextToken()==tokenizer.TT_NUMBER)
{ return (int)tokenizer.nval; }
 
Dan Chisholm
Ranch Hand
Posts: 1865
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It sounds like you want a string representation of the input numbers so why not just read them as a String using nextToken()?
 
henry wu
Greenhorn
Posts: 23
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I tried doing this:
if (tokenizer.nextToken()==tokenizer.TT_WORD) {
return tokenizer.sval;
}
else {
System.out.println("This is not a valid string\n");
}
When I entered in '001' It told me that it wasn't a valid string.. so it didn't pass the tokenizer.TT_WORD test. Any ideas?
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Evidently StreamTokenizer considers 001 to be a number, not a word, and it considers 001 to be exactly equivalent to 1. If it's important to you that 001 is different from 1, you'll need to tell the StreamTokenizer to treat numeric characters as parts of words:
 
henry wu
Greenhorn
Posts: 23
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
cool thanks.. I'll try this out!
 
henry wu
Greenhorn
Posts: 23
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ok the new code looks like this:
tokenizer.wordChars('0','9');
if (tokenizer.nextToken()==tokenizer.TT_WORD) {
return tokenizer.sval;
}
else {
System.out.println("This is not a valid string\n");
}
But it is still not recognizing '001' as a word.. Am I placing the wordChars function call in the wrong place?
 
Thomas Paul
mister krabs
Ranch Hand
Posts: 13974
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
wordChars takes two ints as parameters. Try it without the single quotes.
 
henry wu
Greenhorn
Posts: 23
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ok, changed to -
tokenizer.wordChars(0,9);
if (tokenizer.nextToken()==tokenizer.TT_WORD) {
return tokenizer.sval;
}
else {
System.out.println("This is not a valid string\n");
}
Still didn't work..
Do you guys have any recommendations for keyboard input for integers and strings other than StreamTokenizer and BufferedReader?
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
No, Thomas is on the wrong trail here. Passing int values of 0 and 9 just ends up referring to Unicode values of 0-9. You definitely want the char values, which are '0' and '9'. I think they only use int in the API so that people can use int literals without casting if they wish. Rather silly, and misleading.
Looking further in the API, it appears that characters can be both number chars and word chars simultaneously. Evidently the code checks for number chars before it checks for word chars, and so it ignores the fact that we told it that '0'-'9' are word chars. Instead, we must explicitly tell the tokenizer that '0'-'9' are not numbers:

Note that ordinaryChars() wipes out any special attributes associated with the given chars, so it's necessary to assert the word nature of the chars after ordinaryChars().
On inspection, it appears that StreamTokenizer has one of the more bizarre and confusing API's I've seen from Sun. Maybe even worse than Calendar. Ugh. Depending what you're doing, you may well find it's easier to write your own parsing classes.
 
henry wu
Greenhorn
Posts: 23
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
cool.. that worked! Thanks for all the help!
should i have used stringTokenizer from the start?
[ July 10, 2002: Message edited by: henry wu ]
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic