• Post Reply Bookmark Topic Watch Topic
  • New Topic

StreamTokenizer  RSS feed

 
Chad Risch
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello, I am working on a tokenizer and have come to a problem. I'm using the StreamTokenizer class to parse (i.e.: SETPARAM '0COEFF', 1234, 5678.91011) into tokens. When doing so, I get back the following:
Word: SETPARAM
Quote: 0COEFF
Ordinary: ,
Number: 1234.0
Ordinary: ,
Number: 5678.91011
Well...I want that '1234' in my input string to be tokenized as a TT_WORD instead of a TT_NUMBER. In otherwords, if I have a SETPARAM, I want the second parameter to the SETPARAM to automatically be parsed as a TT_WORD(string). Any suggestions??? THanks in advance!
Chad
 
Cindy Glass
"The Hood"
Sheriff
Posts: 8521
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So either you just can NOT force a numeric to be a text - or none of us folks know HOW.
So why don't I move this to Advanced and see if someone figures this out.
 
David Weitzman
Ranch Hand
Posts: 1365
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
StreamTokenizer is evil. Bad design, bad performance. For maximal control you might find it easiest to build your own tokenizer with a parser-generator like ANTLR. If you haven't done it before there's a learning curve, but parsing is such a common activity that familiarity with a parser generator or two is a good thing.
 
Ernest Friedman-Hill
author and iconoclast
Sheriff
Posts: 24217
38
Chrome Eclipse IDE Mac OS X
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You just want to reset the "numeric" attribute of the characters 0...9, . and "-". Use ordinaryChars() to clear the "number" attribute for these characters, then use wordChars() to make them into word components. Don't skip the ordinaryChars() calls -- a character can be both a word component and number component.
See the javadoc for StreamTokenizer for more info.
 
Cindy Glass
"The Hood"
Sheriff
Posts: 8521
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hmmm . . . Maybe I was reading more into the question than there was. O thought that he was asking how to make the 1234 (integer) be text, but leave the 5678.91011 (decimal) as numeric.
 
Ernest Friedman-Hill
author and iconoclast
Sheriff
Posts: 24217
38
Chrome Eclipse IDE Mac OS X
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I believe you can twiddle the syntax table before retrieving each token, so he could set it up this way before reading that token, then set everything back... or he could read everything as a word, and then convert those things that he wanted as numbers using Double.parseDouble() or some such.
But I'm not doing very well at understanding questions today; must be tired.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!