• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Junilu Lacar
Sheriffs:
  • Rob Spoor
  • Liutauras Vilda
  • Tim Cooke
Saloon Keepers:
  • Tim Moores
  • Piet Souris
  • Tim Holloway
  • Jj Roberts
  • Stephan van Hulst
Bartenders:
  • Himai Minh
  • Carey Brown
  • Frits Walraven

If StringTokenizer is no longer recommended, then what?

 
Greenhorn
Posts: 18
VI Editor Chrome
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am writing a program to parse a homegrown kind of pseudo-code, where the delimiters are almost never white space (except newlines) and they change as you go along, depending on what has been read so far.

E.g. text enclosed in curly braces is treated specially, and in fact you might have nested curly-brace-enclosed phrases. So when I start processing a given line I'm only looking for opening curly braces, but once I find one then I want to be looking for both opening and closing braces.

In general I know we are now supposed to use String.split or Pattern or some other regex-based method rather than StringTokenizer to parse strings or other input. I don't think those methods are the best for my application, but I wanted to get some Expert Opinions in case there's some way of using them that I haven't thought of. After all, if Sun considers StringTokenizer obsolete, then the capability I need must already exist in some other package, right? I just haven't found it yet.

It seems to me that my application is a really good fit for StringTokenizer, unless someone can suggest a good reason why I shouldn't use it (like, if it's going away in Java 6 or something ).

Thanks,
b.
[ November 25, 2006: Message edited by: E McKenney ]
 
author
Posts: 23901
142
jQuery Eclipse IDE Firefox Browser VI Editor C++ Chrome Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

It seems to me that my application is a really good fit for StringTokenizer, unless someone can suggest a good reason why I shouldn't use it (like, if it's going away in Java 6 or something).



I don't think StringTokenizer is going away anytime soon. So if it works, I don't see a problem with it.

Henry
 
Henry Wong
author
Posts: 23901
142
jQuery Eclipse IDE Firefox Browser VI Editor C++ Chrome Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

In general I know we are now supposed to use String.split or Pattern or some other regex-based method rather than StringTokenizer to parse strings or other input. I don't think those methods are the best for my application, but I wanted to get some Expert Opinions in case there's some way of using them that I haven't thought of. After all, if Sun considers StringTokenizer obsolete, then the capability I need must already exist in some other package, right? I just haven't found it yet.



Oops. Sorry for not answering the question.

You can indeed accomplish the same thing using the String.split() method. You just have to use the version that places a limit on the number of splits -- split using a limit of two.

With this, you will only get one token, and the rest of the line as the other token. Later, you can use split again for the next token, using the same limit of two, and a different delimiter.

Henry
 
Wanderer
Posts: 18671
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Yeah, StringTokenizer isn't going away anytime soon - or ever, probably. Not as long as the language is still called Java, anyway. I agree with Henry - if it works for your problem, there's no reason not to use it.

Having said that, I do think that split(), Pattern / Matcher, and the Scanner class are generally much more powerful, flexible alternatives than StringTokenizer. In particular I recommend Scanner as the first and usually best choice for almost all text reading - as long as you're using JDK 5 or greater. There's really nothing you can do with StringTokenizer that you can't do with these other classes. And there are a number of things you can't do with StringTokenizer, that you can with split()/Pattern/Scanner. One is, StringTokenizer can really only handle one-character delimiters. Well, it can group multiple delimiters together as if they're one, if you want, but you can't tell it to look for a particular word as a delimiter. Another thing is that StringTokenizer can't easily handle the possibililty that two consecutive delimiters indicate a zero-length (empty) token. E.g. given this line:

foo,bar,,baz

If the ,, are intended to indicate that the third field (after foo and bar) is blank, well, StringTokenizer will be hard to use. Not impossible, but more trouble than it's worth, I think.

There are other things that StringTokenizer can't do, but those are two of the more common ones I've seen.

The downside of split()/Pattern/Scanner is that they all require you to know how to use regular expressions in order to use them effectively. This takes some time to learn if you haven't already encountered regexes elsewhere. So in the short term, it may be easier to just keep using StringTokenizer as long as you don't need the greater power and flexibility of regular expressions. But in the longer term, regular expressions are very useful things to know about, and they're used in many other languages besides Java. So time spent learning about them is time well spent, I think. I recommend www.regular-expressions.info and the Java tutorial on regular expressions as good places to learn about regexes, should you choose to do so.

Lastly, I should note that for the problem you describe, it may well be useful for you to learn about more complex and powerful tools like ANTLR or JavaCC. Might be worth spending a little time investigating these.

But again, if StringTokenizer works for your problem, that's fine. I'm just suggesting possible reasons you might want to try something else.
[ November 25, 2006: Message edited by: Jim Yingst ]
 
E McKenney
Greenhorn
Posts: 18
VI Editor Chrome
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
This is just the kind of input I was looking for, thanks!

b.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic