As DT says, tokenizer has not had a usage for fifteen years.
You should not worry about which is faster at this stage. Neither is suitable for use with a file because a file is not in the form of a
String. Only when you read a file do you get Strings and the file reading classes all produce multiple Strings representing lines. Some means of reading from text files (e.g.
Scanner can translate the text directly into other datatypes (e.g.
ints), which would make splitting the String unnecessary.