I see your doubt.
The short answer is that if there is no match, then the whole input is returned as the only token. However, this has not been explicitly stated in the API. Why? Read further...
Now, the long answer:
The problem is in understanding the semantics of the
word "delimiter". There is a difference in the usage of a
pattern as a "delimiter" and as a "search string".
Consider the
string "a b". How many tokens will the default scanner return? Or in other words, how many times the loop (of the above program) should be entered?
The scanner starts building (in its buffer) a token right from the first char of the input. It keeps looking (and appending the inspected char to the token) until it finds the delimiter. As soon as it finds the delimiter, it returns whatever token had built up, empties its buffer, and starts looking again. At this point, the input ends. So it simply returns whatever it had in the token buffer, which is the rest of the string after the last occurance of the delimiter.
Now, how many times the string " " occurs in the given string? Only once. So if you use the Pattern/Matcher classes, you will get only one match.
HTH,
Paul.