With a HashSet you would be limited to Integer.MAX_INT entries (due to the backing array) and the performance would depend on the size - too large would be a memory hog, too small would have too many collisions.
As stated, BitSet could be useful if you want to move into extraordinarily large numbers.
I once wrote a bitwise operation that was an array of ints where each bit represented a boolean state for a couple of billion numbers.
It has good performance and is easy to code, and it's almost certainly the best solution if your set of numbers is reasonably sparse (i.e., below about 10 million or so.) If you actually expect more than that, then the other solution is to actually make an array with 4 billion bits (one bit for every integer, say 6.25M longs) and then check/set the presence of an integer by looking at the corresponding bit. This is more complicated code (it could be hidden inside some functions, of course) but it uses just one huge 500M block of memory.
As DOM points out, BitSet implements this for you.
and BitSet is still 'limited' to 2^31 bits, but if you feel like writing your own bitwise operation based on an array of int or longs you can get to 2^63 ie more than you have available memory, fingers and toes included.