I don't have deep knowledge on the implementation of hashing algorithms, but I know that using prime numbers minimizes collisions. See Wiki for a short explanation.
It is more likely to produce slight differences in the right-hand bits of the number. Since HashMap etc use something analogous to hash % 16, hash % 32 etc, the lower-order bits are where differences are beneficial.