I would, however, prefer to use a TreeMap instead of an array, so I can just iterate through the objects that exist, rather than iterating through thousands of array cells waiting to stumble on the ones I'm actually using.
This bit made sense to me. In a real array you'd allocate 0..max slots in memory but only use a very few. You'd have to iterate every slot to find the used ones. You can key a map by the index, put in only the items you are using and iterate them very quickly with the keyset or entryset. Back in the early days of spreadsheets the "sparse array" that doesn't really allocate all the slots was a nifty new trick. It might well work for you here.
I would probably try both in a small experiment and compare times to see if the map is any faster. To try my hand at premature optimization
I might experiment with eliminating the new Integer() from all the accesses, too.
Let us know what you find!!