I would do the "listening" on the JPanel, not the code launched by the Timer.
I would restrict code executed by the Timer to a single command: to call an update() method on the JPanel. But as I look closer, it appears that the Timer really isn't contributing anything to the animation. The way you have things set up, animation is the result of key inputs (each calling a repaint).
More normal use of the Timer is to use util.Timer, calling anywhere from 20 to 60 times per second. In this scenario, the velocity of the objects are stored in variables and used to update the x,y location of objects every animation cycle. The key input (listeners) would be used to alter the velocity values, and these would occur outside of the update/repaint cycle.
Hmm. A lot to explain--maybe check out some game-loop tutorials?