Inventing stuff rarely works. I can't see what led you to believe that "
CheckBoxMenuItem[Selected].checkIcon" would be recognized by
JCheckBoxMenuItem as a relevant
UIDefaults key.
You could read the source of the private static class
javax.swing.plaf.metal.MetalIconFactory.CheckBoxMenuItemIcon to see how the Swing designers approached the issue.
As I see it, you have two primary options, with several variations possible:
1. Set the selected icon for each item in code, or create and use a custom subclass that does that in its constructor. You may need to experiment to determine which Icon of selected/pressed/rollover is used for the state in which you are interested.
2. Write an Icon implementation that wraps the two icons and decides which to paint based on the state of the component, queried after appropriate casting.
For your future reference, one way you can identify valid
UIManager /
UIDefaults keys is by using
Rob Camick's
UIManager Defaults.