The Sun Certified Enterprise Architect for
Java EE Study Guide, is a good source for the diagramming. The one I have is by Cade and Sheil, 2nd Edition.
Regarding your specific questions, a Deployment Diagram will refer to specific machines (or machine spec/types). On those machines, would go things like .ear and .war files. The deployment diagram machines include firewalls and load balancers, and it will show protocols (like https).
Your component diagram will mention containers like
EJB container and Web container, and will have specific EJBs. It will have tiers laid out.
That should give you some kind of clues about what can go where. When I submitted my own diagrams, I included MDBs and JMS (at a moderate level) on the class diagram. That was not the only place I mentioned them. My disclaimer here: I did not get a perfect score, although I did quite well. So, I do not know where I lost points. You have to strike a balance between clutter and detail.