As far as applets go, usually they do software 3D rendering using just the draw functions in AWT. This is for multiple reasons. First, there is the problem with including a third party library with your applet. That means if you used something like Java3D, JOGL, LWGL, etc. to provide hardware accelerated 3D you would have to have the user download that library with your applet every time.
Or have the user do some special setup on their side to make sure the 3D library was on the classpath and that the JVM in their browser made use of that classpath. Also, I'm not completely certain that this is the case, but since a hardware accelerated 3D library would have to make use of system calls, I would also think that an applet making use of one of these would need to be signed.
If you're not stuck on the idea of making this an applet, most of these problems go away... you could always make it an application and use
Java Web Start to provide a way to download and launch it.
As far as doing software rendering, I have several older books that cover this topic. Two of the better ones were "KickAss Java" by Tonny Espeset (which shows how to do all kinds of cool image filtering effects, 3D effects, spinning menus, etc. and it is focused on applets) and "Graphics Programming with JFC" by David Wall and Arthur Griffith (which is a misleading title - really all it covers is Graphics2D, and all the examples use AWT - but there is a section that shows how to do software rendered 3D ). From what I remember, "Black Art of Java Game Programming" had some good coverage of 2D game programming, but the software rendered 3D section wasn't that good... it made use of one of the author's libraries supplied with the book, and didn't really tell you much about how to do it yourself. I'm not sure if any are in print any longer, but you might be able to pick them up used from somewhere.