I was thinking about this, and I believe the situation is a bit murkier than I first indicated. When bean providers combine bean services via remote interfaces, they have to be careful not to over-interpret the information value of receiving a RemoteException, because you won't always know *why* it was received. It doesn't matter so much when the beans share a transaction, but you've got to be careful when the beans don't share transactions - the client bean probably shouldn't be written in a way that it manages its transactionality to depend upon receiving or not receiving RemoteException.
Let's assume you have two beans, B1 and B2, with B1 using services of B2 via a remote interface. We'll ignore EJB 1.1 functionality (in other words, assume that the method implementation code for B2 doesn't deliberately throw a RemoteException). There are 4 different cases to consider if B1 and B2 are deployed in different containers (cases 2 and 4 below are rather unlikely when B1 and B2 are in the same container). Let's call B1's container C1, and B2's container C2. TS will indicate a transaction shared by both B1 and B2 (i.e. B2 method is marked 'Required' or 'Supports' or 'Mandatory'), and when B1 and B2 run in separate transactions I'll refer to T1 and T2 (i.e. B2 method is marked 'RequiresNew').
1: B1 and B2 share transaction, B2 throws system exception
C2 will do a rollback on TS and then C2 throws a RemoteException to B1;
since B1 and B2 share TS, B1 doesn't need to do anything to force a
rollback, and can't do anything to stop the rollback.
2: B1 and B2 share transaction, B2 is ok but communications break down
B2's stub's RMI code (being used by B1's method) will throw a
RemoteException, and C1 doesn't do anything automatically until
the time would come for it to do the commit TS on completion of
the B1 method - which obviously will fail if the communications
problem is still in effect, so as far as C1 is concerned TS will
roll back. C2 will never attempt to participate in a commit on TS
because it'll never get the
word from C1 that it should.
3: B1 and B2 in different transactions, B2 throws system exception
C2 will do a rollback on T2 and then throw a RemoteException to B1.
It is up to B1 to decide if it wants to do a rollback or not;
C1 isn't going to do anything automatically to T1. B1 is in
control of T1's destiny.
4: B1 and B2 in different transactions, communication path to B2 fails
B2's stub's RMI code will throw a RemoteException. B2 might or might
not have committed - depends on whether the method ran to completion
or not. In general B1 won't know, and if the transactions were set
up this way B1 probably isn't supposed to care. B1 will have received
the RemoteException, but since it doesn't share a transaction with B2,
C1 won't care about the exception. B1 is in control of T1's destiny.
So, note that in for cases 1 and 2, the bean provider doesn't have to do anything to manage the transaction - CMT does it all. In cases 3 and 4, the bean provider really has no way of knowing in B1 what happened to B2, so managing the transaction via receiving RemoteException is a bad idea - if B1 really needs to care about what goes on in B2, then B2 should have thrown an application exception instead.
And yes, it is ok to invoke setRollbackOnly when the transaction is already marked for rollback by the container, but you do have to have a transaction or you'll get an
IllegalStateException (spec section 17.6.2.9, pg 361). Also note that if you are using CMT, its hard to come up with a good reason why
you should ever have to mark the transaction manually just because of system or RemoteExceptions. As the spec says (section 17.3.4.2 pg 349):
Typically, an enterprise bean marks a transaction for rollback to protect data integrity before throwing an application exception, because application exceptions do not automatically cause the Container to rollback the transaction.
If you don't have reason to throw an application exception, instead of setRollbackOnly you should probably throw an EJBException.