Thats a good question.
My understanding is this:
When using a BMT MDB, the message receival is allready confirmed prior to invoking the onMessage method. The message receival runs in a separate transaction which is started/ended by the container prior invoking the onMessage method. Even if the onMessage method fails, the message is still acknowledged. You can specify the 'acknowledge-mode' in the DD to indicate weither the container should commit instantly (auto_acknowledge) or it may commit when it desires (dups_ok)
I think this is also the case with CMT. The container will start a transaction (just like with BMT) but this transaction gets propagated to you onMessage method (when the flag is required). This way your onMessage method runds in the same transaction as your message acknowledgement. Thus, if the transaction rollsback at the end, the message receival is not performed.
My guess is, that with an unspecified tx, the message receival transaction is suspended and the onMessage method is executed. If the onMessage throws an exception (which must be a system exception) it would seem logical to me that the message receival transaction rollsback.
-- edit --
The spec says this:
Message acknowledgment in an unspecified transaction context
is handled by the container. Section 17.6.5 describes some of the techniques that the container can
use for the implementation of a method invocation with an unspecified transaction context.
On 17.6.5 there is not much more information regarding this about MDB's. I think that the only correct answer to your question is: 'its not specified by the spec and thus container depended'
About the BMT message acknowledgement , I found also this in manning ejb2.0 (page 211 : <acknowledge-mode>

that when 'Auto-acknowledge' is specified, the message acknowledgement is made after SUCCESSFULL invocation of the onMessage method. So I think my understanding was wrong. With BMT the message is only acknowledged if the onMessage method returns successfully and this regardless of the <acknowledge-mode>. The latter will only determine when (directly, or after a random time specified by the container) a message is acknowledged if the onMessage returned without an error.
[ July 30, 2005: Message edited by: Koen Serneels ]