What will happen if the BMT MDB throws a system exception from onMessage() method?
I guess nothing will happen excitingly other than your MDB gets killed.
As you mentioned "With BMT, the acknowledgement is always sent", this is because the default acknowedge-mode for BMT MDB is AUTO_ACKNOWLEDGE. Actually, it does not support other modes like CLIENT_ACKNOWLEDGE. So once the message has been delievered successfully, the acknowledgement is sent regardless of the transaction will be committed/rolled back and your MDB is still alive or dead.