HeuristicMixedException for XA transaction between JBoss 5.1.0GA and 3.2.0

[Michael Wiezik]

Hi All,

I am trying to migrate from JBoss 3.2.0 to 5.1.0GA. The application running on the server is invoking remotely EJBs deployed on yet another jboss 3.2.0. When I start JBoss 5.1.0.GA, I can see the following logs:

javax.transaction.HeuristicMixedException
javax.transaction.HeuristicMixedException
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commit(TransactionImple.java:254)
at org.jboss.ejb.plugins.TxInterceptorCMT.endTransaction(TxInterceptorCMT.java:501)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:361)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:267)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)
at org.jboss.ejb.plugins.security.PreSecurityInterceptor.process(PreSecurityInterceptor.java:134)
at org.jboss.ejb.plugins.security.PreSecurityInterceptor.invoke(PreSecurityInterceptor.java:81)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)
at org.jboss.ejb.Container.invoke(Container.java:1029)
at sun.reflect.GeneratedMethodAccessor356.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
...

13:42:41,888 WARN [loggerI18N] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] Could not find new XAResource to use for recoverin
g non-serializable XAResource < 131075, 26, 24, 49459748495454985158101574958529851985651100485850575797484954549851581015749585298519856511004858509750 >


Any clues will be appreciated. Thanks!

[Michael Wiezik]

It seems that the implementation of the jboss transaction library (called Arjuna) has been changed between jboss 3.2.0 and 5.1.0.GA:

- in Arjuna 4.2.3.SP5 (used by JBoss 3.2.0), when distributed resource reports heuristic commit, the transaction is not rolled back.
- in Arjuna 4.6.1.GA (used by JBoss 5.1.0.GA), when distributed resource reports heuristic commit, the HEURISTIC_HAZARD is being reported, which causes transaction rollback

The actual change is within the com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelPrepare() method, catch(XAException) block:

old implementation (from line 283):

			/*
			 * XA_RB*, XAER_RMERR, XAER_RMFAIL, XAER_NOTA, XAER_INVAL, or
			 * XAER_PROTO.
			 */

			if (_rollbackOptimization) // won't have rollback called on it
				removeConnection();

			if ((e1.errorCode == XAException.XAER_RMERR) || (e1.errorCode == XAException.XAER_RMFAIL))
				return TwoPhaseOutcome.HEURISTIC_HAZARD;
			
			return TwoPhaseOutcome.PREPARE_NOTOK;

new implementation (from line 289):

			/*
			 * XA_RB*, XAER_RMERR, XAER_RMFAIL, XAER_NOTA, XAER_INVAL, or
			 * XAER_PROTO.
			 */

			if (_rollbackOptimization) // won't have rollback called on it
				removeConnection();

			switch (e1.errorCode)
			{
			case XAException.XAER_RMERR:
			case XAException.XAER_RMFAIL:
			case XAException.XA_RBROLLBACK:
			case XAException.XA_RBEND:
			case XAException.XA_RBCOMMFAIL:
			case XAException.XA_RBDEADLOCK:
			case XAException.XA_RBINTEGRITY:
			case XAException.XA_RBOTHER:
			case XAException.XA_RBPROTO:
			case XAException.XA_RBTIMEOUT:
			case XAException.XAER_INVAL:
			case XAException.XAER_PROTO:
			case XAException.XAER_NOTA: // resource may have arbitrarily rolled back (shouldn't, but ...)
				return TwoPhaseOutcome.PREPARE_NOTOK;  // will not call rollback
			default:
				return TwoPhaseOutcome.HEURISTIC_HAZARD; // we're not really sure (shouldn't get here though).

Having said that, I don't understand why this change has been implemented and what is the workaround for the issue?

[Jaikiran Pai]

Michal Wiezik wrote:

Having said that, I don't understand why this change has been implemented and what is the workaround for the issue?

The JBoss Transactions forum is a better place to ask this.

[Michael Wiezik]

I have posted this in jboss forum as well, but got frustrated by their web server response time. I'm guessing there is a New Year overload at the mo...

Anyway, I am posting the solution here in case anybody else comes across this issue:

The problem was with our custom XAResource. It reports a heuristic commit in a prepare() method. There two ways to do that from XAResource: either

    throw new XAException(XAException.XA_HEURCOM); 

or

  return TwoPhaseOutcome.HEURISTIC_COMMIT; 

. Both of them are acceptable by the older version of Arjuna and only the latter one does not cause transaction rollback in the newer Arjuna implementation.

[Jaikiran Pai]

Michal Wiezik wrote:I have posted this in jboss forum as well, but got frustrated by their web server response time. I'm guessing there is a New Year overload at the mo...

Right, there have been issues with the forum in the past week. From what i hear, they have been sorted out today. At the moment they seem to be fine and hopefully will stay so.

[Jeremy Whiting]

Hello,
I agree try the JBoss Transactions User forum as recommended by Jaikiran.

Also, have a look at the JIRA system for this class. Here is a link showing the change history for XAResourceRecord and the defects these changes are associated with.

https://jira.jboss.org/jira/secure/IssueNavigator.jspa?reset=true&&query=XAResourceRecord&summary=true&description=true&body=true


Usually with heuristic decisions you will need to take manual intervention to resolve the decision. Your database vendor should provide tooling to view and resolve them. Log files will provide additional info.

The ArjunaCore recovery manager will keep a record associated with the problem. To removed expired entries follow these instructions.

http://www.jboss.org/jbosstm/docs/4.2.3/manuals/html/jts/ProgrammersGuide.html#_Toc163466706

Regards,
Jeremy

[Michael Wiezik]

Hi Jeremy,

thanks for the response - the information you posted is very useful. However, as I mentioned above, the root cause was our own custom implementation of XAResource, which was not compatibile with the newer version of Arjuna.