javax.transaction.TransactionRolledbackException : Transaction is ended due to timeout

[Aman Jain]

My application runs on Websphere and is used in multilanguages and is written using EJB 3.0 specification. I am getting the exception as in the subject of this topic. This exception has recently started coming into the logs. My application is in production for over an year now. In the beginning, this exception never came out but now this exception is coming frequently.

My application is basically a report processing system, which reveives reports from various sources and then they are processed in batches. Can anybody help me out in figuring out the root cause of this. Considering that since the application is in production for over an year so the database entries would have increased to a great extent and that this exception never came out until since last one month.

Following is the stack trace of the exception -

javax.ejb.EJBTransactionRolledbackException: Transaction rolled back; nested exception is: javax.transaction.TransactionRolledbackException: Transaction is e
nded due to timeout
Caused by: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout
at com.ibm.ws.tx.jta.TranManagerImpl.completeTxTimeout(TranManagerImpl.java:460)
at com.ibm.ws.tx.jta.TranManagerSet.completeTxTimeout(TranManagerSet.java:632)
at com.ibm.ejs.csi.TransactionControlImpl.completeTxTimeout(TransactionControlImpl.java:1359)
at com.ibm.ejs.csi.TranStrategy.postInvoke(TranStrategy.java:236)
at com.ibm.ejs.csi.TransactionControlImpl.postInvoke(TransactionControlImpl.java:561)
at com.ibm.ejs.container.EJSContainer.postInvoke(EJSContainer.java:4558)
at com.cititech.delphixp.re.report.custom.EJSLocal0SLCustomReportSystem_ef3b1a48.processBatch(EJSLocal0SLCustomReportSystem_ef3b1a48.java)
at com.cititech.delphixp.re.framework.service.ReportSystemMessageBean.onMessage(ReportSystemMessageBean.java:168)
at com.ibm.ejs.container.MessageEndpointHandler.invokeMdbMethod(MessageEndpointHandler.java:1092)
at com.ibm.ejs.container.MessageEndpointHandler.invoke(MessageEndpointHandler.java:777)
at $Proxy32.onMessage(Unknown Source)
at com.ibm.ws.sib.api.jmsra.impl.JmsJcaEndpointInvokerImpl.invokeEndpoint(JmsJcaEndpointInvokerImpl.java:192)
at com.ibm.ws.sib.ra.inbound.impl.SibRaDispatcher.dispatch(SibRaDispatcher.java:888)
at com.ibm.ws.sib.ra.inbound.impl.SibRaSingleProcessListener$SibRaWork.run(SibRaSingleProcessListener.java:528)
at com.ibm.ejs.j2c.work.WorkProxy.run(WorkProxy.java:399)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550)

[Aman Jain]

Can somebody give any pointers to my issue?

[Aman Jain]

Any pointers to this issue would be helpful to me. Can someone reply to it?

[Paul Sturrock]

The exception says it all really - you have an EJB method that cannot complete before a transaction timeout occurs. You need to look at what the MDB ReportSystemMessageBean is up to - is the batch of data it is dealling with realistic? It may be all you have to do is extend the length of the transaction, or it maybe you need to review what the processBatch method actually does so it processes smaller batches; will its batch grow forever? Or is there an expected maximum amount of data?

[Aman Jain]

Loads of files come for processing whole of the day. Before the application processes the files, it first creates batches of the files say - if number of received files are some 250 - 300,then the application would create multiple batches for these files based on a pre-defined batch configuration. Batch configuration is retrieved from the database which has a fixed batch size defined for a particular type of file.

One of my concern here is, this exception was never reported earlier. It started throwing this exception only recently (since last week only). The batch configuration hasn't changed since last one year and since last one year the traffic of files coming in for processing has remained constant. Even the batch size remains constant since it is retrieved from database and that the batch configuration has not changed in the last 1 year. So, what could be reason for this exception coming now after more than an year. Any pointers on this specific concern of mine?

Things that I tried to figure out the root cause - Analysed all the SQL queries that are executed during the processing of files. SQLs are very straight forward - simple insert and update queries. No complex queries. Tried executing those queries directly onto the database using SQLDeveloper. They worked fine.

Looking at the exception and logs of the application - one thing that I noticed is that some database transaction is getting stucked somehow because of which after sometime a Transaction time out exception is thrown. Any pointers with the above explanation on this exception?

[Aman Jain]

Any further pointers based on the problem description I posted last in this topic?

[Jaikiran Pai]

From that exception stacktrace, there's not much that can be guessed other than the obvious timeout issue.

Aman Jain wrote: one thing that I noticed is that some database transaction is getting stucked somehow because of which after sometime a Transaction time out exception is thrown.

You'll have to narrow down to that query which is taking the time. Have you checked with the database administrator to see what activity is going on, on the DB server, when you see this behaviour?

[Pierre-Hugues Charbonneau]

Hi Aman,

In order to troubleshoot these kind of problems, you will need to generate a few JVM Thread Dump during your business process / processing.

Following and analysing each snapshot will allow you to pinpoint the bottleneck and root cause so you can implement corrective measures.

What are your environment specifications? Thread Dump generation technique will depend of your JDK vendor and version.

Once you get a few snapshots, have a look at the Thread executing your EJB method : processBatch and post the Thread snippet to this Thread.

[Aman Jain]

sure thanks a lot for your guidance i will monitor n will post the update

[Pierre-Hugues Charbonneau]

Thanks Aman,

Happy to help. Please just let me know if you need help with the Thread Dump analysis.

Regards.

[Aman Jain]

Thanks Pierre
Unfortunately I cannot get the thread dumps as the issue occurs only in the production environment and we are unable to recreate the same issue in the UAT environment. Still I will post the logs tomorrow that I could manage to get from the production environment.

[Pierre-Hugues Charbonneau]

Hi Aman,

Any problem preventing you to gather/generate Thread dump from your production environment?

Regards.

[Aman Jain]

Not problems but permissions issues. Cannot get the permissions to do anything on the production environment. All that I can get are the logs that are generated by my application which I will post tomorrow here.