strange error causing JVM to hang..???

[asif kidwai]

Here is the exception...!!!

[4/12/04 23:18:34:706 EST] 12a3a6ed ExceptionUtil E CNTR0019E: Non-application exception occurred while processing method "PMInternalFinder_Local". Exception data: com.ibm.websphere.csi.CSITransactionRolledbackException: Global tx rolled back
at com.ibm.ejs.csi.TransactionControlImpl.getCurrentTransactionalUOW(TransactionControlImpl.java(Compiled Code))
at com.ibm.ejs.container.EJSContainer.getCurrentTx(EJSContainer.java(Compiled Code))
at com.ibm.ejs.container.EJSHome.activateBean_Common(EJSHome.java(Compiled Code))
at com.ibm.ejs.container.EJSHome.activateBean_Local(EJSHome.java(Compiled Code))
at com.wiredscholar.persistence.EJSCMPStudentHomeBean_b5050180.findStudentByParentKey_Local(EJSCMPStudentHomeBean_b5050180.java:175)
at com.wiredscholar.persistence.EJSLocalCMPStudentHome_b5050180.findStudentByParentKey_Local(EJSLocalCMPStudentHome_b5050180.java:241)
at com.wiredscholar.persistence.ConcreteParent_1d9edaf7.executeFinderForLink(ConcreteParent_1d9edaf7.java:165)
at com.ibm.ws.ejbpersistence.beanextensions.ConcreteBeanStatefulInstanceExtensionImpl.executeFinderForLink(Unknown Source)(Compiled Code)
at com.ibm.ws.ejbpersistence.associations.LinkImpl.executeFinder(Unknown Source)(Inlined Compiled Code)
at com.ibm.ws.ejbpersistence.associations.OneValuedLinkImpl.fetchTarget(Unknown Source)(Inlined Compiled Code)
at com.ibm.ws.ejbpersistence.associations.OneValuedLinkImpl.getValue(Unknown Source)(Compiled Code)

[Kyle Brown]

I can't imagine that this error would cause the entire JVM to hang. I can imagine that one or more threads may hang given this error since this is related to database locking if you have pessimistic locking turned on, however.
Are you SURE the entire JVM is hung? For instance can you access the admin console in that server while this is running? Can you try accessing a separate test servlet that doesn't call any database calls just to be sure?
Kyle

[asif kidwai]

we are using the default locking (weakest lock).
Yes the entire JVM hangs when this situation happens.

[asif kidwai]

I am not able to stop that server from admin console.Nor am i able to execute a servlet.

[asif kidwai]

Here is an excerpt from the exception today...
[4/16/04 11:44:04:148 EST] 3a4dab14 ExceptionUtil E CNTR0020E: Non-application exception occurred while processing method "getStudent" on bean "BeanId(WiredScholar#WiredScholarEJB.jar#Parent, e6ebff16-275e-11b2-0e30-0000ac13e246)". Exception data: javax.ejb.TransactionRolledbackLocalException: ; nested exception is: com.ibm.websphere.csi.CSITransactionRolledbackException: Global tx rolled back
com.ibm.websphere.csi.CSITransactionRolledbackException: Global tx rolled back
at com.ibm.ejs.csi.TransactionControlImpl.getCurrentTransactionalUOW(TransactionControlImpl.java(Compiled Code))
at com.ibm.ejs.container.EJSContainer.getCurrentTx(EJSContainer.java(Compiled Code))
at com.ibm.ejs.container.EJSContainer.getCurrentTx(EJSContainer.java(Inlined Compiled Code))
at com.ibm.ejs.container.EJSHome.activateBean_Common(EJSHome.java(Compiled Code))
at com.ibm.ejs.container.EJSHome.activateBean_Local(EJSHome.java(Compiled Code))
at
findStudentByParentKey_Local(EJSCMPStudentHomeBean_b5050180.java:175)
Any idea ? what;s going on. I contacted IBM , they asked us to upgrade to 5.0.2.4 (standard answer as always). We upgraded but we are still getting this exception with JVM hanging.

[Kyle Brown]

How long does it hang? Forever (until you kill the process) or for a few minutes? What do you get in the JVM dump with you use a kill -3?
Kyle

[Roland Barcia]

If you can constantly reproduce the problem, I would turn up tracing and select the CMP and database components in the Trace Menu.

[asif kidwai]

When the JVM hangs i have to pskill the process (since its on NT). I cannot even operate the admin console.
The problem is hard to reproduce as sometimes it happens 3-4 times in a day , some other time once in a week. I opened a case with IBM they say from the dump they see that the same daatabase connection handle being used by other transaction.
But my point is since we are using CMP's we rely on the container to take care of when to open connection/use/close connection etc.We are using JTA with CMP beans.
Asif

[Roland Barcia]

Even though the Container manages the connection, it is difficult to predict how you are going to call you CMP. Is your CMP being used in a transaction, is your CMP caching, what type of acess intent, is there pther dta acess (other CMP, Same CMP, JDBC code) in the same transaction, etc... Is the DB set to page level locking instead of row level locking, etc...
There are many reasons why deadlock can happen. No persistence framework can protect you from coding your way into a deadlock.
I am not saying that WAS may not be the cause, just do not eliminate that the way you access CMP code causing the lock.
Even though this link explains Enterprise Edition Features, it should help explain what Access Intent is. WAS Base supports Static Access Intent
http://www-106.ibm.com/developerworks/websphere/registered/tutorials/dl/sw742/htmls/BusinessProcessesinWebSphereProcessChoreographerConcepts/BusinessProcessesinWebSphereProcessChoreographerConcepts044.html
ALso, this link can explain connection sharing:
http://www-106.ibm.com/developerworks/websphere/library/techarticles/0404_tang/0404_tang.html
A mistake in these settings is where I have seen people cause deadlocks.
I hope this helps

[asif kidwai]

Even though the Container manages the connection, it is difficult to predict how you are going to call you CMP. Is your CMP being used in a transaction,
### I am using jta API.
is your CMP caching,
### If it is on by default, it should be what parameter should I look to see if it cached or not.
what type of acess intent,
### Repeatable read (default one for all beans)
is there pther dta acess (other CMP, Same CMP, JDBC code) in the same transaction,
### No this seems to be one call by itself. I even tried to remove the user transaction surrounding this method call but stil I get the same transactionRollbackException (com.ibm.websphere.csi.CSITransactionRolledbackException: Global tx rolled back) with entire JVM coming to halt.
I am wondering if there is no user transaction does the container create one of its own?

etc... Is the DB set to page level locking instead of row level locking,
### DB is at Page level locking.
etc...
There are many reasons why deadlock can happen. No persistence framework can protect you from coding your way into a deadlock.
### True but I am not sure if this is a deadlock problem or not.To me this seems to be a bug with Transaction API.At the most i would imageine the thread or question should timeout but halting a JVM is what worries me.
I am not saying that WAS may not be the cause, just do not eliminate that the way you access CMP code causing the lock.
### My query is a simple select statement. One thing I want to mention is that this query was taking extremely long due to improper indexing but now we changed the indexes we have to test it. Even so I would imagine the local transaction should just timeout instead of calling rollback on global tx.

[Roland Barcia]

You are using JTA UserTransaction object from the calling Session Bean? Or are you using Container Managed Transactions by demarcating your calling Session Bean.
By default there is no CMP caching

##Repeatable read?
Are you using CMP 2.0 or 1.1. For CMP, access intent has different terminology. The default for CMP 2.0 is PessimisticUpdateWeakestLockAtLoad. Meaning it grabs a read lock at the start of a transaction and escalates when an update is done.
Have you turned on tracing the DB just to rule out the DB?

[asif kidwai]

You are using JTA UserTransaction object from the calling Session Bean?
### We are not using Session bean at all. The servlet communicates with a regular java bean which talks to CMP 2.0 in a user transaction.

Or are you using Container Managed Transactions by demarcating your calling Session Bean.
By default there is no CMP caching
### We haven't done anything to cache the CMPs.This is another question I had, if the CMP beans methods are marked with a "Required" transaction , the client to the CMP creates a user transaction and calls the methods on CMP 2.0, will the container still generate its own transaction as well. Or it will use the transaction of the client.And also can the database (or DB2 connect) force a connection to close? Is there a setting in DB2 connect that can interfere with connections. The problem appears to be that the container tries to call resultset.getNext()on a connection that appears to be already closed.The JVM fails with a) "DB2CLI : Function sequence Error when doing resultset.getNext()" b) Transaction timed out after 120 seconds c) TransactionRollbackException : Failed to rollback global tx etc. and hangs the JVM.
Have you turned on tracing the DB just to rule out the DB?
### Yes we did but so far nothing concrete is showing up in jdbc logs. The only thing we found so far is the query involved in the transaction was taking long time (about 3 minutes) and the transaction timeout is set to default (2 minutes). But can a long running transaction halt the JVM , it may/could halt the thread in question.Though we have reindexed the table now so we think the problem will not show up again but it was surprising that a long running query can bring Appserver to halt.
[ April 25, 2004: Message edited by: asif kidwai ]

[Kyle Brown]

Hoo boy. I'm not even sure where to start on this one. First of all, why aren't you using a Session bean as a facade? Using one would seem to make things much simpler, and eliminate a host of possible problems. I'd suggest placing a session facade around your CMP, using CMT on the session bean, and then seeing if your problems don't just disappear.
Kyle

[asif kidwai]

Kyle,
I understand the session facade would have been the best option but our application is a little different. Unfortunately we were the victims of IBM's OR mapping tool called Persistence builder which has ended its life last year end. So we were forced to move to CMP 2.0 in place of PB. The PB architecture did not had session facade pattern and we didn't wanted to do rearchitect/redesign the application as that would have required lot of retrofit in the client calls.

[Roland Barcia]

So right now you have Servlet->JavaBean->CMP from following your threads? How difficult would it be to put an empty stateless session bean layer like this: Servlet->StatelessSessionBean->JavaBean->CMP. If your CMP's are REQUIRED and you set your Stateless Session Bean to REQUIRED_NEW, then your CMP will be under the transaction started when invoking the Session BEan. The layer inbetween does not matter since the TX is associated with the thread.

[asif kidwai]

So you mean to say, remove all the TX from the JavaBean and wrap it with Session bean with REQUIRED_NEW.
OR
Leave the javabean with all the user transactional code as it is and have a stateless session bean wrapper with REQUIRED_NEW around the java bean. In such a case how would the container react to transaction being in 2 places when it has to commit.or will the container only see the transaction of the session bean and ignore any other that follow down its path?

[Roland Barcia]

I would say option one is your best bet. Then the Java Bean data access will be under that of the Stateless Session Transaction