• Post Reply Bookmark Topic Watch Topic
  • New Topic

Blocking Threads

 
Surender Suri
Ranch Hand
Posts: 46
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Hello,

We are seeing blocking threads quite often in our application , not able to identify what is causing the issue .. any help is appreciated. Below is the stack trace of the thread that is starting the blocking. we can see the locking on ms sql server and killing the spid to bring back the application to normal ..

is it with the connection pooling ? or sql driver ? or is it with the java version .. since it's actually happening at the socket level ?



Additional Info:

java 1.6.0.33 build
tomcat 7
dbcp 1.2.1 - connection pool
MS SQL server

Thanks
 
Martin Vajsar
Sheriff
Posts: 3752
62
Chrome Netbeans IDE Oracle
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thread in Blocked state doesn't signify any problem in itself. It's just a thread which waits for some resource to become available or some event (eg. an IO operation) to happen.

The stacktrace shows that the blocked thread very probably waits for a response from the database. When you kill the database process, the database connection is lost and the waiting thread is notified about it somehow, resumes its operation and probably throws an exception (I guess). So, the problem isn't in the application itself, or in the connection pool, but in the database. You need to use database tools to find out what these processes in the database are waiting for, as this is the root cause. The solution will depend on that root cause.

(Note: does killing the SPID really bring the app back to normal? This should produce an exception, as noted above, and you should in turn see that some operation(s) in your app failed, which I wouldn't consider "normal".)
 
Chris Hurst
Ranch Hand
Posts: 443
3
C++ Eclipse IDE Java
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Your executing a prepared statement on a given db connection while the statement is executing on the database another thread wants to do a commit via what looks to be the same connection, your jdbc driver vendor has decided both operations can't happen at the same time on the same connection so one waits fro the other to complete (seems reasonable).

This seems to be more of an architecture \ database performance issue.
 
Martin Vajsar
Sheriff
Posts: 3752
62
Chrome Netbeans IDE Oracle
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Having a closer look at the stack trace, I see that Chris is probably right (so you'd better ignore what I've written above).
 
Surender Suri
Ranch Hand
Posts: 46
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Chris / Martin

Can you give some pointers .. where should I start looking ?

If two threads are sharing the same connection object, then isn't it some issue with the connection pooling ?

Thanks,
 
Martin Vajsar
Sheriff
Posts: 3752
62
Chrome Netbeans IDE Oracle
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
First of all, have a close look at your log files to check there aren't some warning messages there. They could be useful to narrow your search.

I don't think the problem is with the connection pool itself. When used properly, it shouldn't give the same connection to two or more threads. However, the dbcp version you're using is rather old. You might try to update to a newer version. Not because it would have a bug that was meanwhile fixed (albeit it is remotely possible), but mostly because a newer version could contain code to detect wrong usage and warn about it.

Are you using an ORM framework, such as Hibernate? I couldn't help a lot in that. If you are using just plain old JDBC, I can give some more tips as to what to look at.
 
Surender Suri
Ranch Hand
Posts: 46
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

We have been monitoring the logs for a while now and didn't had any luck with them so far. As far as upgrading the dbcp version, I need to prove that's what causing the issue before I can upgrade .. since it's a production. I guess I can try to simulate the issue with old and new versions in test env. if that can be simulated ..

We are not using any ORM .. it's standard JDBC.

Thanks,
 
Martin Vajsar
Sheriff
Posts: 3752
62
Chrome Netbeans IDE Oracle
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If the logs are silent and upgrading dbcp is out of the question (it probably wouldn't help anyway), you'll need to review your code.

You'll need to go over all places in your code where a database connection is obtained from connection pool, and inspect what is happening to the connection thereafter. Ideally, the connection should be used to run a business transaction on it, ending up with a commit or a rollback, and definitely should be closed when the processing is finished. The closing must happen in a finally block, and no command in the same finally block preceding the connection close should be able to throw an exception (Java 7 try-with-resources would be an ideal way to do this, since it ensures that close() is called in all possible cases). Any deviation from this scheme is suspicious and needs to be verified.

You would be especially looking for indications that the connection is somehow transferred to other threads. It could be done by storing it somewhere where it can be accessed by other threads, such as static variables or, say, a session or servlet context.
 
Surender Suri
Ranch Hand
Posts: 46
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Martin,

I have looked at the code and it seems to be fine. Since it's using a connection object from pool, it is returning properly in the finally block.

After debugging more, i think i may have found where exactly it is happening.. but still can't figure out why .. basically there two prepared statement updates happening with in a transaction and it seems to be blocking while commit on connection object is called.

Both the updates are happening on two different tables, but are linked by foreign key. Also, this part of the code is executed all the time ... so not sure what is happening.

any idea?



Another very very strange thing, admin guys reported was .. earlier they used to take only thread dumps.. but since last week i have asked them to take heap dumps as well .. so earlier when the locking is happening .. they had to manually kill the spid to bring back the application to normal .. now since they started taking the heap dumps .. when ever they take heap dump.. the locking seems to go away by itself
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!