How to debug a thread in production environment?

[Rahul Toaikani]

if a thread has some performance issue,slows down the system for some reason or appears in deadlock situation, how do you debug? (other than looking at the logs).

How do you do this in production environment and in developer environment?

Appreciate your input.

[Stan James]

This probably varies by JVM and OS and server, but we were able to get thread dumps - snapshots of what's happening right now - which gave good information about what method every thread is in. We found a huge number of threads in a single method which helped us find something that ran too long. Then again, many threads were in the logger just because it is called so often.

Later we used Wily Introscope which is quite expensive but gives moutains of information about all aspects of performance. I think Mercury has something comparable.

[Chris Hurst]

JMX is worth a look if your on a JVM 1.5 or above, for deadlock then thread dumps (as previously stated) will quickly give the game away how you get them depends on your OS i.e. unix signal, windows key combination etc etc i.e. if you give us an exact idea of what OS, JVM your intending to deploy with, we can help more :-)

[Rahul Toaikani]

Thanks Stan and James for the response.

OS: Windows 2000 server.
I found good artcile about stacktrace from SUn.
http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/

But how about performance? Now I understand using dumps, you can see what is current thread and what method has lock etc., but are there any ways to figure which thread takes up more space and time etc., will the regular profiling tools help?

[M Easter]

Using formal profiling tools is undoubtedly the best practice, but using the log files to your advantage can be useful, if you have the opportunity/time to modify the source.

I have seen effective use of a timer/stopwatch at certain boundaries of a system (e.g. the DB layer, or the client layer to RMI/Corba calls). IMHO, these should be "baked-in" right from the start of development. The stopwatch should be configurable: on/off, a threshold for which "fast" events are not logged, etc. The idea here is not to determine which _thread_ is slow but rather to find out which _activity_ is slow.

FWIW, I've written a modest blog entry on this:

http://codetojoy.blogspot.com/2007/02/stopwatch-idiom.html

[steve souza]

I have seen effective use of a timer/stopwatch at certain boundaries of a system

JAMon does the things you request and more.

It works like a stopwatch that you pass a String to. This String represents what you want to time (a page name, method, sql, etc). Whenever you pass jamon the same String aggregate statistics are gathered. i.e. You will have collected the following statistics (in ms.): hits, avg, total, min, max, std dev, when last invoked, and whether or not the code is currently running in addition to some other useful stats. The data is viewable via the jamon.war or programmatically.

It is easy to use, fast, and open source.

 
import com.jamonapi.*;
 
// Note the String can represent anything such as a method name, page name, sql statement
Monitor mon=MonitorFactory.start("myCodeBeingTimed");
...code to time...
mon.stop();

You can also monitor your database inteteractions and web pages with NO code changes using JAMon. You can use the jamon jdbc driver to gather all statistics for queries and jdbc calls, and you can use the jamon servlet filter to gather statistics for all pages in your web app. Both only take a couple minutes to set up.

See the links below for more info including a live demo...