Win a copy of The Java Performance Companion this week in the Performance forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Troubleshooting High CPU - Request Threads always in service

 
Asad Imtiaz
Greenhorn
Posts: 21
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello everyone,

In our production envoirnment we have a situation where our application deployed on the Tomcat Server starts to utilize 100% CPU. We were not successful in pin pointing the code or area where its happening. Unfortunately we are not able to exactly reproduce it either in our developement envoirnment. We can see in the status page of the "Tomcat Manager" application that some request threads are constantly in S (Service) status which in turn consumes CPU. Each thread eats up 25% of the total CPU (the server has 4 cores) so if we have 4 of such requests then the server reach 100% CPU and its unusable. Only solution at the moment is a tomcat restart. I am open to any suggestion but at the moment I cannot upgrade by JDK version. May be I can move to a slightly higher tomcat version if my current version is buggy.

What I really want to know is how can I can troubleshoot these high CPU consuming threads so that I can fix my code. Here is a screen shot from "Tomcat Manager" application




Production Environment

Tomcat v5.5.17
JVM 1.4.2_06-b03
Windows Server 2003

I am open to any suggestion but at the moment I cannot upgrade by JDK version. May be I can move to a slightly higher tomcat version if my current version is buggy.

If there is any detail that I have missed I would be glad to provide it. Any pointers would be really helpful.


Thanks you !

 
Tim Holloway
Saloon Keeper
Pie
Posts: 18226
53
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
From what little I can see, you have an application that is running its CPU at 100%. A thread can (by definition) run on 1 and only 1 CPU at a time, so it's not really "25%" CPU usage, it's 25% system usage with all 4 cores at 100%.

Tomcat is not a system that consumes a lot of resources, so we can pretty safely say that whatever's going haywire is in a webapp. There are tools that can drill down into it, and it's time to learn about them. For fastest results, bring money. I've gotten by on the freebie stuff, but my "frugal" employers were actually just trading large heaps of my time for the cost of something that would speed up the analysis, so the money they saved was all in the bean-counters' heads.
 
Asad Imtiaz
Greenhorn
Posts: 21
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tim Holloway wrote:From what little I can see, you have an application that is running its CPU at 100%. A thread can (by definition) run on 1 and only 1 CPU at a time, so it's not really "25%" CPU usage, it's 25% system usage with all 4 cores at 100%.

Tomcat is not a system that consumes a lot of resources, so we can pretty safely say that whatever's going haywire is in a webapp. There are tools that can drill down into it, and it's time to learn about them. For fastest results, bring money. I've gotten by on the freebie stuff, but my "frugal" employers were actually just trading large heaps of my time for the cost of something that would speed up the analysis, so the money they saved was all in the bean-counters' heads.



I totally agree with you and I know its our web application that is eating up resources. I need a direction. There have been looking at tools (free and paid) that would help but what I need is responses from experienced members here who can save me time and a bad decision of selecting a tool/approach. Although I really appreciate your response.

I can slightly experiment with one of our production servers and thinking of using jconsole to see if that give me some details on which threads (class) is actually in the service mode. Once I have some clue of which code to look for I will find my way.


 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13064
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You might look into JAMON - I understand it is easy to get started with.

Bill
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic