• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Almost all threads are in "P" status

 
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

I'm encountering a problem:
1. almost all threads are in "P" status, and occasionally there is 1 or 2 in "S" or "R" status,
2. and there is no slow query on MySQL side, because I can't catch any process with command "show full processlist",
3. while tomcat connection rate is merely less than 30req/s,
4. and tomcat CAN response correct result page in acceptable time(1 second)
System architecture: Tomcat*3 + Memcached*2 + MySQL-Proxy*2 + MySQL*2, Linux-Redhat
Software information: apache-tomcat-7.0.6, MySQL--5.5.11-1.rhel5.x86_64, jdk-6u23-linux-x64-rpm
Configuration:
tomcat server.xml:

which provides default maximum number of threads is 200

Screenshot: please refer to attachment.

Question:
1. is this situation(full of "P" status) problematic?
2. how to explain the situation?(waiting for Http request body? sounds unreasonable!)
Screenshot.png
[Thumbnail for Screenshot.png]
 
Sheriff
Posts: 22783
131
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If you check just below the table, you'll see this:

P: Parse and prepare request S: Service F: Finishing R: Ready K: Keepalive


You can read what these mean here.
 
Mei Zhiguan
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Rob Spoor wrote:If you check just below the table, you'll see this:

P: Parse and prepare request S: Service F: Finishing R: Ready K: Keepalive


You can read what these mean here.



thank you for your quick reply.
I did read related information, including official documentation, but I just can't imagine why all of the threads are in the status of "Parse and prepare request", after all, there are merely <30 requests per second, and the service is quite responsive.

Please forget the "Max processing time: 9024 ms", because when it was around 1000ms, the situation is the same.

Anybody have ever met similar problem.
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
How big are these requests anyway? How are they formatted?

Bill
 
Mei Zhiguan
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

William Brogden wrote:How big are these requests anyway? How are they formatted?

Bill



Request size:
I can roughly calculate the average size of requests by dividing Inbound throughput by request count, and the data is from Cacti
Average Request / Second: 84.85
Average Inbound throughput(bits/s): 135.93k(bits/s)
Average Request Size = 135.93 / 84.85 = 1.6k(bit)
= 1.6 * 1024 / 8 = 208(Byte/s)

Request format:

1. Every request is formatted with GET parameters and POST parameters, and below is a typical access log containing URL, which reflects GET parameters:
119.146.56.30 - - [01/Feb/2012:23:59:59 +0800] "POST /RecommendationService/recommend?user=07520560066&ppd=Unnd:PROG/565532@IKBTV.BMM.BMM&Page=page_stop_confirm HTTP/1.1" 200 907

2. POST parameter consists of:
method=seriesbasedrecommend
user=07580420366
ppd=Unnd:PROG/565532@IKBTV.BMM.BMM
spd=Unnd:PROG/565532@IKBTV.BMM.BMM
type=webdoc
tag=tech
pviewhistory=Unnd:PROG/565532@IKBTV.BMM (NOTE: at most 10, seperated by ",")
sviewhistory=Unnd:PROG/565532@IKBTV.BMM (NOTE: at most 10, seperated by ",")
fcount=10
wcount=5
charset=UTF-8

Any further information needed?
connectionrate.png
[Thumbnail for connectionrate.png]
throughput.png
[Thumbnail for throughput.png]
 
Mei Zhiguan
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Hey, am I asking a wrong question at a wrong place? Isn't this a Forum on Tomcat?
I don't think this problem deserves no attention.
 
William Brogden
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The reason you are not hearing from me is because I am completely mystified by your problem. I can think of no reason that those simple requests could be keeping those requests occupied in the P state. I monitor my own server with the management app and I don't think I have ever hit it where a Thread was in the P state.

Bill
 
Rob Spoor
Sheriff
Posts: 22783
131
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Bill's not the only one who's mystified. I don't think the reason you're not getting any answers is because nobody wants to answer your question, I think it is because nobody can answer it.
 
Saloon Keeper
Posts: 27762
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
And the reason you're not hearing from me is that I only look at this kind of stuff when there's a problem, so I don't keep up-to-date on what it all means. As far as I could tell, you're not actually experiencing performance or reliability issues, so I didn't worry about it.

I mean, for a small phenomenal fee, I could dig into your server and do a serious analysis and give you an answer, but in my experience, companies only pay phenomenal fees to people who come in and make trendy noises ("Cloud, Cloud, Cloud"), not to solve things that are only mildly annoying.

There's no conspiracy of silence here. When you don't get an answer on JavaRanch, it means we probably don't know. Since we're not being paid, we have no incentive to sound wiser than we are other than simply for ego inflation.
 
Mei Zhiguan
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Thank you for your replies, and I will post the answer when I get it!
 
Mei Zhiguan
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Problem RESOLVED !!!

Solution:
1. minimize keepAliveTimeout from default 20000 to 3000
2. maximize maxThreads from default 200 to 2000

Effect:
Most of threads come back to "R" or "S" status, and tomcat responses quite quickly. Be precisely:
1. busy threads of tomcat declined at least 60%;
2. respond time decreased at least 80%;
3. throughput increased around 10%;
4. heap size decreased around 10%.

Analysis:
Tomcat default configuration (maxThreads=200, keepAliveTimeout=3000) is for normal web application, which means that
1. 90% of requests are GET, the remains are POST and others;
2. 1 primary GET triggered by user is followed by a bunch of GET triggered by browser.
However, my application is:
1. 100% of requests are POST;
2. every POST requests are triggered by another server without any consequent request;
This determines that the 3000ms of keep-alive is nothing but a waste of time.

That's it. Not sure whether it is correct. Any comments?

 
Tim Holloway
Saloon Keeper
Posts: 27762
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator


2. every POST requests are triggered by server without any consequent request;



That's not right. Whether it's a GET or a POST, if it's HTTP, servers cannot "trigger" unsolicited responses. Some client somewhere had to issue that POST as an HTTP request.

The primary reason why POSTs are offending and not GETs is likely that you are carrying a lot of data in the POST request body. GETs tend to be self-limiting. The larger the request, the longer it's going to take to pipe it in.

There's also a hint that perhaps the application's per-request processing time is a bit long, but that would be have to be measured.

 
Mei Zhiguan
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:


2. every POST requests are triggered by server without any consequent request;



That's not right. Whether it's a GET or a POST, if it's HTTP, servers cannot "trigger" unsolicited responses. Some client somewhere had to issue that POST as an HTTP request.

The primary reason why POSTs are offending and not GETs is likely that you are carrying a lot of data in the POST request body. GETs tend to be self-limiting. The larger the request, the longer it's going to take to pipe it in.

There's also a hint that perhaps the application's per-request processing time is a bit long, but that would be have to be measured.



Hmm... there's a misunderstanding about "triggered by server", I've updated previous post to "triggered by another server".
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic