• Post Reply Bookmark Topic Watch Topic
  • New Topic

Strange webserver performance  RSS feed

 
John Dunn
slicker
Ranch Hand
Posts: 1108
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have an iPlanet webserver that serves xml (~25000 bytes) from an initial request of ~330 bytes. Most take anywhere from 10 (if cached) to ~80 ms. Some can take up to 1500 ms, if the access certain data types. Each server can get about 89000 requests per day. 6am-9pm - with about 20,000 hits during peak hours 8-9:30am.

We track any slow time above 2 seconds, including what part of the app they fall behind. I notice three slow areas.
1. reading xml from the input stream. (~330 bytes) (i.e. get stream, read, close stream)
2. getting data from db and storing in local variables
3. sending modified request to new server and retrieving response

After adjusting databases, adding hardware, adding memory, & cleaning up code we got it so the fallout was about 20-40 out of about 20,000 requests received during peak hours .

Problem:
Lately it seems that the fallout has increased to about 150-200 and seems to be coming from one user/one group. So we might have 110 slow times out of about 12,400, (during a particular hour), and all but two come from one user. That user sends 260 requests during that hour.

So one particular user sends 260 requests out of a total of 12400. Of those 260, 110 are "slow", out of a total of 112 "slow" times. Also many, other hours will receive the same number of requests and have almost no fallout.


Questions: Does a webserver read from the client and store locally BEFORE the servlet opens a stream and reads? I tried sending an request to my server from a client where I slept for 500 ms, after each byte sent. The client took awhile to send the xml but the server read the request and responsed very quickly.



What could cause one particular client's read times to be slow when everything around it is okay? There requests seem normal. Outside of the bad hour or so, every other request of theirs is fine. During the "bad" period, many other clients send & receive perfectly normal request/responses. During the "bad" period, client does send some legitimate requests. Has anyone ever seen something like this? I was thinking of seeing if I could not go to remote source and test from that site to see if I could recreate. (not sure if that is even a possiblity...)

Any other ideas would be greatly appreciated.

Thanks.
 
Virag Saksena
Ranch Hand
Posts: 71
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think you already might be close to the answer.
If only one user is experiencing slow times, it might be something to do with the data that user's requests are accessing. First thing to look at is where is most of the time for this user's requests going ?

I know from a fact in the databases, that data distribution skews can have a significant impact on your response times. Consider the following case :
If you have 20,000 users logging service requests, however 1 user logs 10% of all the service requests. Now if you provide a summary of a user's service requests in the beginning of every request, then most of the users will be okay, but that users's query will experience poor response time because of the data skew.

Again, you need to determine where the time is being spent for this user (waiting for the database, using CPU, waiting for IO, waiting for lock, etc) and contrast it with the cases when response time is okay
 
John Dunn
slicker
Ranch Hand
Posts: 1108
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
In above example, for that one hour, all slowness is ONLY reading the request; the remainder of the processing will be between 15 & 80 ms. Weird thing, is that it is only ~340 bytes!!! Most of the time, it takes 0 or 1 ms, (we're using iPlanet on Solaris.)

I was wondering if a user is pummelling my webserver with requests, will that interfere somehow with the reading of the inputStream. BUT, it seems that the webserver will read all the data from the client before the servlet starts reading.

Anyone come across this, or anything like it?
 
John Dunn
slicker
Ranch Hand
Posts: 1108
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Looks like the CPU usage was being maxed out by another app, in conjunction with the couple of users sending a steady stream of requests.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!