I have an iPlanet webserver that serves xml (~25000 bytes) from an initial request of ~330 bytes. Most take anywhere from 10 (if cached) to ~80 ms. Some can take up to 1500 ms, if the access certain data types. Each server can get about 89000 requests per day. 6am-9pm - with about 20,000 hits during peak hours 8-9:30am.
We track any slow time above 2 seconds, including what part of the app they fall behind. I notice three slow areas.
1. reading xml from the input stream. (~330 bytes) (i.e. get stream, read, close stream)
2. getting data from db and storing in local variables
3. sending modified request to new server and retrieving response
After adjusting databases, adding hardware, adding memory, & cleaning up code we got it so the fallout was about 20-40 out of about 20,000 requests received during peak hours .
Problem:
Lately it seems that the fallout has increased to about 150-200 and seems to be coming from one user/one group. So we might have 110 slow times out of about 12,400, (during a particular hour), and all but two come from one user. That user sends 260 requests during that hour.
So one particular user sends 260 requests out of a total of 12400. Of those 260, 110 are "slow", out of a total of 112 "slow" times. Also many, other hours will receive the same number of requests and have almost no fallout.
Questions: Does a webserver read from the client and store locally BEFORE the servlet opens a stream and reads? I tried sending an request to my server from a client where I slept for 500 ms, after each byte sent. The client took awhile to send the xml but the server read the request and responsed very quickly.
What could cause one particular client's read times to be slow when everything around it is okay? There requests seem normal. Outside of the bad hour or so, every other request of theirs is fine. During the "bad" period, many other clients send & receive perfectly normal request/responses. During the "bad" period, client does send some legitimate requests. Has anyone ever seen something like this? I was thinking of seeing if I could not go to remote source and
test from that site to see if I could recreate. (not sure if that is even a possiblity...)
Any other ideas would be greatly appreciated.
Thanks.