Thanks Macro,
I would like to provide the following additional information about the issue.
All the sockets in CLOSEWAIT state are TCP sockets for sure. When I was listing the CLOSEWAIT sockets using lsof I could see that they are of ipv6 type, [machinename]:webcache->pool-96-228-253-182.tampfl.fios.verizon.net:20559 (CLOSE_WAIT). This is for sure for some http request sent by client. Here I could not derive whether this is socket between client-host or balancer-host.
For how long do you experience these problems?
Once sockets goes to CLOSE_WAIT they are not getting freed, and once file descriptors exhausts tomcat is not coming up. Once I restart the server the issue again occurs from 24 hours to 48 hours time period. Our server gets around 2.5 lac requests.
Please give me a little detail about how to trace connections that cause problem.
Thanks,
Prabhakar