First I would like to brief you all about what we are doing. We had developed a automatic call distribution
a) Open Source Telephony Platform FreeSWITCH (FS).
b) FreeSWITCH ESL(Event Socket Library).
c) A socket application developed in java
c) MySQL Database.
It works this way,
When a calls comes to FS server, it send out RFC messages with a header and key-value pairs in body to the
listener listening on the socket port throughout the life-cycle of the call. Upon receiving these messages, a new thread is spawned for every message it receives.
We had gone with these two approaches
(1) When a call enters the server, a call bean is created and added to hash-map. And on respective call activity such call ringing, call put on hold etc.,
the details are updated in the call bean in the hash-map. When the call hangs up this call details is inserted in MySQL database and finally removed from the call map.
The problem with this approach is that solution is not scalable.
(2) So we decided to move in to database for this, i.e logging and updating call details as and when they are received etc.
Here is the main problem, upon successful testing we put this in to production after every run for nearly 5-6 hours an OutOfMemory Exception is thrown and the application crashes.
Using htop the linux utility, we found that when this happens, the memory utilization is around 1-2 gig out of 48 gig on every run.
So the problem is not with the memory or java heap getting exhausted.
After struggling for few days we found that using htop, when OS thread count increase above 32K the application crashes by throwing an OutOfMemory exception.
But by going with approach (1) the thread count never increase beyond 800 and comes down to around 300.
We had checked our code for any unused reference, any exception caught, moved objects to methods definition but still there are blocked threads when going with approach (2).
Using JConsole and VisualVM is not of much help, as the number of threads spawned is high, an average 20-30 rfc messages are sent on socket for every single call activity.
Don't know what to do.
Any help is greatly appreciated
Thanks In Advance