site causing high cpu load in hosting

[Alan Yap]

Hi there,

This is regarding JSP/servlets, but I think more towards performance tuning.

I've just received an email from our hosting that the site we're hosting is causing a high CPU load, hence Resin needs to be restarted every few days. Well this cause some disruption in the running of the site, as expected.

While liaising with them regarding this issue, I hope the members here can help out in solving this problem.

Q1) What is the major perpetrator that cause high CPU load? too many database(db) connections/user sessions? Or db connection accidentally left open (especially in the servlets). Or is it just a simple case of too many page hits/web traffic, hence a signal for a better dedicated hosting package with more RAMs/CPU power/JVM size ?

Q2) Will restarting Resin every few days be a good temporary solution? We need to keep the site (with paid members) running, so I plan to do this around midnight, 2 times a week.



thanks

[vijay jamadade]

same problem we were facing in our application, but main reason behind that was, we forgot to close the database connections. thats what we did and problem got solved.
may be it will be due to technology problems. but i think mostly it is due to keeping the connections open for a longer time. try to close the connections as soon as the use is over.
thats what i can share.

[Nitesh Kant]

Alan:
What is the major perpetrator that cause high CPU load? too many database(db) connections/user sessions? Or db connection accidentally left open (especially in the servlets). Or is it just a simple case of too many page hits/web traffic, hence a signal for a better dedicated hosting package with more RAMs/CPU power/JVM size ?

Open DB connections account to use of file descriptors and not CPU usage. CPU usage indicates processing i.e. some program is running machine instructions.
At high load, high cpu usage is kind of desirable and a good indication of the scalability of the system. However, if the system is taking unusally long times to process requests and hogs a lot of CPU then it generally means something is wrong, eg:

Some thread is getting into an infinite loop

Some deamon thread is waking up and doing a lot of CPU intensive work and not giving the time to the request threads.

So, it is not kindda straight forward to tell what would be the reason for the high CPU usage. But, if you think the usage is high in absence of large number of requests, then you would have to profile your application and find out what is wrong.

Alan:
Q2) Will restarting Resin every few days be a good temporary solution? We need to keep the site (with paid members) running, so I plan to do this around midnight, 2 times a week.

I dont think this will be a great solution. Restarting an application twice a week is an indication of a very poorly built application.

[Vilmantas Baranauskas]

If after [re]starting the app, CPU load is little and then it increases over period of several days, you most likely have a leak.

This might be a thread leak or some other resource leak. Leaked DB connections may increase CPU load in some cases as well.

[William Brogden]

There are many more kinds of statistics that your hosting site could be looking at/providing to you to help figure this out.

With my Tomcat system I used the management app to discover that an orphan application was starting Threads that could not terminate, just consuming memory. Surely Resin provides some sort of remote monitoring facillity that you could use.

Bill

[Peter Johnson]

Taking some thread dumps should pinpoint what tasks are taking up the CPU. I suggest doing 3 or 4 dumps, about 30 seconds apart. Look for threads that are still in the same code. Several times I have caught code in an endless loop (even after vociferous protests by the developers that there were no endless loops in their code...)

[steve souza]

<<What is the major perpetrator that cause high CPU load? >>
It could be anything. You would need to profile and figure out where the hotspots are. A quick first step is to look at page level stats provided by your ISP to see if you can tell if a particular page is the problem, or it is pervasive against all pages. As another poster mentioned anything they can provide you would be a good starting point.

[Alan Yap]

Sorry for bumping this topic up again. I've been too busy to the last few weeks to deliver a project till I almost forgot to return here to read all your feedbacks.

Thank you so much for all your replies. I am much more enlightened now.

Yes, the very first thing that I'd done is read the site stats provided by the hosting. (webalizer). Happy to find out that..

- Very few page count that caused the dreaded Servlet Error (some occured obviously during when I tested and debugged my newly uploaded jsp/codes).
- 99.9% of the pages are served correctly.

Still, I did proceed to iron out potential runaway codes by ensuring all the loops did exit properly .

If I'm not wrong, Resin in my hosting has been running for more than 12 days without restarting now.

Since I'll be expecting the site usage to triple by end of this year (as our marketing team is busy promoting it), I won't be sitting down so comfortably yet, which brings me to a new query:

Will the "load" be lessened if Resin "serve" less volume of HTML but accept more requests, compared to less request, but more volume of HTML?

Here's a scenario of a shoutbox I made:

CURRENT SCENARIO:

On first load, the shoutbox.jsp will display recent 100 "shouts" (or messages). All members can post a new message into the shoutbox. My javascript timer ensure the shoutbox is refreshed completely in client browser (yes.. a simple page reload of 100 latest shouts). This means the Resin in the server needs to "serve" out a HTML page containing 100 latest "shouts" at an interval I've set (currently its 10 seconds delay). Can't imagine how the server felt if 1000 members are active in that page.

WHAT I PLAN TO DO:
On first load, the shoutbox.jsp will display recent 100 shouts. A javascript timer (now set at 4 or 5 seconds interval) will call a fetch_new_shout.jsp (in hidden iframe) that fetches only the latest shouts . And with the magic of DOM and some javascripts, I will append these new entries back into the main shout listing.

Since the interval is shorter, the number of requests sent to the server will double or more compared to my current implementation. I'm afraid it's still going to tax the server, or worse, crash it.

please advise. Thanks again










WHAT I

[steve souza]

why do you want to send back full changes every 10 seconds or partial changes every 4 or 5. What is wrong with sending partial changes every 10seconds? For that matter if you have scalability problems you could extend this interval based on how many people are logged in. Something like this: int refreshPeriod=(1+log(numSessions))*5;