• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

GC taking lots of time

 
Prashant Chandra
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi All,

Ours is retail domain Point Of Sale application. With normally not more than 5 concurrent users accessing application at a time through out the year. We have our application designed using IceFaces 1.8.2 (Open Source) and below are the other environment details.

ICEFaces : 1.8.2 (Open Source)
JSF : 1.2_13
container : tomcat 6.0.20
OS : Linux Red Hat 4.1.2-46
java version "1.6.0_17" HotSpot
database : Oracle Database 10g Release 10.2.0.4.0 - 64bit Production
startup params :
/usr/java/default/bin/java -Djava.util.logging.config.file=/opt/xx-xxx-web/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx4096m -XX:PermSize=512m -XX:MaxPermSize=2048m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -verbose:gc -Xloggc:/var/log/xx-xxx-web/gc.log -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:HeapDumpPath=/opt/xx-xxx-web/temp -XX:+HeapDumpOnOutOfMemoryError -Djava.library.path=/usr/lib64 -Dorg.apache.catalina.connector.RECYCLE_FACADES=true -Djava.endorsed.dirs=/opt/xx-xxx-web/endorsed -classpath :/opt/xx-xxx-web/bin/bootstrap.jar -Dcatalina.base=/opt/xx-xxx-web -Dcatalina.home=/opt/xx-xxx-web -Djava.io.tmpdir=/opt/xx-xxx-web/temp org.apache.catalina.startup.Bootstrap start

Problem:
Due to Christmas festival the number of requests have increased 10 times. suddenly the server heap memory utilisation has jumped all time high to 100%(then Full GC happens every 2 minutes which takes 7 seconds, all the users faces slow responses during that time), Perm Gen memory utilisation is never above 512 mb. we have increased number of application servers but still even increasing application servers(from 6 to 10 ) has not solved our problem.
We are planning to migrate to IceFaces EE version. but it is going to take time of 2 months. Please suggest me if we can do any quick fix to avoid this problem.

 
Deepak Bala
Bartender
Posts: 6663
5
Firefox Browser Linux MyEclipse IDE
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The real question is - 'What causes the GC to be triggered once every 2 minutes and what causes it to pause for 7 seconds ?'.

I would begin by putting your application through a stress test and using a profiler to look at which objects are causing a bottle neck. 7 seconds is a lot of time. Have you tried tuning you GC ?
 
Winston Gutkowski
Bartender
Pie
Posts: 10509
64
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Prashant Chandra wrote:Problem:
Due to Christmas festival the number of requests have increased 10 times. suddenly the server heap memory utilisation has jumped all time high to 100%(then Full GC happens every 2 minutes which takes 7 seconds, all the users faces slow responses during that time), Perm Gen memory utilisation is never above 512 mb. we have increased number of application servers but still even increasing application servers(from 6 to 10 ) has not solved our problem.

This is pure guesswork, but 6 application servers to run a POS application that normally has 5 concurrent users seems like a fabulous amount of power to me, even allowing for a dedicated db server. Are you sure there aren't other things going on that you haven't told us about?

Winston
 
Rojan punn
Greenhorn
Posts: 17
Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Disable all the GC verbose and add the below in the start up option

-XX:+UseParNewGC
-XX:+DisableExplicitGC
-XX:NewRatio=2
 
Randall Twede
Ranch Hand
Posts: 4469
3
Java Python Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
this doesn't really address the question(sorry), but i tend to agree with Winston. i had a mock e-commerce site once that ran on a Pentium 1 with a 56K dial-up modem and even with more than one user at a time they didn't complain it was too slow.
 
chander shivdasani
Ranch Hand
Posts: 206
Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Your best bet would be to use a profiler to see if you have any memory leaks. Another thing is to analyze your GC log file to see what your average heap usage is. If, Full GC is happening every 2 mins, it hints that you might have insufficient heap size. Try to increase it by a GB or two.
Can you run the "top" command along side and make sure that swap space is not being used. A swap space is nothing but a portion of Hard Disk which JVM uses as a backup memory when you dont have enough RAM. Accessing pages on hard disk can increase your pause times by several magnitudes. The chances of this happening is very less, but there is no harm in confirming.

 
Chris Hurst
Ranch Hand
Posts: 443
3
C++ Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Try turning gc logging on and try gcviewer or the like , should give you a visualisation of what's going on.

Visual VM is your friend as is Eclipse with the Visual MAT plugin run its leak report on a hprof you take with jmap what's it say the suspects are ?

Memory Analyzer

Leaks can cause this but so can memory living too long, i.e. taking too long to free up unused memory as it gets promoted to old and can sit there till a full GC (no CMS) e.g. remove not just cancel timers, look at every cache. Carefully look at any soft / weak reference usage.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic