GC problems, heap grows

[Siarhei Kaur]

Hello,

I’ve a problem with performance.
Java+jruby web application works under Tomcat 5 onto server with the following parameters: Solaris x64, amd64, 64-bit Java, RAM 8GB, CPU 2,7 GHz.

I have load test which simulates workload with any number of users.

Configurational setclasspath.sh file is attached (in zip). I ran tests with the parameters as in the attached file, only newRatio parameter was changed during passing tests.

server.xml is attached too (in zip).

First Test:
newRation=1
number of user: 50

First Test Results:
GC_newRatio=1.log
GC_newRatio=1.JPG

---------------------------------------

Second Test:
newRatio=2
number of user: 50

Second Test Results:
GC_newRatio=2.log
GC_newRatio=2.JPG

---------------------------------------

Third Test:
newRatio=10
number of user: 50

Third Test Results:
GC_newRatio=10.log
GC_newRatio=10.JPG


All files in the attached results.zip archive which is available via http://socialenginezzz.com/kovrigin/

With the lapse of time heap size grows and after a few minutes tomcat frozen and application is not accessible. I think that GC cannot clean the necessary heap.

Could you please help me to understand and solve current problems? Maybe we must change something in GC properties in Tomcat or something else?


With my true regards,
Sergey K.

[William Brogden]

What make you think it is a GC problem?

I would suspect that objects or system resources are not being released so GC can't do a thing about it.

If this was my problem I would be using the Manager app to watch memory use and the request Thread status.

Bill

[Siarhei Kaur]

Bill,

What do you mean under Manager app? Do you mean any tool for analyzing memory and threads?

I've used Visual VM (standart tool in jdk1.6). "heap usage diagram.JPG" picture shows heap usage and threads count. Is it what you mean?

Please correct me if I was wrong.


Sergey K.

[Remko Strating]

Just do some profiling and see what objects are created.

During the profiling you will see that some objects are not deleted and then you know there is a memory leak.

Mostly the reason is that you keep a reference of the object in memory.

If you find the cause you can adjust your code so that the problem will not occur again.

[William Brogden]

I meant the Manager app - which ships with Tomcat but is not turned on by default. If this shows an accumulation of request Threads which have been waiting for a long time, that is a clue you need to follow up.

Presumably VisualVM is showing you accumulation of objects which should have been discarded, right?

Bill

[Siarhei Kaur]

First of all I would like to say that all mentioned below files are available by the link - http://socialenginezzz.com/kovrigin

Please allow me to explain the situation on the other hand.

I ran a test with 5 users. Duration of this test was near 3.5 hours.

Heap usage is represented at "heap_usage.jpg".

My test is consists of 1 request (except login and logout actions) which you can see in "piece of catalina log.out". As you can see, sometimes time of the request reaches 1,5 min instead of normal execution time which equals 1.5 sec!

At "GC usage.JPG" you can see the GC activity. As I right understand tenured generation consists of objects which still have references. On screenshot we can see that the tenured generation space is growing. Does it means that there is a memory leak in app?

Or GC can not cope with the work?


Application is not accessible when heap is big:
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.
Reason: Error reading from remote server


Heap dump was created by mat tool too - "mat.jpg". We can see a big jruby objects.

Bill,
maybe you could provide me with another tool for analyzing threads?

[William Brogden]

1. GC will ALWAYS be able to collect objects that have no reference, trying to blame GC is a blind alley

2. Accumulating JRuby objects is certainly suspicious as is the occasional long delay.

3. If I had to bet, I would bet your code is not properly closing/disposing of some resource such as database connections or graphic contexts. Since you don't say what resources the app uses, thats just my guess.

Bill

[SumitPal Pal]

Please use a profiler and see which objects are growing in size as the program executes.
That will give you an idea of where the problem area in your program could be.
GC is not a super magic thing - it can release objects only if your program allows and is written correctly.
GC cannot release objects to which references still exist in your program - because those objects can be used again by your program.

I hope this makes sense and is helpful