• Post Reply Bookmark Topic Watch Topic
  • New Topic

PMAP and GCORE to troubleshoot native memory usage

 
Marcelo Jara
Greenhorn
Posts: 3
Java
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello,

I would like to know how to use PMAP and GCORE effectively to determine a possible memory leak in our application. The application is a webapp running in Jetty 7.2.2. It's using Java 1.6.0_25-b6. It sits on Oracle Linux 5.6 64-bit OS. The server is a virtual (vmware) server.

Summary:
We have a java application configured with a 3GB min/max heap size and 256MB of perm size. The server has 8GB of physical memory. After about 5-6 days of load, the server's physical memory is low and swapping activity increases. Using TOP and PMAP, the total RSS memory of the process is over 6GB. I am trying to determine what is taking the rest of the memory and why it keeps growing.

I decided to use pmap -x, and can see the ANON memory growing, but I don't know how to interpret the PMAP output to see what exactly is taking up the memory. Next I used gcore to dump the process memory. Since the process was using 8GB of virtual memory, this created an 8GB file. I am not sure how to analyze this file now.


1) PMAP output sample. In the sample below, do I interpret as saying that all ANON blocks following dnsns.jar are used by dnsns.jar? In other words, does PMAP list the output in a way that all contiguous ANON blocks belong to the library, jar, .so, etc. file right before it?

00002aaac7dd5000 12 0 0 ----- [ anon ]
00002aaac7dd8000 1016 44 44 rwx-- [ anon ]
00002aaac7ed6000 12 0 0 ----- [ anon ]
00002aaac7ed9000 1016 48 48 rwx-- [ anon ]
00002aaac7fd7000 8 0 0 r-xs- dnsns.jar
00002aaac8000000 65532 56968 56784 rwx-- [ anon ]
00002aaacbfff000 4 0 0 ----- [ anon ]
00002aaacc000000 65536 57688 57440 rwx-- [ anon ]
00002aaad0000000 12 0 0 ----- [ anon ]
00002aaad0003000 1016 44 44 rwx-- [ anon ]
00002aaad0101000 12 0 0 ----- [ anon ]
00002aaad0104000 1016 48 48 rwx-- [ anon ]
00002aaad0202000 12 0 0 ----- [ anon ]
00002aaad0205000 1016 48 48 rwx-- [ anon ]


2) GCORE file: This is basically a core dump of the process. What tool(s) can I use to look in it to see what is taking up the memory?




Thanks!
 
Jayesh A Lalwani
Rancher
Posts: 2762
32
Eclipse IDE Spring Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you sure it's a native memory leak? Have you looked at the Java heap to find if that there are any leaks in the Java code

I have to say it's been years since I have heard of native leaks. Because of the number of people who use Java, it;s very rare for a native leak to survive. Have you tried just upgrading Java?
 
Marcelo Jara
Greenhorn
Posts: 3
Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
No, I'm not sure if it's a native memory leak. The symptoms I see is that the server starts to swap after a 5-6 days under production load. We have GC logging enabled and the GC times are fine for the first 5 days. The Full GCs slowly start to increase from 1-2 seconds to over 100 seconds. The following article explains how swapping may be causing high GC times (https://blogs.oracle.com/poonam/entry/troubleshooting_long_gc_pauses).

The amount of free memory available also goes down over time. We get alerts whenever over 95% of the physical memory is utilized on these servers. We never get OOM alerts. This could be due to the server becoming unresponsive due to the full GCs. The application is just restarted and placed back in rotation.

When looking at top, the RES memory for the process starts at about 3.5 GB and slowly goes over 6GB over time. The heap memory seems fine to me which is why I was leaning towards a native memory issue. The application makes use of many 3rd party libraries such as Oracle TimesTen cliet, C3p0, etc..


As for the Java version, yes we are going to upgrade to Java 1.7 very soon.


I tried to attach the full pmap output as file, but it's not letting me attach .txt, .log, or .zip files.
 
Jayesh A Lalwani
Rancher
Posts: 2762
32
Eclipse IDE Spring Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yea yeah, if GC logs show ever increasing GC times, then it's a typical Java memory leak, then. It's not a memory leak inside the JVM. It;s a bug in the Java code. The first sign of a typical memory leak in Java code is that it get slower before it goes out of OOM. GC starts running more often, and uses up most of the resources. Long GC pauses slow down the processing, so the entire application slows down. THis means that the application starts leaking slower and slower too. This just staves off the inevitable OOM. There's a negative feedback loop that just slows down the problem but doesn't solve it. I've seen applications run at 10% of the throughput for 2 days before running out of OOM

What you need to do is use JMap to get a heap dump. Don't use PMap. the tools provided by OS won;t help you diagnose Java memory leaks. JMap is a utility provided by Java that can dump and inspect the heap. JMap itself can give you some stats of the objects in the heap. However, it's better to just take the full heap dump and analyze it with a memory analysis tool like Eclipse MAT. This will tell you which objects are present in the heap. You will need someone who is familiar with the code to look at the heap dump to tell you where the problem is.

Or you can take a before and after heap dump. Take a heap dump after the server has been running 1 day. Then take it after it has ran for 3 days. Open both heap dumps in MAT and compare. You will find your culprit. You will still need someone who is familiar with the code to fix it.
 
Marcelo Jara
Greenhorn
Posts: 3
Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ok. Will look into that. So you are saying that the every increasing Anonymous memory blocks shown in PMAP, which sit outside the heap, is caused by a leak somewhere in the heap?

So in the snippet below, the 3GB heap for the application is marked as one large anon block. All other anon blocks in the pmap output then are due to a leak of some sort in the application?


0000000740000000 3145728 3064900 3063136 rwx-- [ anon ]
000000374a600000 112 104 0 r-x-- ld-2.5.so
000000374a81b000 4 4 4 r-x-- ld-2.5.so
000000374a81c000 4 4 4 rwx-- ld-2.5.so
000000374aa00000 1336 620 0 r-x-- libc-2.5.so
000000374ab4e000 2048 0 0 ----- libc-2.5.so
000000374ad4e000 16 16 8 r-x-- libc-2.5.so
000000374ad52000 4 4 4 rwx-- libc-2.5.so
000000374ad53000 20 20 20 rwx-- [ anon ]


Entire PMAP output (hope it posts here).

 
Jayesh A Lalwani
Rancher
Posts: 2762
32
Eclipse IDE Spring Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
All I'm saying is that in 12 years of developing applications in Java, I've never had to go to the core dump from the OS to solve a memory leak.. and there have been years in which, I've solved more memory leaks than most people have seen in their career. I would look at the Java heap dump before looking at the core dump. Whatever is happenning in the core is most probably a side effect of whatever memory leak is hapenning in Java. The leaked object is probably holding on to some other native resources that don;t show up on the Java heap. The simplest explanation is a Java memory leak, and you should look at that first.

If you find no memory leak in Java but a huge memory leak in JVM, then you have found a bug in JVM, and you will need to file a bug report with Oracle and/or try a differrent version of Java.
 
vijay naga
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
1) PMAP output sample. In the sample below, do I interpret as saying that all ANON blocks following dnsns.jar are used by dnsns.jar? In other words, does PMAP list the output in a way that all contiguous ANON blocks belong to the library, jar, .so, etc. file right before it?
--> I also have the same doubt, can anybody confirm if an anon block belong to the library/jar/so right before it ?, thanks!
 
Winston Gutkowski
Bartender
Posts: 10571
64
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Marcelo Jara wrote:Entire PMAP output (hope it posts here)...

Marcelo,

I changed your code block to a text block (just put "code=text" inside the brackets), which is much better for displaying non-Java output like this.

As a general point though, I'd avoid posting that much data. Nobody is ever going to read it.

Winston

[Edit] Just noticed that Marcelo's probably long gone. I'll leave the post for information though.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!