How to read memory usage for process running in Linux,

[Susan Smith]

I have a Java program that I run in LINUX machine:

I want to analyze the memory usage used by this program

This article (http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html ) recommends "pmap" to get more accurate picture [please see code snippet for result]. It mentions that I need to look at the "writeable/private" at the bottom of the result of running pmap command. I'm really confused why it is using 225,024K. Why is this?

Is "writeable/ private" result of the pmap command giving you accurate picture of the cost of this process, factoring out the shared libraries?

Thanks in advance for the help.

[Daren Wilson]

would valgrind to what you need? http://www.valgrind.org/

[Jesper de Jong]

I don't think Valgrind would be the most useful tool for Java programs. It's really meant for native code (written in C or C++ for example).

You could try JConsole (included with the JDK since Java 5) or VisualVM (included in the latest update of JDK 6; you can also download it separately from that link).
[ August 20, 2008: Message edited by: Jesper Young ]

[Susan Smith]

I run a program and monitor using visualvm....
perm gen + heap size total only 7M .. but the "writeable/ private" part is 118M ...

Is Java using more that what's in JVM?? much much more?

[Peter Johnson]

The memory usage for the java process is more that just the Java heap. It consists of (hope it didn't forget anything):

the Java heap (where allocated objects go)

the permgen (where static class data goes)

the thread stacks (where method-local references, basic data values, and other information used to maintain method calling go)

various other C/C++ data structures used by the JVM to do its job

the binary code for the JVM and the libraries that it uses to do it job

the binary, compiled code for the Java methods (I don't think that this resides in the permgen, but don't quote me on that - I haven't dug enough into the JIT compiler to know exactly where the machine ops go)

In a small program (hello world, for example), all of the non-heap stuff occupies the vast majority of the memory.

In a large program (and application server hosting many EJBs, web services, web app, etc.) with a large heap, the heap occupies the vast majority of the memory.

Using OS-native tools to monitor a Java application's memory usage is usually not that enlightening because such tools cannot distinguish between memory used by Java objects (what you are really interested in) and all of the other memory the JVM uses to do its job (what you are not interested in and have no (or little) control over anyway).

[Pat Farrell]

Originally posted by Peter Johnson:
The memory usage for the java process is more that just the Java heap. It consists of (hope it didn't forget anything):

You left out the code of the JVM itself and any local variables.

[Susan Smith]

(1.)
So how do I know exactly the total memory used by a Java process in Linux?

(2.)
And is there a way to know separately "memory used by Java objects" and "all the other memory" the JVM uses to do its job

(3.)
Is there any details reference (e.g.: URLs) that mentions the complete list of memory being used by a Java process?

[Peter Johnson]

You left out the code of the JVM itself

No I didn't this was covered under "the binary code for the JVM and the libraries that it uses to do it job".

You left out ... any local variables.

Which local variables, the Java ones (covered under "the thread stacks") or the JVM's C/C++ ones (covered under "various other C/C++ data structures used by the JVM to do its job")?
[ August 27, 2008: Message edited by: Peter Johnson ]

[Peter Johnson]

Question 1, what do you mean by "total memory used by a Java process"? I interpret this to mean you are looking at memory usage from an OS point of view. In that case, any of the OS tools that provide this kind of data will do. Just do not confuse the amount of memory used for the Java process with the amount of memory used by the Java application (which is the amount of heap space used by the live Java objects plus the space used in the permgen for the classes)

Question 2, you can use JConsole of VisualVM to monitor heap space usage and permgen usage. They will tell you the current heap and permgen size, and the amount of those areas that the currently in use. If you add the heap size and permgen size and subtract that from the actual memory used by the Java process, you should get the "all the other memory" size. On Windows this calculation can get tricky because you have to know which memory counter to actually look at; I am not sure how tricky the calculation is on Linux. (I'll give it a try later and report back.)

Question 3, that I do not know. But there are many places that describe how the heap is managed.

[Susan Smith]

Thanks Peter. Yes, I think eventually I want the Java process (the overall memory used by the process) even though I'm also interested to monitor the heap and perm-gen only.

Looking forward to what you find in running the process in Linux.

Thanks again for the help.

[Susan Smith]

If you try this code below, then you run "pmap -d PID",
then the writeable/ private: 1217748K. Doesn't really make sense to me why it is using that much.

public class HelloWorld
{
        public static void main(String[] args)
 
{
try
{
        while ( true )
{
        System.out.println("Hello World!!!");
        Thread.sleep (1*60*1000);
}
}

catch (Exception ex)
{
ex.printStackTrace();
}
}
}

[Peter Johnson]

What JDK are you using and what is the command line? I just now tried on Ubuntu 8.04, JDK 6u7, with the command "java Hello". For pmap, I got 195640K for writeable/private, which is 1/10th what you posted. But that still seems too large. According to a stack dump, the heap size is around 6MB and permgen around 12MB. The *.so files use up another 15M or so. In the pmap I see two anon areas taking over 50M, I have no idea what those are about.

Looking at top, I see a total of 919M of memory in use - when I kill the Java app it goes down to 916M. There is no change in the swap memory usage. This implies that the HelloWorld app is in fact using only 3M of memory.

Reading the man page for pmap, it explains that pmap is a memory map for the process. That means, within the address space of the process, what addresses are currently allocated (which implies that process can access that memory, though I image that some of the addresses are accessible only in kernel mode because the addresses belong to the operating system). Note that this is not an accurate reflection of memory usage for a process - many of the address areas are for use by the OS. Now the writeable/private size sounds more reasonable, and if we compared what is running on your box to what is running on mine we could also account for why your writeable/private size was 10 time mine.

[Susan Smith]

I'm using JDK 1.5

Command line: java -classpath .... HelloWorld

But when I add "-mx2m" to the command line, the writeable/ private becomes low (164364K) --> similar to yours.

[Peter Johnson]

Add a System.gc() call right after the println, and add the -XX:+PrintHeapAtGC option to the command line. That will let you know the heap size when you do not specify it on the command line (the default size depends on a lot of variables).

[Susan Smith]

I don't see GC running yet for my simple HelloWorld program. So I run another program with "-verbose:gc" and "-XX+PrintHeapAtGC"
The heap size seems small but the writeable/ private is still large: 214828K

Here's some snapshot of the result of the "-verbose:gc" and "-XX+PrintHeapAtGC":

{Heap before gc invocations=46:
 PSYoungGen      total 2240K, used 1941K [0x00002b33d3c80000, 0x00002b33d3f10000, 0x00002b33d3f10000)
  eden space 1856K, 90% used [0x00002b33d3c80000,0x00002b33d3e25448,0x00002b33d3e50000)
  from space 384K, 66% used [0x00002b33d3eb0000,0x00002b33d3ef0000,0x00002b33d3f10000)
  to   space 384K, 0% used [0x00002b33d3e50000,0x00002b33d3e50000,0x00002b33d3eb0000)
 PSOldGen        total 3520K, used 2695K [0x00002b33d3910000, 0x00002b33d3c80000, 0x00002b33d3c80000)
  object space 3520K, 76% used [0x00002b33d3910000,0x00002b33d3bb1c58,0x00002b33d3c80000)
 PSPermGen       total 21248K, used 8149K [0x00002b33ce510000, 0x00002b33cf9d0000, 0x00002b33d3910000)
  object space 21248K, 38% used [0x00002b33ce510000,0x00002b33ced05588,0x00002b33cf9d0000)
[GC 4636K->2895K(5760K), 0.0031940 secs]
Heap after gc invocations=46:
 PSYoungGen      total 2240K, used 192K [0x00002b33d3c80000, 0x00002b33d3f10000, 0x00002b33d3f10000)
  eden space 1856K, 0% used [0x00002b33d3c80000,0x00002b33d3c80000,0x00002b33d3e50000)
  from space 384K, 50% used [0x00002b33d3e50000,0x00002b33d3e80000,0x00002b33d3eb0000)
  to   space 384K, 0% used [0x00002b33d3eb0000,0x00002b33d3eb0000,0x00002b33d3f10000)
 PSOldGen        total 3520K, used 2703K [0x00002b33d3910000, 0x00002b33d3c80000, 0x00002b33d3c80000)
  object space 3520K, 76% used [0x00002b33d3910000,0x00002b33d3bb3c58,0x00002b33d3c80000)
 PSPermGen       total 21248K, used 8149K [0x00002b33ce510000, 0x00002b33cf9d0000, 0x00002b33d3910000)
  object space 21248K, 38% used [0x00002b33ce510000,0x00002b33ced05588,0x00002b33cf9d0000)
}
{Heap before gc invocations=47:
 PSYoungGen      total 2240K, used 192K [0x00002b33d3c80000, 0x00002b33d3f10000, 0x00002b33d3f10000)
  eden space 1856K, 0% used [0x00002b33d3c80000,0x00002b33d3c80000,0x00002b33d3e50000)
  from space 384K, 50% used [0x00002b33d3e50000,0x00002b33d3e80000,0x00002b33d3eb0000)
  to   space 384K, 0% used [0x00002b33d3eb0000,0x00002b33d3eb0000,0x00002b33d3f10000)
 PSOldGen        total 3520K, used 2703K [0x00002b33d3910000, 0x00002b33d3c80000, 0x00002b33d3c80000)
  object space 3520K, 76% used [0x00002b33d3910000,0x00002b33d3bb3c58,0x00002b33d3c80000)
 PSPermGen       total 21248K, used 8149K [0x00002b33ce510000, 0x00002b33cf9d0000, 0x00002b33d3910000)
  object space 21248K, 38% used [0x00002b33ce510000,0x00002b33ced05588,0x00002b33cf9d0000)
[Full GC 2895K->1499K(5760K), 0.0608040 secs]
Heap after gc invocations=47:
 PSYoungGen      total 2240K, used 0K [0x00002b33d3c80000, 0x00002b33d3f10000, 0x00002b33d3f10000)
  eden space 1856K, 0% used [0x00002b33d3c80000,0x00002b33d3c80000,0x00002b33d3e50000)
  from space 384K, 0% used [0x00002b33d3e50000,0x00002b33d3e50000,0x00002b33d3eb0000)
  to   space 384K, 0% used [0x00002b33d3eb0000,0x00002b33d3eb0000,0x00002b33d3f10000)
 PSOldGen        total 3520K, used 1499K [0x00002b33d3910000, 0x00002b33d3c80000, 0x00002b33d3c80000)
  object space 3520K, 42% used [0x00002b33d3910000,0x00002b33d3a86c08,0x00002b33d3c80000)
 PSPermGen       total 21248K, used 8149K [0x00002b33ce510000, 0x00002b33cf9d0000, 0x00002b33d3910000)
  object space 21248K, 38% used [0x00002b33ce510000,0x00002b33ced05588,0x00002b33cf9d0000)
}

Just a side note of why I am investigating how exactly to know the memory usage used by a JAVA process:
--------------------------------------------------------------
We have a server where around 100+ console-based java programs running at the same time in a REDHAT linux server (looping continuously).

This is what happen:
1. Every time just a little after we reboot the server and restart all the programs, the memory usage is very low.

2. Then after some times (the next day/ ..), the memory starts building up until it almost uses the full capacity.

Then we buy some more memory and put it into the server but #1 and #2 starts happening again.
We can keep buying and putting more memory but it seems it tries to always use everything.

My colleagues are blaming the way Java works. I'm not convinced yet that this is the case.
I'm thinking that this is maybe because of how LINUX works, so most of the memory for a Java process is actually not on the heap size.
--------------------------------------------------------------

[Peter Johnson]

You have over 100 Java apps running at the same item? Ouch! By default, the JVM starts with a small heap and then grows the heap automatically based on various bits of data that it collects. The heap will grow until the maximum allowed, which could be almost anything if you do not set it at the command line (the max is based on the OS, machine RAM, whether client or server, and probably a few other factors).

But even if the max heap is set for each app, there could be an issue. For example, lest say that each app has max heap set to 256M. Times 100 users, that is 25GB(!) if each JVM decides it needs more heap. And if you let a Java app run long enough, and the app remains busy, the JVM will eventually ask for the max heap allowed. Of course, if you could verify that the apps will run adequately with smaller heaps, you could set a smaller max heap (32M, 48M, 64M ...)

I don't know if you can move to JDK 6, but you might find it to work better in this environment. With each release Sun tweaks the heap manager to make it more efficient. Perhaps it will not grow the heap that much.

[Susan Smith]

I set the heap size for the programs ... mostly set to have from 8M - 64 M only ...

Yesterday evening, I finally able to do remote-monitoring for one of those 100 programs using JConsole. I need to do this because my server machine isn't GUI based.
This morning I saw:
- The heap memory usage is steady. In average around 3 - 4MB
- The non heap memory usage keeps increasing (slowly) ---> I notice it is caused by the "Code Cache" part.

So I think, if I try to summarize the issues again:
1. What is up with the increasing memory that keeps making us buying more memory?
2. How to check memory usage for a Java process in Linux? I'm still curious if the "writeable/ private" result of the "pmap" command really giving accurate picture of the total memory for the processes.

Thanks for your ongoing help, Peter.
I really appreciate it.

[Peter Johnson]

AS I explained about pmap in an earlier port, what it gives is a breakdown of the process' address space. This does not equate to the memory used by the process. You primarily want to use ps - it tells you memory used, but even that is suspect if you are running multiple copies of the same process (which you are - you have multiple copies of the java process). Here is a good explanation: http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html . The best way to tell how an individual process effect memory usage is to terminate the app and see how much free memory increases. In my earlier example, that was 3MB for HelloWorld.

Also, googling "linux process memory usage" will turn up a lot of information.

[Susan Smith]

1.) Regarding "pmap"
That article is what I referenced on my first post. This article that leads me to using pmap instead of ps. I saw lots of people referencing this post.

But what confuses me is that whether it's really reliable or not. This, I haven't found any answer yet by googling.

It doesn't make sense to me .. let say in my case... if one process 200MB in average then with my server 16 GB only and the number of processes is 100 --> the total is 20 GB.

It seems the "writeable/ private" number is too big.

2.)

The best way to tell how an individual process effect memory usage is to terminate the app and see how much free memory increases

Let me think how I can incorporate this to my testing.

3.)
Any thoughts about why the "code cache" steadily increasing?

[Peter Johnson]

1) Please carefully re-read my last post. There is apparently something that you are not understanding about my pmap explanation, but I have no idea of where the misunderstanding is. (My point of referencing that web page was to warn you that the data shown by the ps is not an accurate reflection of memory used by an individual process. The pmap utility is not an acceptable utility for determining the amount of memory used by a process.)

3) From a Sun tech article:

Code Cache (non-heap) The HotSpot JVM also includes a "code cache" that contains memory used for compilation and storage of native code.

As the JVM runs more code, it compiles. The compiled code is in the code cache. As your app employs new functionality and runs new code, those methods get compiled. For example, a user might first open a file (all of the file opening methods get compiled), then edit that file (the editing methods get compiled), then saves the file (the file saving methods get compiled). In this example, the size of the code cache increases three times. The longer you run your app, the more functionality will be invoked, the more methods will get compiled, the larger the code cache gets. If you app dynamically generates method code, then this cache can increase fairly rapidly, depending on the speed at which the code generates new code.

[Peter Johnson]

By "dynamically generates method code" I mean that you have some code that itself writes, and then compiles, more Java code. Application servers (JBossAS, Websphere, Weblogic Server, etc) do this quite often when supporting various Java EE entities.

In general, once your application has run every method it contains (and every method in the various JAR files used by your application) then all of your methods will have been compiled and you should not see code cache increases. However, if you are using the "server" JVM, it will periodically recompile methods that it compiled previously. It does this because the more data it gathers about how a method executes, the better it can generate efficient machine op codes. When it does this, you could see the code cache size increase. (I do not think the memory for the previously compiled machine op-codes are released to the operating system - this would lead to memory fragmentation. I think, instead, that the JVM maintains a list of available memory locations and if the new code will fit into an existing memory location (were some old op-codes were located), will place the new op-codes; otherwise it will use new memory.)

If you are using the "server" JVM, you could try using the "client" JVM instead and see if that helps with the code cache size.

[Pat Farrell]

Originally posted by Susan Smith:
We have a server where around 100+ console-based java programs running at the same time in a REDHAT linux server (looping continuously).

This may be a bit OT, but I would never expect 100+ java programs to run well. Can you combine them? You have a lot of potential overhead keeping all of them separate.

[Susan Smith]

This may be a bit OT, but I would never expect 100+ java programs to run well. Can you combine them? You have a lot of potential overhead keeping all of them separate.

Pat,
I use multiple process with the considerations that I need to:
1. Monitor the console output for each individual process in detail. I think if I use threads the output will be hard to read.
2. One process could hung from time to time or throw Exception, I need to restart the process during Exception or retry a process during hung.

I forgot some other reasons.... I will try to remember and let you know more later.

[Susan Smith]

One quick question about JConsole:

Based on the definition of "committed memory":
Total amount of memory allocated for use by the heap.

So this memory already claimed by JVM even though the "current heap size" is much lower than it?
[ September 04, 2008: Message edited by: Susan Smith ]

[Peter Johnson]

That is correct. "Committed memory" is the amount of memory allocated to the heap, while "Current heap size" is the actual amount of space used by Java objects in the heap.

You can verify this by observing the GC statistics provided by options such as -verbose:gc and -XX:+PrintHeapAtGC.

[Susan Smith]

Peter, back to this point you mentioned:

The best way to tell how an individual process

effect memory usage is to terminate the app and

see how much free memory increases

I run a program, then monitor using Jconsole:
Committed memory for non heap: 28,416
Committe memory for heap: 43,392
Using "top" command, the memory free, around 40 K

Then I kill the program, the memory free: 140 K (after killing it)

Am I in the righ direction as determining the amount of memory actually used by the process?

[Susan Smith]

Another question that came to my mind:

When is it exactly best to use "server" option and when is it best to use "client" option of java command?

[Susan Smith]

I found this URL:
http://java.sun.com/docs/hotspot/HotSpotFAQ.html#compiler_types

And it says that:

The client system is optimal for applications which needs fast startup times or small footprints. The server system is optimal for applications where the overall performance is most important

- What does it mean by "overall performance" regarding the server system?
- I assume "small footprints" meaning resources like memory, etc (the client system)?

[Peter Johnson]

The only real way to tell if the 'server' or 'client' JVM will work best for you is to try them.

In general, the client VM is targeted at short-running applications (an Ant build, for example), while the server VM is targeted at long-running applications (application servers, for example). Neither VM has a smaller memory footprint.

[Susan Smith]

Just realized, I can't run -client option in my server machine to test.

It's not available since my server machine is 64 bit.
http://java.sun.com/j2se/1.5.0/docs/guide/vm/server-class.html