JVM Memory on Linux

[Drew Lane]

I have two computers that I use as servers.
They both have the same hardware.
They are kind of old but they both work. :-)

Both have 512MB RAM

One has Linux kernel 2.4 with Java 1.4.1

I recently upgraded the other machine to kernel 2.6 with Java 1.4.2

I seem to be having a problem with the JVM using virtual memory instead of RAM on the "upgraded" computer.

After I start my Java application server (Resin) there is hard drive activity every 5 seconds and the performance is slower on the computer with the newer kernel. The top command shows about half of the real memory is being used on this computer.

The computer without the upgrade shows almost all of the memory is being used by the system and I don't get the hard drive activity every 5 seconds.

Does anyone know what's going on here?

[Charles Lyons]

Have a look at this article on Tuning Swappiness (and swapping behaviour). It may or may not help (post back and let us know).

512MB is not a lot of RAM for a Java EE server under load though. I generally recommend at least 1GB for any server these days, with RAM being so cheap. How large is your swap?
[ December 16, 2008: Message edited by: Charles Lyons ]

[Charles Lyons]

It is also possible that your server isn't set to overcommit (seems unlikely given you say not much RAM is in use). A JVM allocates a lot more memory than it ever uses, so that can be a problem. Here's a quote about this 2.6 kernel feature:

Memory overcommit is a Linux kernel feature that lets applications allocate more memory than is actually available. The idea behind this feature is that some applications allocate large amounts of memory "just in case", but never actually use it. Thus, memory overcommit allows you to run more applications than actually fit in your memory, provided the applications don't actually use the memory they've allocated. If they do, then the kernel terminates the application.

There's more advice in this article on Linux Memory Overcommit.

[Drew Lane]

Originally posted by Charles Lyons:
Have a look at this article on Tuning Swappiness (and swapping behaviour). It may or may not help (post back and let us know).

512MB is not a lot of RAM for a Java EE server under load though. I generally recommend at least 1GB for any server these days, with RAM being so cheap. How large is your swap?

[ December 16, 2008: Message edited by: Charles Lyons ]

I set the swappiness to 0 but I'm not seeing any dif.

I hear you about the RAM, but this is a very old computer and it's not so cheap - not worth investing more money into but it's been such a workhorse it's hard to give it up. :-)

The swap size is 458MB and I don't even see it getting used when I run top!

I don't know...maybe I should go back to the 2.4 kernel for this old dog.

Any idea why it's only using half of the RAM? The computer with the 2.4 kernel is using almost all of the RAM in the computer.

[Drew Lane]

Originally posted by Charles Lyons:
It is also possible that your server isn't set to overcommit (seems unlikely given you say not much RAM is in use). A JVM allocates a lot more memory than it ever uses, so that can be a problem. Here's a quote about this 2.6 kernel feature: There's more advice in this article on Linux Memory Overcommit.

Overcommit sounds like it might be whats happening.

I'm not sure how I set this and make it stick?

There is definitely some differences in memory management with the 2.6 kernel.

When I look at top, the VIRT heading show 353m for Java. The top manual says VIRT is the amount of Virtual memory for the task.

How do I tell linux not to use virtual memory for this task?

[Charles Lyons]

Any idea why it's only using half of the RAM? The computer with the 2.4 kernel is using almost all of the RAM in the computer.

It may be due to 2.6's pre-emptive swapping, where (using swappiness), it tries to swap ahead of time to avoid delays when a new process actually needs RAM.

What is the output of "cat /proc/meminfo"? That should profile your system's memory usage for us, all in one place.

BTW, what distro are you using?

[Drew Lane]

I'm using Yellow Dog Linux 4.0 on an old Mac.

cat /proc/meminfo

    
MemTotal:       543792 kB
MemFree:        277736 kB
Buffers:         12944 kB
Cached:         130212 kB
SwapCached:          0 kB
Active:         143952 kB
Inactive:       107040 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       543792 kB
LowFree:        277736 kB
SwapTotal:      458744 kB
SwapFree:       458744 kB
Dirty:             136 kB
Writeback:           0 kB
Mapped:         127144 kB
Slab:             8428 kB
CommitLimit:   1002536 kB
Committed_AS:   492840 kB
PageTables:       1144 kB
VmallocTotal:   421796 kB
VmallocUsed:     12568 kB
VmallocChunk:   408788 kB

[Charles Lyons]

MemTotal:       543792 kB
MemFree:        277736 kB
...
SwapTotal:      458744 kB
SwapFree:       458744 kB

So this is saying that you have ~530MiB RAM and 450MiB swap. None of the swap is used (it is all free). Roughly half the RAM is being used.

The "free" command should also confirm this... so the process isn't being swapped!

Top is misleading you because VSIZE includes allocated but unused pages of memory. RSS is all the pages of your process in RAM (and since you're using no swap, the entire amount of memory being used by Resin). Overcommit means VSIZE != used memory; in fact:

VSIZE = (used RAM) + (used swap) + (allocated but unused)
RES = (used RAM)

This should mean the 2.6 kernel has more memory available for other processes, whereas in your 2.4 kernel some memory is being reserved by Resin when it needn't be. Far from being less efficient, 2.6 will make your old server go further!
[ December 16, 2008: Message edited by: Charles Lyons ]

[Drew Lane]

Well, okay...

But why am I getting hard drive access every 4-5 seconds?

Also, it's taking quite a bit longer to compile java files.

It just seems to be a little less peppy overall.

[Charles Lyons]

Drive access every 5 seconds doesn't sound too much of a problem. It could be writing log files, or doing file system caching. Obviously if it's doing that on a completely idle machine, that might be a bit concerning. Does the regular drive access stop if you stop Resin? How often is your Resin server accessed and how much processing does it do? How do you know it's accessing---is it just that you can hear it, or are you using some other utility?

I'm surprised it doesn't perform equally as well---there shouldn't be any immediate reason why that is. I guess it's just a case of narrowing down what is different, like which extra processes might be running. Are the JVMs from the same provider too (e.g. both Sun, or both GCJ or what)?

[Drew Lane]

The hard drive activity mostly stops when I shut down resin. I would say it sounds normal when resin is not running. I wish I knew what it was doing because it's not in production and nothing much is running on it. I'm not using any utility to check it but I can see the green light go on every 4-5 seconds. That sucker is a loud SCSI drive too!

I suppose the other difference is that I've got resin 2.x on the other machine and resin 3.x on the one with the newer kernel.

The JVM's are from IBM and are quite good.

In fact, I just installed the Java 5 SDK so I'll try that now.

Is there any way to tell what the heck that hard drive is doing?

[Charles Lyons]

Is there any way to tell what the heck that hard drive is doing?

As with everything Linux, yep... this should do the trick:

# echo 1 > /proc/sys/vm/block_dump
[wait a bit, until after a few writes, but no longer than 15s or so]
# echo 0 > /proc/sys/vm/block_dump
# cat /proc/kmsg

If the last command doesn't work (and it may block if you have "klogd" active for example), you can get the data via "dmesg" command too. It'll log all reads and writes to your disks. You should then be able to work out what process is causing all the activity.

[Drew Lane]

Cool!

Kjournald? Journaling file system? Can I turn it off?

Also, what is dirtied inode?

    
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda7, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
ieee1394: Initialized config rom entry `ip1394'
Linux Tulip driver version 1.1.13 (May 11, 2002)
PCI: Enabling device 0000:00:0e.0 (0004 -> 0007)
tulip0:  MII transceiver #1 config 3000 status 7829 advertising 01e1.
eth1: Lite-On 82c168 PNIC rev 32 at 0xe307e000, 00:A0:CC :mrgreen: 0 :mrgreen: 6:89, IRQ 24.
ip_tables: (C) 2000-2002 Netfilter core team
Linux Tulip driver version 1.1.13 (May 11, 2002)
tulip0:  MII transceiver #1 config 1000 status 782d advertising 01e1.
eth1: Lite-On 82c168 PNIC rev 32 at 0xe307e000, 00:A0:CC :mrgreen: 0 :mrgreen: 6:89, IRQ 24.
ip_tables: (C) 2000-2002 Netfilter core team
eth1: Setting full-duplex based on MII#1 link partner capability of 41e1.
NET: Registered protocol family 10
IPv6 over IPv4 tunneling driver
Installing knfsd (copyright (C) 1996 [e-mail]).
eth1: no IPv6 routers present
java(6110): dirtied inode 958 (lib) on sda6
kjournald(839): WRITE block 29776 on sda6
kjournald(839): WRITE block 29784 on sda6
kjournald(839): WRITE block 29792 on sda6
kjournald(839): WRITE block 29800 on sda6
kjournald(839): WRITE block 29808 on sda6
kjournald(839): WRITE block 29816 on sda6
kjournald(839): WRITE block 29824 on sda6
kjournald(839): WRITE block 29832 on sda6
kjournald(839): WRITE block 29840 on sda6
java(6103): dirtied inode 801859 (webapps) on sda6
java(6111): dirtied inode 801864 (webapps) on sda6
java(6101): dirtied inode 655013 (lib) on sda6
java(6104): dirtied inode 785132 (webapps) on sda6
kjournald(839): WRITE block 29848 on sda6
kjournald(839): WRITE block 29856 on sda6
kjournald(839): WRITE block 29864 on sda6
kjournald(839): WRITE block 29872 on sda6
kjournald(839): WRITE block 29880 on sda6
kjournald(839): WRITE block 29888 on sda6
kjournald(839): WRITE block 29896 on sda6
kjournald(839): WRITE block 29904 on sda6
kjournald(839): WRITE block 29912 on sda6
java(6108): dirtied inode 801739 (webapps) on sda6
java(6103): dirtied inode 801845 (webapps) on sda6
java(6101): dirtied inode 801851 (webapps) on sda6
java(6112): dirtied inode 801873 (webapps) on sda6
java(6121): dirtied inode 784630 (lib) on sda6
java(6103): dirtied inode 801878 (webapps) on sda6
java(6102): dirtied inode 801883 (webapps) on sda6
java(6106): dirtied inode 758045 (stdout.log) on sda6
java(6106): dirtied inode 758045 (stdout.log) on sda6
java(6106): dirtied inode 757471 (lib) on sda6
java(6106): dirtied inode 757472 (web.xml) on sda6
kjournald(839): WRITE block 12239096 on sda6
kjournald(839): WRITE block 29920 on sda6
kjournald(839): WRITE block 29928 on sda6
kjournald(839): WRITE block 29936 on sda6
kjournald(839): WRITE block 29944 on sda6
kjournald(839): WRITE block 29952 on sda6
kjournald(839): WRITE block 29960 on sda6
kjournald(839): WRITE block 29968 on sda6
kjournald(839): WRITE block 29976 on sda6
kjournald(839): WRITE block 29984 on sda6
kjournald(839): WRITE block 29992 on sda6
kjournald(839): WRITE block 30000 on sda6
kjournald(839): WRITE block 30008 on sda6
kjournald(839): WRITE block 30016 on sda6
kjournald(839): WRITE block 30024 on sda6
kjournald(839): WRITE block 30032 on sda6
kjournald(839): WRITE block 30040 on sda6

[Drew Lane]

Okay, this seems to be working:

/etc/fstab

/dev/sda6 / ext3 defaults,commit=60 1 1

/etc/sysctl.conf

    
vm.dirty_ratio = 40
vm.dirty_background_ratio = 1
vm.dirty_writeback_centisecs = 30000

Let me know if you see any problems with that.

Thanks for your help!

[Charles Lyons]

Setting the commit time higher than the default (5 seconds) would seem sensible if that's what's causing the problem. I guess your other machine doesn't have a journalling system like ext3, or its commit is longer too?

[Drew Lane]

Originally posted by Charles Lyons:
Setting the commit time higher than the default (5 seconds) would seem sensible if that's what's causing the problem. I guess your other machine doesn't have a journalling system like ext3, or its commit is longer too?

Hmm...I just checked this.

The other system is using ext3 and the commit time is set to 5 seconds also but it's certainly a lot quieter.

I just ran fsck and it found some problems and dropped me into a shell, so I did it again and it came up clean.

Perhaps there was a corrupt file in one of my java apps?

Or it could be that my hard drive is going south...

However, it seems fine now and I got the performance up a bit by using the correct Apache module for Resin. Seems to be on par with the other system now.

[Drew Lane]

Just found this is dmesg:

   
program scsi_unique_id is using a deprecated SCSI ioctl, please convert it to SG_IO
program scsi_unique_id is using a deprecated SCSI ioctl, please convert it to SG_IO
program scsi_unique_id is using a deprecated SCSI ioctl, please convert it to SG_IO
program scsi_unique_id is using a deprecated SCSI ioctl, please convert it to SG_IO

What does that mean?

[Charles Lyons]

Or it could be that my hard drive is going south...

That's always a possibility, though if it has now stopped making the noises, it was more likely a corrupt block or set of blocks on the disk. Glad it's all sorted now though! I have had no real problems with the 2.6 kernels; quite the reverse, they seem to perform better for me (on reasonable spec. machines at least).

Oh, and the error you mention is nothing to worry about really (it's reasonably common). "scsi_unique_id is using a deprecated SCSI ioctl" means the scsi_unique_id program isn't the latest version, so is using an old and deprecated API. Upgrade to the latest version of the program to get rid of this message.
[ December 17, 2008: Message edited by: Charles Lyons ]

[Drew Lane]

Okay. I better leave it alone now. :-)

Thanks again!