• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Bear Bibeault
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
Sheriffs:
  • Tim Cooke
  • Liutauras Vilda
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • fred rosenberger
  • salvin francis
Bartenders:
  • Piet Souris
  • Frits Walraven
  • Carey Brown

Memory Leak using System.arraycopy?

 
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator


Using the above code in a web-app where we upload a file a chunk at a time to the server. I would think that once all the objects are out of scope, and cleaned up, that the memory would be reclaimed, but that doesn't seem to be the case.

On a windows service, I can watch the 'Mem Usage' in task manager and see it jump after this arraycopy method is called. It goes up by the size of the byte[] being copied.

After all is said and done, the msgData is nulled out, but this memory does not get reclaimed.

Any ideas what may be happening here?

thanks in advance,
Tom
 
Bartender
Posts: 9615
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator


This is a red flag. Nulling the reference to msgDataChunk doesn't work because msgDataChunk is a copy of the reference in the caller. The caller would still have a valid reference and the array would not get garbage collected.
 
Tom Katz
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks. I'll try taking that out, and see how it goes. I"m pretty sure I added that recently in order to *hopefully* (tho apparently not, per your comments) help out in this situation - so I'm not sure it's the actual cause.
 
Tom Katz
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Taking out that finally clause still results in the jump in memory - tho I'm going to let things run awhile and see if the GC takes care of it. I've done that in the past, let it sit, and the mem usage stays at the same level.

Below is the calling code. setMsgDataChunk was originally being passed the readObject call (cast as byte[]), but I split this up a bit in order to narrow down what was happening.



Shouldn't 'data' be eligible for collection once we're out of the scope of the 'if(rv)' clause? Maybe I'm misunderstanding something basic/obvious, but not sure what it is yet...
 
Sheriff
Posts: 21972
106
Eclipse IDE Spring VI Editor Chrome Java Ubuntu Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
When do you ever clear your "this.msgData" variable? If this keeps being filled, then of course your memory usage will increase.
 
Tom Katz
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Of course, indeed.

Once all the chunks have been sent from the applet, and we do what we want to do with the data, this.msgData is nulled out (I tried setting it to new byte[0] too for good measure). I have breakpoints in this part of the code, and I see them getting hit once everything's complete.

BTW: It's an instance variable of a session object.
 
Joe Ess
Bartender
Posts: 9615
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Do you have an object output stream in the same code? ObjectOutputStream caches references to objects written to it. This could be the cause of your memory leak. See this article (scroll down to "Caching Objects in the Stream").
 
Tom Katz
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks.. Below is the applet code that sends the data.



I am not sure why getInputStream is called there - we don't do anything with it (and no comments to explain why, of course - grrr) - could that be part of the problem. We are flushing/closing/nulling out the ObjectOutputStream. (But maybe reset is needed, per the article linked to above)
 
Joe Ess
Bartender
Posts: 9615
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator


I am not sure why getInputStream is called there - we don't do anything with it (and no comments to explain why, of course - grrr) - could that be part of the problem.



HTTP is a request-response protocol. It is a good idea to read the response (the client "input") because your web server expects you to. If your web server is sitting on dozens of requests because the responses haven't been read, that would be a problem as well.
 
Tom Katz
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Right you are! and we do do that in other applet<->servlet calls we make. When I took that line out, my breakpoints in the servlet weren't even being hit.

Going to try calling reset on the ObjectOutputStream, see if that helps any.

Thanks all for the info/tips!
 
Tom Katz
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Still the same issue after calling reset. I did replace the flush call with the reset (seemed like good idea after reading that article)

Anything else that I should be doing with the ObjectInputStream? I"m already closing/nulling it out.
 
Joe Ess
Bartender
Posts: 9615
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Originally posted by Tom Katz:
[CODE]
On a windows service, I can watch the 'Mem Usage' in task manager and see it jump after this arraycopy method is called. It goes up by the size of the byte[] being copied.



I just thought of something. Do you expect the memory to be reclaimed by Windows? The last time I checked, and it has been years, the VM will never surrender allocated memory back to the OS. The memory will be free for reuse by the VM but not other applications.
 
Tom Katz
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Maybe I should clarify: I was watching java.exe in task manager.

And no, I don't expect windows to reclaim the memory. What I expected was the Mem Usage for java.exe to go back down once the huge file/byte[] was done being processed.
 
Tom Katz
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So, are you saying that once memory is used by java.exe/JVM (are they the same thing?) that it will hold onto that memory until it's shut down?
 
Joe Ess
Bartender
Posts: 9615
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Originally posted by Tom Katz:
So, are you saying that once memory is used by java.exe/JVM (are they the same thing?) that it will hold onto that memory until it's shut down?



I believe this to be the case (I haven't looked into it for years). The Windows task manager will only give you the OS view of the application (Java VM), so the memory use reported in it will not go down.
If you want to know about the memory within the VM, look at java.util.Runtime which has several methods to gauge memory use.
jconsole , included with the JDK, can also be of use in examining the workings of the VM.
You really don't have a memory leak unless the VM grows without limit, eventually getting an OutOfMemoryError. A well-behaved program will grow somewhat until it reaches a stable state where it has enough internal free memory to handle whatever processing is going on. The default VM size is 2MB and the default max is 64, so there's some room to grow.
 
Tom Katz
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Great, thanks for all the help!

I think that'll probably be loads better with which to monitor this, rather than watching the bobbing/weaving rows in task manager!
 
Joe Ess
Bartender
Posts: 9615
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

. . .rather than watching the bobbing/weaving rows in task manager!



You can sort the rows in Task Manager by clicking on the column headers (Image Name, User Name, CPU, Mem Usage, VM Size). If you click on "Image Name", your rows won't change position as the other values change.
Not that I've ever spent hours monitoring misbehaving programs or anything
 
Ranch Hand
Posts: 1970
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
While jconsole is probably the best thing to use, there is something you should know about Windows Task Manager.

The column Mem Usage is one of the columns shown by default. However, this is the amount of physical RAM in use, which is not what you want to know. The physical RAM in use depends on all sorts of semi-random things, only one of which is your heap size. You want the VM Size column, which is not shown by default but can be added by the View menu (IIRC). This is not a small point - it's really important (if you are using Task Manager).

Also, people have said on this thread that Java never gives back memory to Windows. I can say for sure that this is untrue. It does give back memory. Or, at least, it can be told to do so. There is a threshold of heap percentage emptiness at which the heap will be shrunk and memory returned to Windows. By default, this is rather low, but you can change it so memory is returned more aggressively; this may not always be wise, but it is an available option. There is a similar setting for the percentage fullness at which the heap will be expanded.

IIRC, the settings are -XXMinHeapFreeRatio and -XXMaxHeapFreeRatio. Both are between 0 and 100 exclusive.
 
Rob Spoor
Sheriff
Posts: 21972
106
Eclipse IDE Spring VI Editor Chrome Java Ubuntu Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Question: why are you using ObjectOutputStream (on the other side) and ObjectInputStream to send the data in byte arrays?

OutputStream and InputStream themselves can handle byte arrays just perfectly.
 
Tom Katz
Ranch Hand
Posts: 169
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
That's a good question, ill look into that, thanks.

Is there harm in doing it that way, or is it just unnecessarily complicating things?
 
Joe Ess
Bartender
Posts: 9615
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
That was a good catch by Rob, by the way.
There's no real harm, but there are non-trivial costs to using object serialization. If you don't have to do it, don't.
 
Don't get me started about those stupid light bulbs.
    Bookmark Topic Watch Topic
  • New Topic