Reading Large Files

[Richard West]

Hi everyone,

I am trying to to read and write large files about 600M.
The thing is when i use the normal FileOutputStream methods an exception gets thrown saying

java.lang.OutofMemory

This is how i am trying to read the file

FileOutputStream out = new FileOutputStream("C:/my_life_story.zip");
InputStream in = //an input stream from a socket, etc 
 
int len = 0;
byte[] buffer1 = new byte[1024]; 
 
while ((len = in.read(buffer1)) > 0)
{ 
out.write(buffer1, 0, len);
}
 
in.close(); 
out.close();

My computer does have sufficient memory

I tried adding this in the while loop but it does not work and i got the same exception thrown by the JVM again

System.gc();

My question is basically how do i increase the memory programatically
or is there away in which i can read the file without having the above exception thrown by the JVM?

Any help i greatly appreciated

Thank You

Yours Sincerely

Richard West

[Ernest Friedman-Hill]

This code will not cause your program to run out of memory -- the problem lies elsewhere. Is this your real code, or a simplification?

[Richard Anderson]

Are you passing the min and max heap size arguments to the JVM?

You can do that by....

java -Xms256M -Xmx1G MYCLASS

where -Xms256M allocates 256MB for the mininum heap size and -Xmx1G allocates 1G for the max heap size. I believe by default the JVM sets the max heap size at 256M. Can someone verify this?

Sorry....looks like the default is only 64M.
[ August 11, 2005: Message edited by: Richard Anderson ]

[Rick O'Shay]

Couple points and then a solution. First, there are approximiately zero reasons to run gc() manually. Second, there are few reasons to read 600 megabytes in to memory in one fell swoop. Use a memory mapped file.

http://javaalmanac.com/egs/java.nio/CreateMemMap.html

P.S. Make sure you set your memory switches and usually it's a good idea to clamp both ends so that re-sizing at runtime is not required: -Xms256m -Xmx256m. A quarter gig of RAM should be enough to run the Goggle mainframe at peak hours. Well, maybe exaggerating but it's a heckuvalotta memory.

[Stuart Gray]

Sorry to hijack, but the impression I get from the posts here is that this code will try to read all 600MB of data into memory at once.

My understanding was that it would read the buffer size (1K) of data at once, write it out, and then read another 1K, and so on. Have I misunderstood something?

[Richard West]

Hi everyone,

Ernest to answer your question, its a simplification. Don't mind if i ask you but what makes you say it would not run out of memory?

Anderson this what you mean

java -Xmx650m MyProgram

Unfortunately i can't use the above code because the thing is the application is in a jar file on windows platform and users that use this application to run it usually double click on the jar file to run it. Is there a way to do what you suggested programatically maybe by the use of properties?

Stuart, you have understood the question correctly

Hoping to hear from you guys

Richard West

[Norm Radder]

See posts on this topic on: http://forums.devx.com/forumdisplay.php?f=104

[Ernest Friedman-Hill]

Originally posted by West Richard:
Ernest to answer your question, its a simplification. Don't mind if i ask you but what makes you say it would not run out of memory?

Because the code you showed us uses a small, fixed amount of memory. There is nothing here with memory usage that will grow over time. This loop could copy the biggest file your OS could support, and it could do it with -Xmx8m -Xms8m on the command line, because it hardly uses any memory at all.

As a result, the only suggestions you're getting are to up your VM image size, and that's not helping you, right? If you showed us more code, maybe we'd spot where the memory was going, and we could help you fix the real problem.

[Richard West]

Hi everyone,

I tried searching somewhere else in the program and you were right Ernest the leak was not coming from the loop but it seemed to come from this part

FileOutputStream fStream = new FileOutputStream(SF);
ObjectOutput stream = new ObjectOutputStream(fStream);
 
//The memory exception occurs at the below command line
 
stream.writeObject(TextPane1.getDocument());
 
stream.flush();
stream.close();
fStream.close();

I know for a fact that the document in the JTextPane spans couple of of thousand pages but it seems that the exception is coming from that area.

Could it be that because the stream is writing the document at one go. If it is then is there a better way to do it?

Richard West