Win a copy of The Java Performance Companion this week in the Performance forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

is it possible to process a 500MB zipfile in java?

 
pavi
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
In my project i need to split a zipfile of any size into 1MB files and then merge it to get the original file.My java code works fine upto 58MB zipfile but for the file size greater than 58MB i get the following exception

"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"

Is it possible to process a 500MB zipfile in java?
If possible how to process.
please help me to resolve this.

Thanks in advance,
Latha
 
Rob Spoor
Sheriff
Pie
Posts: 20552
57
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You can use the -Xmx flag; type in "java -X" in a command window for more info.

Moving to the beginner's forum.
 
Joe Ess
Bartender
Posts: 9300
10
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Don't try to read the entire file into memory.
Read a small chunk, write it out and repeat.
 
pavi
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
i read that The -X options are non-standard and subject to change without notice.

below is a code to find the memory size

System.out.println("MAX MEMORY = " + Runtime.getRuntime().maxMemory() / 1048576 + "MB");
System.out.println("TOTAL MEMORY = " + Runtime.getRuntime().totalMemory() / 1048576 + "MB");


when i run this it prints 63MB as MAX MEMORY and 4MB as total.

and after the below command

java -Xms512m -Xmx512m MyApplication

it prints 512MB.

but again if i just give

java Application
i get the memory size as only 63MB

what does it means?so for every time we run the file should we set the size or what? how to set the size permanantly for the application to handle bigfile?please do help me .

Thanks in advance
Latha
 
Paul Sturrock
Bartender
Posts: 10336
Eclipse IDE Hibernate Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

i read that The -X options are non-standard and subject to change without notice.

Indeed. If sun release a new JVM and the rename these to "minimumHeapSize" and "maximumHeapSize" you will need to change how you run your process. Is that an issue?
 
Henry Wong
author
Marshal
Pie
Posts: 21214
81
C++ Chrome Eclipse IDE Firefox Browser Java jQuery Linux VI Editor Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
what does it means?so for every time we run the file should we set the size or what? how to set the size permanantly for the application to handle bigfile?please do help me .


For most bigger applications, there are generally a ton more switches than settings than the memory size. And generally, they are done with a batch file (shellscript).

Henry
 
pavi
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Henry

I created a batch file and is working fine.

now i want to know is there is any maximum size that we can specify or the size can reach upto 1GB or more.do the size depends on the ram size of our system?

thanks in advance
Latha.s
 
Joe Ess
Bartender
Posts: 9300
10
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It depends on the memory available.
Of course, if you follow my suggestion and don't read the whole file into memory, you will not have to worry about memory.
 
Vlado Zajac
Ranch Hand
Posts: 245
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There is definitely a maximum. Memory used by JVM cannot be higher than what the OS allows to allocate.

Heap size should also be somewhat smaller than total RAM since otherwise garbage collection will cause trashing (slowing down entire system very badly).

It is possible to split arbitrary large file without need for large memory.
 
pavi
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Don't try to read the entire file into memory.
Read a small chunk, write it out and repeat.


Hi joe

i didnt get your concept.actually i need to get the file length and then split that file into 100kb file and pass it to queue and from queue another program collects that files and merge all that splitted files into one file to get the original file size.here how to do as you have told.
do help me if you have any suggestions.
thanks in advance
Latha
 
Joe Ess
Bartender
Posts: 9300
10
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by latha selva:
actually i need to get the file length

java.io.File has a method for that.


and then split that file into 100kb file

read 100kb and write it to a file. Do that over and over until the input stream ends.


and pass it to queue and from queue another program collects that files

Presumably you are doing this now with the files you are creating. Nothing would change.


and merge all that splitted files into one file to get the original file size.

I'm unclear as to whether your program is merging the files, or if the other program is. If your program has to do it, it is simply the reverse of splitting the files. Open an output stream and read each of your 100kb inputs in turn.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic