Win a copy of The Java Performance Companion this week in the Performance forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Split and Merge large zip files

 
Vinod Borole
Greenhorn
Posts: 26
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello

Can anyone tel me how can i split and then merge large zip files using java?

Thank you!
 
Greg Charles
Sheriff
Posts: 2987
12
Firefox Browser IntelliJ IDE Java Mac Ruby
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
By split you mean unzip, and by merge you mean zip everything into one archive? Check into the java.util.zip package.
 
Vinod Borole
Greenhorn
Posts: 26
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have a .NET application that reads a 1GB size zip file and breaks it into 5 parts of each 200MB, one by one each part is sent to my java application in a multipart request. My java applications downloads each part and store in a location. Now as i have all the parts, i need to merge them together into a 1GB zip file as it was original.
I am not finding a way how to merge them, Can you please suggest?

Note: The .NET application was throwing Out of memory exception when it tried sending a 1GB zip directly, the maximum it could send was 200MB.
 
Greg Charles
Sheriff
Posts: 2987
12
Firefox Browser IntelliJ IDE Java Mac Ruby
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ah, so it's slicing apart a large file into smaller chunks. In that case, it doesn't really matter that it was a zip file to start with. You need a FileInputStream to read the five files in order, and a FileOutputStream to output them to a single reassembled file. For each file, read some bytes into a byte array (4096 is probably a good size to use) then write that byte array to the assembled file. When you run out of bytes in one file, open the next one and keep going.
 
Rob Spoor
Sheriff
Pie
Posts: 20550
57
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
That will work if the file is split in the simple "let's cut here and continue with the next one" way. Other splitting / merging methods may put some metadata in the split files.
 
Paul Clapham
Sheriff
Posts: 21133
32
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Greg Charles wrote:For each file, read some bytes into a byte array (4096 is probably a good size to use) then write that byte array to the assembled file. When you run out of bytes in one file, open the next one and keep going.
A SequenceInputStream might help with this process.
 
Vinod Borole
Greenhorn
Posts: 26
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have tried a sample project where I have broken down 67MB zip file into 2 parts of one of 50MB and the other of 17MB. It's taking me more than 30mins to merge them together with the below code.
the zip file after merging that i get is of perfect size, but when i extract it its saying corrupted.

Can any tel me what am i doing wrong? Also, if anyone can suggest me fastest way of doing the same work, as its taking long time to merge with this code.



 
Greg Charles
Sheriff
Posts: 2987
12
Firefox Browser IntelliJ IDE Java Mac Ruby
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Don't use readLine(). That's for reading text, and may be changing the line endings. You don't need anything fancy. Even the DataInput/OutputStreams seem unnecessary, and you will do your own buffering so the the Buffered Streams are overkill as well. Create an array of bytes, read bytes from the input into the array, from the return value figure out how many bytes you read, and write them to the output. It shouldn't take anywhere near that long for 67 megs.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic