Unable to zip multiple large files in java using jdk1.6

[Vinod Vijay]

Hi, I'm trying to zip multiple large files which are already zipped means multiple zip files into one main(single) zip file using java utils, zipoutputstream but it is not zipping large files. My requirement is to have .zip file only. Just for testing purpose, I have zipped it into .gz using Gzoutputstream which worked fine but im not able to zip using zipoutputstream. Interestingly it is not throwing any exceptions also. It generates .zip file in the end but that .zip file is not correct because when i try to unzip it using winzip or 7zip utility on windows it throws error saying 'end of file not found' something. Total size of these multiple zip files are approximately 3GB. Is there anyway to zip such a large files using JDK1.6. Any suggestions.

FileInputStream fileInputStream = null;
		ZipOutputStream zipOS = null;
		ByteArrayInputStream in = null;
		BufferedInputStream bis = null;
		File destFolder = new File("C://Documents and Settings//dhandc//Desktop//finalzip//text1");
        File tmpDir = new File("C://Documents and Settings//dhandc//Desktop//finalzip//01Jan-31Jan2012_with Attachments");
        File[] listOfFiles = tmpDir.listFiles();	   
        FileOutputStream outputStream = null;     
               
        try {
			// create byte buffer
        	byte bytes[] = new byte[2048];  
        	if(!destFolder.exists()){
        		destFolder.mkdir(); //Create Final Zip directory if not exist
        	}
        	outputStream = new FileOutputStream(destFolder+File.separator+zipFileName+".zip");
        	zipOS = new ZipOutputStream(outputStream); 
      		if (null != listOfFiles) {
				for (int i = 0; i < listOfFiles.length; i++) {
					if (listOfFiles[i].isFile()) {
						File srcFile = listOfFiles[i];
						byte buf[] = IOUtils.toByteArray(new FileInputStream(srcFile));
						in = new ByteArrayInputStream(buf);  
						bis = new BufferedInputStream(in); 
						zipOS.putNextEntry(new ZipEntry(srcFile.getName()));						
			            int bytesRead;  
			            while ((bytesRead = bis.read(bytes)) != -1) {  
			                zipOS.write(bytes, 0, bytesRead);  
			            }  
			            zipOS.closeEntry(); 			            
			            bis.close();
					}					
				}
			}
      		zipOS.close();
			logger.debug("SUCCESSFULLY ZIPPED");
	    } catch (FileNotFoundException e1) {
			e1.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		} catch (Exception e) {
			e.printStackTrace();
		}finally {
			try {
				zipOS.close();
				
			} catch (Exception e) {
				e.printStackTrace();
			}
		}

It is not printing logger SUCCESSFULLY ZIPPED.

[Ulf Dittmer]

Well, what is it printing instead?

It also sounds like you might be running into a 32bit/64bit issue, so be sure to run the 64bit JVM.

Also, the code looks rather complicated. For example, what do you need the ByteArrayInputStream and BufferedInputStream for? You already have the data to be written in the buf[] - which you can write directly to the ZipOutputStream.

[Paul Clapham]

If I recall correctly, the original PKZIP specification could only handle files which were no greater than 2 MB in size. (The size of the file was stored in a fixed-size field which couldn't store numbers larger than that.) By the time 2-megabyte-files started to appear, Phil Katz didn't have control of the spec any more and various actors implemented various workarounds for that issue. But Java's zip-archive classes continued to implement the original spec rather than any of the extensions. Whether that's changed or not, I don't know.

[Vinod Vijay]

Ulf Dittmer wrote:Well, what is it printing instead?

It also sounds like you might be running into a 32bit/64bit issue, so be sure to run the 64bit JVM.

Also, the code looks rather complicated. For example, what do you need the ByteArrayInputStream and BufferedInputStream for? You already have the data to be written in the buf[] - which you can write directly to the ZipOutputStream.

Hi Dittmer, that was my second approach to make the things work.


My original code is like this only:-

FileInputStream fileInputStream = null;
			FileOutputStream fileOutputStream = null;
	        ZipOutputStream zipOutputStream = null;
	        File destFolder = new File(ccms_properties.getProperty(LOCAL_DOCUMENTUM_FINAL_ZIP_DIR));
	        GZIPOutputStream
	        droppingDir(ccms_properties.getProperty(LOCAL_DOCUMENTUM_TMP_DIR)); //Clean Up Documentum TMP
	        File srcDir = new File(ccms_properties.getProperty(LOCAL_DOCUMENTUM_ZIP_DIR));
	        File tmpDir = new File(ccms_properties.getProperty(LOCAL_DOCUMENTUM_TMP_DIR));
	        
	        try {
	        	FileUtils.moveToDirectory(srcDir, tmpDir, true); //Copying All Zipped attachments to Tmp directory
	        } catch (IOException e) {
	        	e.printStackTrace();
	        }
	        
	        File[] listOfFiles = new File(tmpDir+File.separator+"zip").listFiles();	     
	        
	        try {
				// create byte buffer
	        	byte[] buffer = new byte[2048];
	        	if(!destFolder.exists()){
	        		destFolder.mkdir(); //Create Final Zip directory if not exist
	        	}
				fileOutputStream = new FileOutputStream(destFolder+File.separator+zipFileName+XMLConstants.ZIP_EXTENSION);
				zipOutputStream = new ZipOutputStream(fileOutputStream);
				if (null != listOfFiles) {
					logger.debug("@@@Total Files to be Zipped "+listOfFiles.length);
					for (int i = 0; i < listOfFiles.length; i++) {
						if (listOfFiles[i].isFile()) {
							File srcFile = listOfFiles[i];
							fileInputStream = new FileInputStream(srcFile);
							zipOutputStream.putNextEntry(new ZipEntry(srcFile.getName()));
							int length;
							while ((length = fileInputStream.read(buffer)) > 0) {
							 zipOutputStream.write(buffer, 0, length);
							}
							zipOutputStream.closeEntry();
							fileInputStream.close();
						}
						
					}
				}
				zipOutputStream.close();	
				logger.debug("@@@@@@@@@@@@@ Zipped Successfully @@@@@@@@@@@@@@@@");
		    } catch (FileNotFoundException e1) {
				e1.printStackTrace();
			} catch (IOException e) {
				e.printStackTrace();
			}catch (Exception e) {
				e.printStackTrace();
			}  finally {
				try {
					logger.debug("@@@@@@@@@@@@@ Closing zipOutStream @@@@@@@@@@@@@@@");
					zipOutputStream.close();
					fileInputStream.close();
				} catch (Exception e) {
					e.printStackTrace();
				}
			}

Now can you please suggest me if it is JVM 32/64 bit issue or something else? It is not printing the logger "@@@@@@@@@@@@@ Zipped Successfully @@@@@@@@@@@@@@@@", going to finally block, closing all the streams. In the end, .zip is there but as I said, it is not complete zip (Max size of final zip is 1GB). I'm not able to unzip it. Looks like after certain writing to zip file, it is skipping, incomplete only. is there any size limit issue of creating a zip file? Thanks

[Paul Clapham]

Vinod Vijay wrote:is there any size limit issue of creating a zip file? Thanks

I thought I mentioned that in my post. Perhaps you overlooked it?

[Vinod Vijay]

Paul Clapham wrote:

Vinod Vijay wrote:is there any size limit issue of creating a zip file? Thanks

I thought I mentioned that in my post. Perhaps you overlooked it?

Paul, I have checked that. JVM is 64bit only. Restriction is there as this program is running on production.

[Ulf Dittmer]

Vinod Vijay wrote:Paul, I have checked that. JVM is 64bit only.

What exactly have you checked? Paul didn't say anything about 32bit/64bit JVMs. He was talking about zip file sizes, and potential limits inherent to the ZIP format.

I doubt that there is such a limit, though, at least not one as low as 2MB. But you can -and should- test that. You should also test whether this works with the same number of files, but if they are all small in size.

[Jaikiran Pai]

This:

                        byte buf[] = IOUtils.toByteArray(new FileInputStream(srcFile)); 

can potentially cause a OutOfMemoryError (note that it's an java.lang.Error) and won't be caught in any of the catch blocks which catch java.lang.Exception types. Perhaps you are indeed running into an OutOfMemoryError? To test it out, you can change the catch block to:

catch (Throwable t) {
  System.out.println("Caught a throwable: " + t.getMessage());
  t.printStackTrace();
}

[Vinod Vijay]

Ulf Dittmer wrote:

Vinod Vijay wrote:Paul, I have checked that. JVM is 64bit only.

What exactly have you checked? Paul didn't say anything about 32bit/64bit JVMs. He was talking about zip file sizes, and potential limits inherent to the ZIP format.

I doubt that there is such a limit, though, at least not one as low as 2MB. But you can -and should- test that. You should also test whether this works with the same number of files, but if they are all small in size.

With small number of files, it works fine but not with huge files with large size