OutOfMemoryError when accessing Zip File

[Garret Schweitzer]

Hello all,

I get an OutOfMemoryError when accessing large ZipEntry(ies) in a ZIP file (using java.util.zip). It happens when trying to get the input stream. I can go through small entries and grab the input streams, and inflate the files inside, but if I hit a large file, it blows up.

Example:
... code before this....

if (!entry.isDirectory())
{
try
{
**************** blows up on this next line ******************
InputStream tempInputStream = zipFile.getInputStream(entry);
BufferedInputStream is = new BufferedInputStream(tempInputStream);

int currentByte;
// establish buffer for writing file
byte data[] = new byte[BUFFER];

// write the current file to disk
FileOutputStream fos = new FileOutputStream(destFile);
BufferedOutputStream dest = new BufferedOutputStream(fos, BUFFER);

// read and write until last byte is encountered
long totalBytes = 0;
long lastPrinted = 0;
Date startTime = new Date();
while ((currentByte = is.read(data, 0, BUFFER)) != -1)
{
totalBytes += BUFFER;
if ((totalBytes) > (lastPrinted + (ONE_HUNDRED_K * 3)))
{
System.out.println("extracted... " + totalBytes);
lastPrinted = totalBytes;
}
dest.write(data, 0, currentByte);
}
dest.flush();
dest.close();
is.close();
... more code follows this ...


I'm guessing that the getInputStream() must read the entire file into memory somehow, but I'm not a Java Guru. I don't know how I can unzip large files since the getInputStream() method doesn't take buffer sizes as arguments.

Any ideas would be greatly appreciated.

Thanks,
Garret

[Ramender Mall]

Hi,

InputStream tempInputStream = zipFile.getInputStream(entry);

My theory is, that getInputStream() will return a "stream", not data, so there shud not be a question of buffer at least here..as a stream consumes or delivers data...which YOU do using read() and write()...

hmmm,
sort of a no use reply...but HOPE it helps ..

Ramy....
[ December 28, 2005: Message edited by: Ramender Mall ]

[Garret Schweitzer]

Just to clarify that post (which I did late last night), here's a more thoughtful version....







I have a Java issue and I don't know how to solve it, so I thought I'd run it by the folks here and see if anyone here knows what to make of it.

First: The (simplified) code below works fine on a PC, but not in a limited memory environment, like a PDA.

Second: I have no control over the Java environment, so I can't control startup switches. In fact, I'll be running in a WebSphere environment. This code will be running in a J2ME environment as well as a J2SE and J2EE environment, so it needs to be friendly.

The problem: It runs out of memory when inflating a large file (greater than ten megs) from a ZIP. If I have 14 files in the ZIP and two are larger than ten megs, it will inflate all but the two large ones.

Where it does NOT occur: Strangely enough, it doesn't happen at a controllable place, like a buffered read. It isn't happening at any place where I can control the number of bytes read.

Where it DOES occur: It's happening at the line where I try to get the input stream getInputStream() for an entry in the ZIP file. Specifically, the large (greater than ten megs) entry in the file.

Why it happens: I have some theories. 1) It may be standard practice to read the entire file during a getInputStream() method call. 2) In order for the inflator to determine strategy, it may have to read the entire file. 3) The blocksize for the stream is determined in some way by the size of the file.

What you can do to help me: If you know the answer, great, let me know. I need to finish this project. ALSO, tell me how I can find this stuff out on my own. If there is some method of seeing the source code behind getInputStream() in java.util.zip.ZipFile, that might lend some clues. Perhaps looking at the bytecode would be handy...if I knew how to intrepret it. If there is a workaround that is more memory-friendly, that would be outstanding. I'm easy.

I'm going to post code at the end of this message.

Thanks in advance for help.

Garret

  
static final int BUFFER = 4096;
 
 public void extract()
 {
  try
  {
 
   File sourceZipFile = new File(zipFile);
   File unzipDestDir = new File(destDir);
 
   // Open Zip file for reading
   ZipFile zipFile = new ZipFile(sourceZipFile, ZipFile.OPEN_READ);
 
   // Create an enumeration of the entries in the zip file
   Enumeration zipFileEntries = zipFile.entries();
 
   // Process each entry
   while (zipFileEntries.hasMoreElements())
   {
    // grab a zip file entry
    ZipEntry entry = (ZipEntry) zipFileEntries.nextElement();
 
    String currentEntry = entry.getName();
    int lastBackslash = currentEntry.lastIndexOf("\\");
    if (lastBackslash >= 0)
     currentEntry = currentEntry.substring(lastBackslash + 1);
 
    File destFile = new File(unzipDestDir, currentEntry);
 
    // grab file's parent directory structure
    File destinationParent = destFile.getParentFile();
 
    // create the parent directory structure if needed
    destinationParent.mkdirs();
 
    // extract file if not a directory
    if (!entry.isDirectory())
    {
     try
     {
---------------------> OutOfMemoryError occurs next line <----------------------
      InputStream tempInputStream = zipFile.getInputStream(entry);
--------------------------------------------------------------------------------
      BufferedInputStream is = new BufferedInputStream(tempInputStream);
 
      int currentByte;
      // establish buffer for writing file
      byte data[] = new byte[BUFFER];
 
      // write the current file to disk
      FileOutputStream fos = new FileOutputStream(destFile);
      BufferedOutputStream dest = new BufferedOutputStream(fos, BUFFER);
 
      // read and write until last byte is encountered
      while ((currentByte = is.read(data, 0, BUFFER)) != -1)
       dest.write(data, 0, currentByte);
 
      dest.flush();
      dest.close();
      is.close();
     }
     catch (OutOfMemoryError outofmemory)
     {
      System.out.println("Out of memory trying to unzip!");
      System.out.println("Error message: " + outofmemory.getMessage());
      outofmemory.printStackTrace();            
     }
    }
   }
   zipFile.close();
  }
  catch (IOException ioe)
  {
   ioe.printStackTrace();
  }
 }

[Ramender Mall]

well,
matter seems to be way out of my sight...

But for source code, try this post...
it might help......
hopefully, atleast a bit more than the last reply...

Ramy...
[ December 28, 2005: Message edited by: Ramender Mall ]

[Alexis Krenvalks]

For work with zip files I usually use-file recovery zip,because this tool is free as far as I know and helped me many times.Moreover program can too extracts data and saves it to the hard disk while saving folders and files.

[Tim Holloway]

Alexis,

Thanks for trying to be helpful, but http://faq.javaranch.com/java/DontWakeTheZombies