• Post Reply Bookmark Topic Watch Topic
  • New Topic

Corrupt GZIP trailer  RSS feed

 
bob connolly
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello this seems to work fine for smaller files, ie 10 million or so, but when i try to create a 40+ million file i get that Corrup GZIP trailer!

I've tried many combinations of .close, .finish, .flush ect, but for some reason, i quess it's not writing the last buffer of data!

Thanks very much for an insights!



[code]
try
{
textReader = new BufferedReader(new FileReader(xp.dat_file));

GZIPOutputStream gz = new GZIPOutputStream(new FileOutputStream(xp.xml_file));
PrintWriter outf = new PrintWriter(gz);

String line = null; String sToken = null;

String hdr1=("<?xml version=#1.0# encoding=#UTF-8#?>"); hdr1=hdr1.replace('#','"');

outf.write(hdr1+"\n");
outf.write(" <hlp_data>"+"\n");

int pcnt = 0;
while ( (line = textReader.readLine()) != null )
{
pcnt++; debug_cnt++;
if (line.length() < 5) { System.out.println("File has bad data"); break;}

String[] sa = new String[NUM_COLS];
int previ=0; int t=0;int fcnt=0;

for (int s=0;s<line.length();s++)
{
if (line.charAt(s)=='|')
{
sa[t]=line.substring(previ,s);fcnt++;
t++; previ=s+1;
}
}
sa[t]=line.substring(previ,line.length());fcnt++;

if (NUM_COLS!=fcnt) {throw new NullPointerException("ddl cnt != field cnt: "+NUM_COLS+" != fieldcnt:

"+fcnt);}
outf.write(" <"+FILE_NAME+">"+"\n");

if (NUM_COLS==fcnt)
{

for (int i = 0; i < NUM_COLS; i++)
{
if (i>(sa.length-1)) { sToken = new String(); } else { sToken = sa[i]; }

if (sToken.indexOf('<')>-1 || sToken.indexOf('>')>-1 || sToken.indexOf('"')>-1||

sToken.indexOf('&')>-1 )
{
String sToken2 = new String();
sToken2 = normalize(sToken);
sToken = sToken2;
}
outf.write(" <"+_columnNames[i]+">"+sToken+"</"+_columnNames[i]+">"+"\n");
}

}
outf.write(" </"+FILE_NAME+">"+"\n");
}

outf.write("</hlp_data>"+"\n");

textReader.close();
gz.finish();
gz.close();
outf.flush();
outf.close();
}
catch (IOException e) {System.out.println("HlpValPipExtXml1/Error2/reading dat_file" + e);}
}

[\code]
 
Serge de Schaetzen
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Bob,

I don't know what file system you're using, but it sounds like ntfs.
And you can't create a zip file greater than 4gb in Windows !

I had a similar problem and I had to change to code so that it would switch to another zip file (z00, z01, etc...) when a certain limit was reached.
 
bob connolly
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks very much Serge, i'll be looking into that!

Have a nice week!
 
bob connolly
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi again Ernest!

I think there might be a 'bug' in the GZIPInputStream pgm, because when i use the following override fix program from Sanger labs, my SAX parse validation works fine!

And actually, the GZIPInputStream works fine for files under 300million or so, but over that it runs into the 'Corrupt Trailer' message!

By the way, I got this override pgm from one of the references in the previous post!

//package uk.ac.sanger.artemis.util;

import java.io.*;

/**
* A wrapper class to work around a bug in GZIPInputStream. The read()
* method sometimes throws a IOException complaining about a Corrupt GZIP
* trailer on the jdk 1.1.8-13 on the Alphas. This wrapper catches and
* ignores that exception.
* @author Kim Rutherford <kmr@sanger.ac.uk>
* @version $Id: WorkingGZIPInputStream.java,v 1.1 2004/06/09 09:53:17 tjc Exp $
**/

public class WorkingGZIPInputStream extends java.util.zip.GZIPInputStream {
/**
* Creates a new input stream with the specified buffer size.
* @param in the input stream
* @param size the input buffer size
* @exception IOException if an I/O error has occurred
**/

public WorkingGZIPInputStream (InputStream in, int size)
throws IOException {
super (in, size);
}

/**
* Creates a new input stream with a default buffer size.
* @param in the input stream
* @exception IOException if an I/O error has occurred
**/
public WorkingGZIPInputStream (InputStream in)
throws IOException {
super (in);
}

/**
* Calls super.read() and then catch and ignore any IOExceptions that
* mention "Corrupt GZIP trailer".
**/
public int read (byte buf[], int off, int len)
throws IOException {
try {
return super.read (buf, off, len);
} catch (IOException e) {
if (e.getMessage ().indexOf ("Corrupt GZIP trailer") != -1) {
return -1;
} else {
throw e;
}
}
}
}

Thanks again Ernest!
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!