Using streams

[jay vas]

Hi guys :

I usually write files in java by creating a byte[], and then using a bytearrayoutputstream to create the file.

Im thinking this is bad because for large files (files larger than the JVM memory allocation) this would cause an error.

My question is : What are the best practices for copying a file of considerable size (> than the JVM memory size) to disk using java ?

[Stan James]

I usually write files in java by creating a byte[], and then using a bytearrayoutputstream to create the file.

I don't follow this bit. I usually write to a ByteArrayOutputStream to capture what I'm writing into a byte array for some later use. Can you show a code snippet of what you're doing now?

[jay vas]

Sure

 
        //method to get all bytes from a file. 
public static byte[] getBytesFromFile(File f) throws Exception
{
        InputStream is = new FileInputStream(f);
    
        // Get the size of the file
        long length = f.length();
    
        if (length > Integer.MAX_VALUE) {
            // File is too large
        }
    
        // Create the byte array to hold the data
        byte[] bytes = new byte[(int)length];
    
        // Read in the bytes
        int offset = 0;
        int numRead = 0;
        while (offset < bytes.length
               && (numRead=is.read(bytes, offset,  bytes.length-offset)) >= 0) {
            offset += numRead;
        }
    
        // Ensure all the bytes have been read in
        if (offset < bytes.length) {
            throw new IOException("Could not completely read file "+f.getName());
        }
    
        // Close the input stream and return bytes
        is.close();
        return bytes;
    }
 

 
//Method to write the bytes out to a file. 
 public static File writeToFile(File f , byte[] content )
{
try
{
FileOutputStream f2 = new FileOutputStream(f);
Logln("bytes output : " + content.length);
 
f2.write(content);
 
f2.close();
return f;
}
catch(Exception e )
{
e.printStackTrace();
return null;
}
}

It seems the most intuitive way to do it for me...but Im
sure it breaks when youre file is larger than
the jvm memory.

[Stan James]

As an alternative, you can read and write a smaller chunk at a time. Set your buffer size to 1 or whatever memory will hold.

while((chunkSize= inputStream.read(buffer)) > 0)
{
   outputStream.write(buffer, 0, chunkSize);
}

If you have to work on the data rather than just copy it, things get harder. Is that the case?
[ October 06, 2006: Message edited by: Stan James ]