JAVA HEAP SIZE and files to byte[]-Array

[nimo frey]

The original issue was created in http://www.coderanch.com/t/448993/Performance/java/JAVA-HEAP-SIZE-files-byte, but I want to move this issue here, as it s more I/O-related:

I have a file which should be converted into a byte[]-Array.

All works fine. But when the file is larger than the maximum JAVA HEAP SIZE than this error occurs

Caused by: java.lang.OutOfMemoryError: Java heap space

I use the Apaches Commons "FileUtils.readFileToByteArray()"-Method to convert from File to byte[].

Also, the Apaches "IOUtils.toByteArray(new FileInputStream(myfile)))" returns a java.lang.OutOfMemoryError.


I know, I have to set the "Java heap space" to a higher value, but this is not always possible.

Is there a way to handle such big files without exceeding the "Java heap space".

I also tried this http://balusc.blogspot.com/2007/07/fileservlet.html, but without success.

I also tried it this way:

BufferedInputStream input = new BufferedInputStream(new FileInputStream(myBigFile, 10240);
ByteArrayOutputStream output = new ByteArrayOutputStream();

            byte[] buffer = new byte[20000*1024];
            
            int length;
            while ((length = input.read(buffer)) > 0) 
            {
            	output.write(buffer, 0, length);
            }

            output.flush();
            output.close();
            input.close();

but java.lang.OutOfMemoryError occurs.

Is there a way, maybe with the new NIO-API to handle such scenarios in a performant way? java.nio offers ByteChannel and FileChannel, maybe converting from FileChannel to ByteChannel, but how?

[William Brogden]

Are you expecting some sort of magic to take place? You want a byte[] in memory that is bigger than available memory??

Exactly what do you need to do? Most people would use a random access approach to look at byte contents of a file too big for memory. What operations do you have to perform on this large file?

Bill

[nimo frey]

I get a uploaded file and want to store that file into a database as blob.
So I need to convert this file as a byte[].

I guess, you are right - it is not possible without extending the heap-size.

So this is not a solution? http://forums.sun.com/thread.jspa?threadID=5378438

This could be a solution:
To read the stream (file) not completly, but in for example, 4 MB-blocks with a BufferedReader.

Would be fine to have this solution with java.nio:
get a FileChannel of that file and convert it to a ByteChannel and from ByteChannel to byte[]. Is this possible?

[Rob Spoor]

nimo frey wrote:I get a uploaded file and want to store that file into a database as blob.
So I need to convert this file as a byte[].

No you don't. The following code would put the file into a database just as well:

// stmt is a java.sql.PreparedStatement object
// index is the index to put the blob in
InputStream input = new FileInputStream(myBigFile);
stmt.setBinaryStream(index, input);

Should you get a java.sql.SQLFeatureNotSupportedException if you try this, or are not using Java 6, you need to specify the size, but you can get that from the File object. You may run into problems if the file is larger than Integer.MAX_VALUE in size though; 2TB-1 is the maximum in that case.

[nimo frey]

thank you!!

your solution works fine !!

[Rob Spoor]

You're welcome.

I think you should close the FileInputStream after the statement is executed, or there will be errors (trying to copy from a closed stream or something like that). I haven't tried it, but if I would implement that method I would copy the data from the InputStream to the database directly upon updating (executing), not before.