Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

JAVA HEAP SIZE and files to byte[]-Array  RSS feed

 
nimo frey
Ranch Hand
Posts: 580
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The original issue was created in http://www.coderanch.com/t/448993/Performance/java/JAVA-HEAP-SIZE-files-byte, but I want to move this issue here, as it s more I/O-related:

I have a file which should be converted into a byte[]-Array.

All works fine. But when the file is larger than the maximum JAVA HEAP SIZE than this error occurs

Caused by: java.lang.OutOfMemoryError: Java heap space

I use the Apaches Commons "FileUtils.readFileToByteArray()"-Method to convert from File to byte[].

Also, the Apaches "IOUtils.toByteArray(new FileInputStream(myfile)))" returns a java.lang.OutOfMemoryError.


I know, I have to set the "Java heap space" to a higher value, but this is not always possible.

Is there a way to handle such big files without exceeding the "Java heap space".

I also tried this http://balusc.blogspot.com/2007/07/fileservlet.html, but without success.

I also tried it this way:




but java.lang.OutOfMemoryError occurs.

Is there a way, maybe with the new NIO-API to handle such scenarios in a performant way? java.nio offers ByteChannel and FileChannel, maybe converting from FileChannel to ByteChannel, but how?
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you expecting some sort of magic to take place? You want a byte[] in memory that is bigger than available memory??

Exactly what do you need to do? Most people would use a random access approach to look at byte contents of a file too big for memory. What operations do you have to perform on this large file?

Bill
 
nimo frey
Ranch Hand
Posts: 580
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I get a uploaded file and want to store that file into a database as blob.
So I need to convert this file as a byte[].

I guess, you are right - it is not possible without extending the heap-size.

So this is not a solution? http://forums.sun.com/thread.jspa?threadID=5378438

This could be a solution:
To read the stream (file) not completly, but in for example, 4 MB-blocks with a BufferedReader.

Would be fine to have this solution with java.nio:
get a FileChannel of that file and convert it to a ByteChannel and from ByteChannel to byte[]. Is this possible?
 
Rob Spoor
Sheriff
Posts: 20893
81
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
nimo frey wrote:I get a uploaded file and want to store that file into a database as blob.
So I need to convert this file as a byte[].

No you don't. The following code would put the file into a database just as well:
Should you get a java.sql.SQLFeatureNotSupportedException if you try this, or are not using Java 6, you need to specify the size, but you can get that from the File object. You may run into problems if the file is larger than Integer.MAX_VALUE in size though; 2TB-1 is the maximum in that case.
 
nimo frey
Ranch Hand
Posts: 580
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thank you!!

your solution works fine !!

 
Rob Spoor
Sheriff
Posts: 20893
81
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You're welcome.

I think you should close the FileInputStream after the statement is executed, or there will be errors (trying to copy from a closed stream or something like that). I haven't tried it, but if I would implement that method I would copy the data from the InputStream to the database directly upon updating (executing), not before.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!