Win a copy of Practical SVG this week in the HTML/CSS/JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

FileChannel: 2 gig limit for read but not write

 
Emilio Gagliardi
Greenhorn
Posts: 16
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi there!
I am attempting to use a FileChannel to read the contents of a file that is 18.6416 gigs of doubles. The specific error is an IllegalArgumentException generated at this line:

where j is a integer set in a for statement (it is never negative) which means that position was being set to a number larger then MAX_VALUE = 2147483647 the point in execution where the error occurs when there is an attempt to set the position value to larger than 2147748864.
What confuses me is that I wrote this file with a fileChannel in the first place. I wasn't manually setting position though.

Is there a simple workaround to this or do I need to write files smaller than MAX_VALUE in size in order to manually set where in the file I want to read? I need to be able to specify the position because I am looping over the file and I want to read at specific intervals (390764 bytes) that do not overlap. Is there another class that can handle large file sizes? Or is it that i'm asking too much to store all this data in one file?

Thanks for reading!~!
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
From what you'd said, this sounds like a bug; FileChannel really should be able to handle this without problems. Are you sure it's the FileChannel that's being set to > 2147483647 , and not a ByteBuffer or something? You might try posting some code here, to see if we can spot some problem. From what you've said though it sounds like you're doing everything correctly, and there's a bug in the JDK. (What version are you using?) Perhaps you can try using a different JDK? If that doesn't work, the only workaround I can see is to convert the one big file into a set of smaller files, all <= 2147483647 in length.
 
Emilio Gagliardi
Greenhorn
Posts: 16
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
OK, this is getting weirder!
In an attempt to solve the read problem I modified the write code as follows

Now, when I run the above code fileSize_0 starts at zero, and increases by 390784 bytes as expected.
However, when the file reaches Integer.MAX_VALUE in size, instead of returning the fileSize as expected this is the output I get:
and fileSize_0 continues to decrease in magnitude.

Can anyone decipher this?? This is beyond my understanding. What I don't understand is that in my original code, I wrote 50000 word vectors to file which created the original 19 gig file. How can the size of the file as reported by the FileChannel get smaller when in fact its getting bigger??? Could this be part of the reason I can access locations larger than Integer.MAX_VALUE?

Cheers,

[ June 16, 2004: Message edited by: Emilio Gagliardi ]
[ June 16, 2004: Message edited by: Emilio Gagliardi ]
 
S. Ganapathy
Ranch Hand
Posts: 194
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
fc_1 = new FileOutputStream(fileNameNormalized_1,true).getChannel();
fileSize_0 = (int)fc_0.size();

fc_0.size() returns a long value. But this is type cast to int. once size returns the value greater than Integer.MAX_VAL, the value becomes negative.
Please check this.
Regards,
Ganapathy, S.
[ June 16, 2004: Message edited by: S. Ganapathy ]
 
Emilio Gagliardi
Greenhorn
Posts: 16
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you very much for that! Can I ask a further question? Can you explain why I can't set a FileChannel's position to greater than Integer.MAX_VALUE and yet I can pass 19 gigs worth of doubles through the ByteBuffer and FileChannel to a file in the first place? That is the original problem that brought me here. The code I posted was my debugging efforts to figure out how to access portions of a single 19 gig file.

Since then I have reduced my files to hold no more than Integer.MAX_VALUE worth of bytes but it is a real hack job and still doesn't work. I keep getting IllegalArgumentException. Grrr...

Thank you for any suggestions.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!