FileChannel: 2 gig limit for read but not write

[Emilio Gagliardi]

Hi there!
I am attempting to use a FileChannel to read the contents of a file that is 18.6416 gigs of doubles. The specific error is an IllegalArgumentException generated at this line:

fc_level2.position(j);  

where j is a integer set in a for statement (it is never negative) which means that position was being set to a number larger then MAX_VALUE = 2147483647 the point in execution where the error occurs when there is an attempt to set the position value to larger than 2147748864.
What confuses me is that I wrote this file with a fileChannel in the first place. I wasn't manually setting position though.

Is there a simple workaround to this or do I need to write files smaller than MAX_VALUE in size in order to manually set where in the file I want to read? I need to be able to specify the position because I am looping over the file and I want to read at specific intervals (390764 bytes) that do not overlap. Is there another class that can handle large file sizes? Or is it that i'm asking too much to store all this data in one file?

Thanks for reading!~!

[Jim Yingst]

From what you'd said, this sounds like a bug; FileChannel really should be able to handle this without problems. Are you sure it's the FileChannel that's being set to > 2147483647 , and not a ByteBuffer or something? You might try posting some code here, to see if we can spot some problem. From what you've said though it sounds like you're doing everything correctly, and there's a bug in the JDK. (What version are you using?) Perhaps you can try using a different JDK? If that doesn't work, the only workaround I can see is to convert the one big file into a set of smaller files, all <= 2147483647 in length.

[Emilio Gagliardi]

OK, this is getting weirder!
In an attempt to solve the read problem I modified the write code as follows

 public ByteBuffer bbDbl = ByteBuffer.wrap(new byte[390784]);
 public DoubleBuffer db = bbDbl.asDoubleBuffer();
 private void writeVector(double[] aWordVector)
    {
int fileSize_0= 0;
try
{
    String fileNameNormalized_0 = Util.getFilePath()+"normalizedMatrix2D_0.txt";
    String fileNameNormalized_1 = Util.getFilePath()+"normalizedMatrix2D_1.txt";
    fc_0 = new FileOutputStream(fileNameNormalized_0,true).getChannel();
    fc_1 = new FileOutputStream(fileNameNormalized_1,true).getChannel();
    fileSize_0 = (int)fc_0.size();
}catch(IOException ioe)
{
    System.out.println(ioe.getMessage());
}
/*
 *Put array into byte buffer
         *aWordVector is simply an array of 50000 doubles
 */
    db.put(aWordVector);
   //rewind the DoubleBuffer
    bbDbl.rewind();
    //rewind the ByteBuffer, limited is unchanged
    db.rewind();
    
    try
    {
System.out.println("file size:"+fileSize_0 + " remainging space: " + (Integer.MAX_VALUE-fileSize_0));
    if(fileSize_0<Integer.MAX_VALUE)
    {
fc_0.write(bbDbl);
    }else
    {
fc_1.write(bbDbl);
    }

 
    }catch(IOException ioe)
    {
System.out.println("ERROR Aggregator: Could not write to file.");
    }
bbDbl.clear();//clears the buffer;sets position to 0; sets limit to capacity
db.clear();
    }

Now, when I run the above code fileSize_0 starts at zero, and increases by 390784 bytes as expected.

file size:0 remainging space: 2147483647
file size:390784 remainging space: 2147092863
file size:781568 remainging space: 2146702079
file size:1172352 remainging space: 2146311295

However, when the file reaches Integer.MAX_VALUE in size, instead of returning the fileSize as expected this is the output I get:

file size:2146967296 remainging space: 516351
file size:2147358080 remainging space: 125567
file size:-2147218432 remainging space: -265217
file size:-2146827648 remainging space: -656001
file size:-2146436864 remainging space: -1046785
file size:-2146046080 remainging space: -1437569

and fileSize_0 continues to decrease in magnitude.

Can anyone decipher this?? This is beyond my understanding. What I don't understand is that in my original code, I wrote 50000 word vectors to file which created the original 19 gig file. How can the size of the file as reported by the FileChannel get smaller when in fact its getting bigger??? Could this be part of the reason I can access locations larger than Integer.MAX_VALUE?

Cheers,

[ June 16, 2004: Message edited by: Emilio Gagliardi ]
[ June 16, 2004: Message edited by: Emilio Gagliardi ]

[S. Ganapathy]

Hi,

fc_1 = new FileOutputStream(fileNameNormalized_1,true).getChannel();
fileSize_0 = (int)fc_0.size();

fc_0.size() returns a long value. But this is type cast to int. once size returns the value greater than Integer.MAX_VAL, the value becomes negative.
Please check this.
Regards,
Ganapathy, S.
[ June 16, 2004: Message edited by: S. Ganapathy ]

[Emilio Gagliardi]

Thank you very much for that! Can I ask a further question? Can you explain why I can't set a FileChannel's position to greater than Integer.MAX_VALUE and yet I can pass 19 gigs worth of doubles through the ByteBuffer and FileChannel to a file in the first place? That is the original problem that brought me here. The code I posted was my debugging efforts to figure out how to access portions of a single 19 gig file.

Since then I have reduced my files to hold no more than Integer.MAX_VALUE worth of bytes but it is a real hack job and still doesn't work. I keep getting IllegalArgumentException. Grrr...

Thank you for any suggestions.