Forums Register Login

File Size Issue while delivering file more than 2gb using java.nio.channels.FileChannel.transferFrom

+Pie Number of slices to send: Send
Hi,

I am facing an issue while delivering a file more than 2gb using the API java.nio.channels.FileChannel.transferFrom. The size of the file copied from source is only till 2gb.

Kindly let me know if anyone could help me out with this.
Basically I need to find out about some JAva api which could help in coping files more than 2 Gb.


thanks
+Pie Number of slices to send: Send
Too difficult for Beginning Java. Moved to I/O and Streams.
+Pie Number of slices to send: Send
thanks Wouter..

also to add give more information...The file delivery is done using FTP copy method...in my code I have something like this:

destChannel.transferFrom(srcChannel, 0, srcChannel.size());

This java.nio.channels.FileChannel.transferFrom is basically transferring only 2 gb of the data and hence there is mismatch between the source file and destination file size.
So I am assuming it does not alllow file copy for more tha 2 gb...So if any one has any idea and then please do let me know...


Thanks in advance
+Pie Number of slices to send: Send
I am not aware of a limit on the amount of data that the Java IO API can handle. You are probably running into a file system limit. What's the file system on the destination?
+Pie Number of slices to send: Send
Hi Joe Thanks for the reply.....
I am checking the file size limit with the system. So according to you the there cannot be any issue with the java api in delivering files over 2 gb size. are there any specific ways or any methods or any API by which we could deliver such huge files?
Please let me know.
+Pie Number of slices to send: Send
Is the destination file system FAT32? Because that has its own limit of 2GB.
+Pie Number of slices to send: Send
May be an additional info:

how about the available memory in terms of total RAM and allocated size to JVM? will it be of some help if being fine tuned?

As far as I have experienced, the I/O related stuff behave differently on different platforms. One good example is the way /tmp file works. Have you checked the documentation on the target platform?
+Pie Number of slices to send: Send
Are you calling transferFrom() in a loop, checking the return value to ensure that all bytes are transferred? The API does indicate it's possible that not all bytes are transferred in one call, though it's vague about why this may occur. I would do something like this:
+Pie Number of slices to send: Send
Thats a good tip Mike Simmons!
+Pie Number of slices to send: Send
Hi ALL,

There seems to be no size limit in the destination system.

the filesize limit that you mean witth 2 gb files is in FAT 16 Filesystem (like MS DOS 3.x)
the FAT32 Filesystem has a file limit with 4GB, (like WIN95,...)

we are on UNIX (Solaris) there are know filesize limits on UFS or Storenext Filesystems
+Pie Number of slices to send: Send
 

Raghavan Muthu wrote:how about the available memory in terms of total RAM and allocated size to JVM? will it be of some help if being fine tuned?



If he were running into a RAM issue, I'd expect to see an OutOfMemory exception. Now there is a size limit on a MappedByteBuffer: Integer.MAX_VALUE (which just happens to be 2GB), but that shouldn't affect a plain transfer.
Sup, are you catching Exceptions correctly?
Have you tried using the plain java.io classes to accomplish this copy?
Have you tried a non-Java method to copy the file?
Do you have some code so we can reproduce this problem?
+Pie Number of slices to send: Send
So, Sup: have you tried calling the method in a loop like I suggested above?
+Pie Number of slices to send: Send
Hi Mike,

I tried the for loop but facing the same issue. Only 2gb of the file is getting copied on the destination. i tried this:

srcChannel = new FileInputStream(sourceFile).getChannel();

destChannel = new FileOutputStream(fileDeliveryTemp).getChannel();

for (long n = 0, s = srcChannel.size(); n < s; ) {
s_logger.debug("inside for loop " + destChannel.size());
n += destChannel.transferFrom(srcChannel, n, s - n);
}
+Pie Number of slices to send: Send
Interesting.

By the way, please UseCodeTags in the future.

Have you tried closing the stream and/or channel? Either should be sufficient, and I should have asked this earlier. That should be enough - but if it doesn't work, also try FileChannel's force() method. I don't think it matters much whether you use force(true) or force(false).

If it's still not working, what does your new debug statement tell you? Better yet, let's add a few more debugs:

What does the log output tell you? How many times is the loop really executing? What is the total number of bytes that the methods claims have been written (n)?
Rototillers convert rich soil into dirt. Please note that this tiny ad is not a rototiller:
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com


reply
reply
This thread has been viewed 10099 times.
Similar Threads
Error while read zip file
JVM memory limitation
Java performace issue in 32 bit Red Hat
Error during unzip ...
how to resolve OutOfMemoryError
More...

All times above are in ranch (not your local) time.
The current ranch time is
Mar 28, 2024 23:26:34.