Win a copy of Functional Reactive Programming this week in the Other Languages forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

File Size Issue while delivering file more than 2gb using java.nio.channels.FileChannel.transferFrom

 
sup rty
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

I am facing an issue while delivering a file more than 2gb using the API java.nio.channels.FileChannel.transferFrom. The size of the file copied from source is only till 2gb.

Kindly let me know if anyone could help me out with this.
Basically I need to find out about some JAva api which could help in coping files more than 2 Gb.


thanks
 
Wouter Oet
Saloon Keeper
Posts: 2700
IntelliJ IDE Opera
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Too difficult for Beginning Java. Moved to I/O and Streams.
 
sup rty
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thanks Wouter..

also to add give more information...The file delivery is done using FTP copy method...in my code I have something like this:

destChannel.transferFrom(srcChannel, 0, srcChannel.size());

This java.nio.channels.FileChannel.transferFrom is basically transferring only 2 gb of the data and hence there is mismatch between the source file and destination file size.
So I am assuming it does not alllow file copy for more tha 2 gb...So if any one has any idea and then please do let me know...


Thanks in advance
 
Joe Ess
Bartender
Posts: 9319
10
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am not aware of a limit on the amount of data that the Java IO API can handle. You are probably running into a file system limit. What's the file system on the destination?
 
sup rty
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Joe Thanks for the reply.....
I am checking the file size limit with the system. So according to you the there cannot be any issue with the java api in delivering files over 2 gb size. are there any specific ways or any methods or any API by which we could deliver such huge files?
Please let me know.
 
Rob Spoor
Sheriff
Pie
Posts: 20667
65
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Is the destination file system FAT32? Because that has its own limit of 2GB.
 
Raghavan Muthu
Ranch Hand
Posts: 3381
Mac MySQL Database Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
May be an additional info:

how about the available memory in terms of total RAM and allocated size to JVM? will it be of some help if being fine tuned?

As far as I have experienced, the I/O related stuff behave differently on different platforms. One good example is the way /tmp file works. Have you checked the documentation on the target platform?
 
Mike Simmons
Ranch Hand
Posts: 3090
14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you calling transferFrom() in a loop, checking the return value to ensure that all bytes are transferred? The API does indicate it's possible that not all bytes are transferred in one call, though it's vague about why this may occur. I would do something like this:
 
Raghavan Muthu
Ranch Hand
Posts: 3381
Mac MySQL Database Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thats a good tip Mike Simmons!
 
sup rty
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi ALL,

There seems to be no size limit in the destination system.

the filesize limit that you mean witth 2 gb files is in FAT 16 Filesystem (like MS DOS 3.x)
the FAT32 Filesystem has a file limit with 4GB, (like WIN95,...)

we are on UNIX (Solaris) there are know filesize limits on UFS or Storenext Filesystems
 
Joe Ess
Bartender
Posts: 9319
10
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Raghavan Muthu wrote:how about the available memory in terms of total RAM and allocated size to JVM? will it be of some help if being fine tuned?


If he were running into a RAM issue, I'd expect to see an OutOfMemory exception. Now there is a size limit on a MappedByteBuffer: Integer.MAX_VALUE (which just happens to be 2GB), but that shouldn't affect a plain transfer.
Sup, are you catching Exceptions correctly?
Have you tried using the plain java.io classes to accomplish this copy?
Have you tried a non-Java method to copy the file?
Do you have some code so we can reproduce this problem?
 
Mike Simmons
Ranch Hand
Posts: 3090
14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So, Sup: have you tried calling the method in a loop like I suggested above?
 
sup rty
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Mike,

I tried the for loop but facing the same issue. Only 2gb of the file is getting copied on the destination. i tried this:

srcChannel = new FileInputStream(sourceFile).getChannel();

destChannel = new FileOutputStream(fileDeliveryTemp).getChannel();

for (long n = 0, s = srcChannel.size(); n < s; ) {
s_logger.debug("inside for loop " + destChannel.size());
n += destChannel.transferFrom(srcChannel, n, s - n);
}
 
Mike Simmons
Ranch Hand
Posts: 3090
14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Interesting.

By the way, please UseCodeTags in the future.

Have you tried closing the stream and/or channel? Either should be sufficient, and I should have asked this earlier. That should be enough - but if it doesn't work, also try FileChannel's force() method. I don't think it matters much whether you use force(true) or force(false).

If it's still not working, what does your new debug statement tell you? Better yet, let's add a few more debugs:

What does the log output tell you? How many times is the loop really executing? What is the total number of bytes that the methods claims have been written (n)?
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic