This week's book giveaway is in the Programmer Certification forum.
We're giving away four copies of OCP Oracle Certified Professional Java SE 21 Developer (Exam 1Z0-830) Java SE 17 Developer (Exam 1Z0-829) Programmer’s Guide and have Khalid Mughal and Vasily Strelnikov on-line!
See this thread for details.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Tim Cooke
  • Campbell Ritchie
  • paul wheaton
  • Ron McLeod
  • Devaka Cooray
Sheriffs:
  • Jeanne Boyarsky
  • Liutauras Vilda
  • Paul Clapham
Saloon Keepers:
  • Tim Holloway
  • Carey Brown
  • Piet Souris
Bartenders:

Java nio

 
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,
I was trying out java nio and I wanted to see the difference between nio and the normal file stream.
I tried copying over 115MB using both methods and they both got the job done in 3 sec. Am i not supossed to see considerable difference in copying or is this for only very large files in GB.
Should I try something else to see a difference.

Jubin
 
jubin kart
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

I was trying to run the following code for memory mapped copying of files but it kept giving me java.io.IOException: Map failed. The file is about 700MB and does get copied only thing its corrupted and this exception is thrown.


any ideas... ?
 
Sheriff
Posts: 22849
132
Eclipse IDE Spring Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
"being_nice",

Please read your private messages regarding an important announcement.

Thank you,

Rob
 
jubin kart
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
done...
 
Sheriff
Posts: 67754
173
Mac Mac OS X IntelliJ IDE jQuery TypeScript Java iOS
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Not done. "jubin", please check your private messages again. And read carefully.
 
Ranch Hand
Posts: 225
Eclipse IDE Debian Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The JavaDoc for the map() methods says:

The behavior of this method when the requested region is not completely contained within this channel's file is unspecified.

If the output file doesn't already exist, you should probably make it the right length first.

I don't think this will work very well though. As far as I can tell, the put() method you're calling will copy the file byte by byte. Copying from a FileInputStream to a FileOutputStream using a large enough byte array as a buffer should work better.
 
Rancher
Posts: 5114
38
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You are not seeing the TRUE cause of the problem.
Add a printStackTrace() call to all the catch(){} blocks and you'll see!
 
jubin kart
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
thanks guys,
Printing out the stack trace helped. So how would I know how much memory I can copy over for a system , can I can I do it in parts?

 
jubin kart
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I did this in the end. Although it copied, the file came out corrupt and there isn't much difference between stream copy and nio.

 
Master Rancher
Posts: 5161
83
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Unfortunately, the API tells us nothing about whether memory-mapped files can ever be freed up after the mapping occurs. Depending on the operating system, the memory usage may be permanent. There's no particular reason to expect that it should be possible to map part of the file, free it up, and then map another part. Maybe it will work, or maybe not. But I wouldn't count on it.

Generally, if you want to copy files quickly using NIO, don't use map() - use transferTo():



Depending on your operating system and hardware, you may see substantial improvement with this code - or not. In many cases NIO is not significantly faster than regular IO, because many of the improvements of NIO have already been incorporated into IO beneath the covers. In some cases, however, further improvements are still possible. You won't really know until you try.
[ July 22, 2008: Message edited by: Mike Simmons ]
 
jubin kart
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks mike there is a very good difference in copy.
 
Norm Radder
Rancher
Posts: 5114
38
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I tried a few different methods and file sizes to see which way was fastest to copy a file. I used 3 files of about: 53M, 280M and 680M.
It appears no one way is fastest. Depended on the size of the file.
I'm on XP with 512M ram.
The program:

Results in next post
 
Norm Radder
Rancher
Posts: 5114
38
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The system rejected this when I had it tagged as QUOTE.

Fastest way to copy a large file

After reading the nio post I tried some experiments trying various ways to copy a large file.
Using


** copy2 - FileChannel 53M in 1.2 sec / 280M in 17.8-21 sec / 687M in 90-100 sec(79 w/128M)
D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Java\jdk-6-doc.zip size=54898268
copy2 done
copy time=1203, rate=45634
D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Java\jdk-6-doc.zip size=54898268
copy2 done
copy time=1031, rate=53247

D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Java.zip size=281183548
copy2 done
copy time=20969, rate=13409

D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Java.zip size=281183548
copy2 done
copy time=17656, rate=15925

D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Java.zip size=281183548
copy2 done
copy time=18985, rate=14810

D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Miscellaneous\CentOS-5.1-i386-LiveCD.iso size=720766976
copy2 done
copy time=89984, rate=8009

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy2 of E:\Downloads\Miscellaneous\CentOS-5.1-i386-LiveCD.iso size=720766976
copy2 done
copy time=79922, rate=9018

D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Miscellaneous\CentOS-5.1-i386-LiveCD.iso size=720766976
copy2 done
copy time=100703, rate=7157
>>>>>>>>>>>>> System now very sluggish/slow - effected first copy test below

-----------------------------------------------------------------------------
** copy3 - RandomAccessFile w/64M blocks 53M in 1.4-6.5sec / 280M in 23.5 sec / 687M in 65.5
D:\JavaDevelopment\Testing>java FileCopyTest
copy3 of E:\Downloads\Java\jdk-6-doc.zip to C:\Temp\jdk6doc.zip
setLength took=47, length=54898268, copySize=54898268
i/p seek took=0, seekPt=0
output seek took=0
read took=4875
write took=1078
Close took=0
copy time=6484, rate=8466

>>>>>>>>> System response time more normal
D:\JavaDevelopment\Testing>java FileCopyTest
copy3 of E:\Downloads\Java\jdk-6-doc.zip to C:\Temp\jdk6doc.zip
setLength took=0, length=54898268, copySize=54898268
i/p seek took=0, seekPt=0
output seek took=0
read took=219
write took=1047
Close took=0
copy time=1359, rate=40396 <<<<<<<<< 1.4 sec

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest <<<<<<<<<< NB took 128M
copy3 of E:\Downloads\Java.zip to C:\Temp\java.zip
setLength took=15, length=281183548, copySize=67108863
i/p seek took=0, seekPt=214074685
output seek took=0
read took=2172
write took=5531
Close took=0
opens took=0
i/p seek took=0, seekPt=146965822
output seek took=0
read took=1938
write took=2703
Close took=0
opens took=0
i/p seek took=0, seekPt=79856959
output seek took=0
read took=1844
write took=2547
Close took=0
opens took=0
i/p seek took=0, seekPt=12748096
output seek took=0
read took=1797
write took=2672
Close took=0
opens took=0
i/p seek took=0, seekPt=0
output seek took=0
read took=593
write took=1578
Close took=0
copy time=23531, rate=2851

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy3 of E:\Downloads\Java.zip to C:\Temp\java.zip
setLength took=0, length=281183548, copySize=67108863
i/p seek took=0, seekPt=214074685
output seek took=0
read took=1890
write took=5516
Close took=0
opens took=16
i/p seek took=0, seekPt=146965822
output seek took=0
read took=1860
write took=2593
Close took=0
opens took=0
i/p seek took=0, seekPt=79856959
output seek took=0
read took=1828
write took=2672
Close took=0
opens took=0
i/p seek took=0, seekPt=12748096
output seek took=0
read took=1797
write took=2640
Close took=0
opens took=0
i/p seek took=0, seekPt=0
output seek took=0
read took=609
write took=1500
Close took=0
copy time=23093, rate=2906

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy3 of E:\Downloads\Miscellaneous\CentOS-5.1-i386-LiveCD.iso to C:\Temp\Testing.iso
setLength took=0, length=720766976, copySize=67108863
i/p seek took=0, seekPt=653658113
output seek took=0
read took=2078
write took=15672
Close took=0
opens took=0
i/p seek took=0, seekPt=586549250
output seek took=0
read took=2671
write took=2750
Close took=0
opens took=0
i/p seek took=0, seekPt=519440387
output seek took=0
read took=2063
write took=2765
Close took=0
opens took=0
i/p seek took=0, seekPt=452331524
output seek took=0
read took=2063
write took=2656
Close took=0
opens took=0
i/p seek took=0, seekPt=385222661
output seek took=0
read took=1984
write took=2625
Close took=16
opens took=0
i/p seek took=0, seekPt=318113798
output seek took=0
read took=2000
write took=2640
Close took=0
opens took=0
i/p seek took=0, seekPt=251004935
output seek took=0
read took=2000
write took=2625
Close took=0
opens took=0
i/p seek took=0, seekPt=183896072
output seek took=0
read took=2000
write took=2656
Close took=0
opens took=0
i/p seek took=0, seekPt=116787209
output seek took=0
read took=2000
write took=2578
Close took=0
opens took=0
i/p seek took=0, seekPt=49678346
output seek took=0
read took=2015
write took=2563
Close took=0
opens took=0
i/p seek took=0, seekPt=0
output seek took=0
read took=2031
write took=2891
Close took=0
copy time=65531, rate=1024

-----------------------------------------------------------------------------
** copy1 MappedByteBuffer 53M in 0.15-2.3 sec / 280M in 20-24 sec
??? Was first one slow because System sluggish from above??? 2nd&3rd VERY FAST
D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java\jdk-6-doc.zip size is 54898268, dur=484
MaxRec=54898268
ix=0 at 1216742943687, map input=31, map output=78, put=1609
close=0
copy time=2281, rate=24067 <<<<<<<<<<<<<<<???

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java\jdk-6-doc.zip size is 54898268, dur=0
MaxRec=54898268
ix=0 at 1216743046937, map input=0, map output=0, put=141
close=0
copy time=157, rate=349670

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java\jdk-6-doc.zip size is 54898268, dur=16
MaxRec=54898268
ix=0 at 1216743171734, map input=0, map output=0, put=156
close=0
copy time=188, rate=292012

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java.zip size is 281183548, dur=31
MaxRec=67108863
ix=0 at 1216743254265, map input=0, map output=31, put=2000
ix=67108863 at 1216743256296, map input=0, map output=0, put=1688
ix=134217726 at 1216743257984, map input=0, map output=0, put=4875
ix=201326589 at 1216743264140, map input=31, map output=0, put=9922
ix=214074685 at 1216743274109, map input=0, map output=31, put=3594
close=562
copy time=24141, rate=11647

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java.zip size is 281183548, dur=234
MaxRec=67108863
ix=0 at 1216743448265, map input=31, map output=32, put=6172
ix=67108863 at 1216743454500, map input=0, map output=0, put=1046
ix=134217726 at 1216743455546, map input=0, map output=0, put=4344
ix=201326589 at 1216743460578, map input=78, map output=0, put=2625
ix=214074685 at 1216743464437, map input=266, map output=0, put=2547
close=625
copy time=19906, rate=14125

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java.zip size is 281183548, dur=235
MaxRec=67108863
ix=0 at 1216743575750, map input=31, map output=31, put=1734
ix=67108863 at 1216743577546, map input=0, map output=32, put=6906
ix=134217726 at 1216743585781, map input=156, map output=16, put=56250
ix=201326589 at 1216743643828, map input=203, map output=234, put=57891
ix=214074685 at 1216743704968, map input=546, map output=266, put=8750
close=500
copy time=139688, rate=2012 <<<<<<<<<<<<< OS went away ???
 
Carey Evans
Ranch Hand
Posts: 225
Eclipse IDE Debian Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I don�t know how it compares to mapping the files, but you can gain quite a bit of performance by using file channels with a direct byte buffer. For example:
 
Mike Simmons
Master Rancher
Posts: 5161
83
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
[Carey]: I don�t know how it compares to mapping the files

Or more pertinently, how it compares to using transferTo() or perhaps transferFrom(). The key here is: if you don't need to change or analyze anything about the content - if you only need to copy the bytes and nothing else: use a technique that communicates this basic intent. And trust that it may be implemented by people who aren't total morons, and are capable of being efficient once they realize that we don't care about the content, as long as it's transferred accurately. Most other APIs (really, all other Java APIs I'm aware of) try to transfer the content to some intermediate form - most commonly, a byte[] array. Which would allow our program to read individual bytes, and analyze them. If we cared to do so. But often, we don't. Why waste time transferring bytes to an intermediate readable data structure if you don't need to read the bytes? Just tell the underlying system what you want: move content from foo to bar, and don't waste time making those bytes available to our Java program (located somewhere else entirely). We don't care. Just move the bytes. That's what transferTo() and transferFrom() can accomplish.
[ July 24, 2008: Message edited by: Mike Simmons ]
 
reply
    Bookmark Topic Watch Topic
  • New Topic