New IO

[Brian Spindler]

I've started looked at the nio packages and wondering if I should consider some re-write for performance gain (No I have not run any test to determine if there is an I/O bottleneck, I am merely curious).

The following code:

    
import java.io.BufferedReader;
import java.io.CharArrayReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.nio.CharBuffer;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileChannel.MapMode;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
 
public class Test {
 
private String FILE = "/home/owner/tmp/t.pl";
 
private static CharsetDecoder decoder = Charset.defaultCharset()
.newDecoder();
 
public Test() {
 
}
 
public void doFileChannel() throws IOException {
/*
 * FileChannel is obtained by first getting a new FileInputStream
 * and calling getChannel() 
 */
File f = new File(FILE);
FileInputStream fis = new FileInputStream(f); 
FileChannel fc = fis.getChannel();
/*
 * CharBuffer is obtained by calling FileChannel's map() method
 * this method takes a ByteBuffer (In this case a MappedByteBuffer)
 * 
 * we pass it to decoder.decode() to decode bytes to the Charset  
 * being utilized.
 */
MappedByteBuffer mbb = fc.map(MapMode.READ_ONLY, 0, fc.size());
CharBuffer cb = decoder.decode(mbb);
/*
 * BufferedReader to walk through the file
 */
BufferedReader br = new BufferedReader(new CharArrayReader(cb.array())); 
String line;
while ((line = br.readLine()) != null) {
//System.out.println(lineMatcher.group());
}
fc.close();
}

private void doFileStream() throws IOException {
/*
 * New File
 * FileReader from new File
 * BufferedReader to do the readLine() 
 */
File f = new File(FILE);
FileReader fr = new FileReader(f);
BufferedReader br = new BufferedReader(fr);
String line;
while((line = br.readLine()) != null) {
//System.out.println(line);
}
br.close();
}
/**
 * @param args
 * @throws FileNotFoundException
 */
public static void main(String[] args) throws IOException {
long start = System.currentTimeMillis();
new Test().doFileChannel();
long finish = System.currentTimeMillis();
long time = finish-start;
System.out.println("doFileChannel : " + time + "ms");

start = System.currentTimeMillis();
new Test().doFileStream();
finish = System.currentTimeMillis();
time = finish-start;
System.out.println("doFileStream : " + time + "ms");
}
}
 

(t.pl is a small 11 line perl script)
yields this output:
doFileChannel : 14ms
doFileStream : 1ms

Am I missing something here, the nio packages seem to be much more overhead and much slower. I'm sure there are particular cases where nio shine but basic file operations don't appear to be that.

[steve souza]

A couple points:

1) Be wary of testing order - I noticed that the slower test is the first one. This often happens as first invocations often have more overhead such as loading classes, jvm start up, and even loading the file into an OS cache.

2) Repeat tests - One repetition on a test, particularly one that is so fast won't be definitive. Repeat the test in a loop randomly determining the order of the invocations of the different approaches.

3) Be cautious when interpretting small execution times - Such small times are often misleading as the clock accuracy on certain machines is only say 10 ms.. So often times that are really the same will show up as say either 0 ms., or 10 ms. even though there is no real difference between the two. It was just random whether when the code ran if the OS clock was ready to tick or not.

4) Make sure you are testing the same things - As you extra code in one approach make sure that it is a fair comparison, i.e. it isn't doing more and different things on the data set.

5) Read a larger file - You may not get the full benefit of faster IO on a file that only takes a couple ms. to read. This is more a measurement of startup costs than IO costs.

I would address these problems in the test before interpretting any results.

[Jim Yingst]

Your code tests one particular way of using NIO; it's not very meaningful as a generalization about perfomance of NIO in all cases. And note this sentence from the API for map():

"For most operating systems, mapping a file into memory is more expensive than reading or writing a few tens of kilobytes of data via the usual read and write methods. From the standpoint of performance it is generally only worth mapping relatively large files into memory."

For small files, you're probably better off not using MappedByteBuffer. You may want to try using FileChannel's read() method to read in to a ByteBuffer, and then convert the ByteBuffer to s CharBuffer using a CharsetDecoder. Or not. Often NIO just ends up being more work to figure out, with little or no gain. There are specific niches where it really excels, but for what you seem to be doing (reading lines as Strings), I wouldn't expect any great improvements from NIO. If it weren't a text file, you'd have better luck with reading a raw ByteBuffer - but since it's text, the added complications probably aren't worth the effort.

If you're using JDK 5 or later, I'd just use a java.util.Scanner, which uses some NIO under the covers, and makes the whole process of reading text somewhat simpler than it was previously.
[ May 20, 2007: Message edited by: Jim Yingst ]

[Brian Spindler]

Thanks guys/gals, some good feedback.

[Brian Spindler]

Wow! by changing to the following code block:

    
ByteBuffer bb = ByteBuffer.allocate((int)fc.size());
fc.read(bb);
//MappedByteBuffer mbb = fc.map(MapMode.READ_ONLY, 0, fc.size());
CharBuffer cb = decoder.decode(bb);

And changing to a 34M file, the output is now:

doFileStream : 1618ms
doFileChannel : 150ms

This output is the same regardless of which one is executed first (you can see I swapped them from previous example).

[Dhaneshkumar Parthasarathi]

1. File FILE = new File("C:\\primes.txt");
2. FileChannel fileChannel = new FileInputStream(FILE).getChannel();

//MappedByteBuffer mbb = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
3. ByteBuffer buffer = ByteBuffer.allocate((int)fileChannel.size());
4.fileChannel.read(buffer);
5.CharBuffer cb = decoder.decode(buffer);
6.BufferedReader br = new BufferedReader(new CharArrayReader(cb.array()));
7.String line;
8.while ((line = br.readLine()) != null) {
9. System.out.println(" FileChannel : "+line);
10.}

in the above program , the line number 9 is not printing the line(String value which is read from the file) ,

if i add buffer.flip(); after 4th line , then i can able to print the line.

could you someone let me know is there any mistake above code .

Regards
Dhaneshkumar

[Nitesh Kant]

DontWakeTheZombies. If you have a question, create a new topic.