• Post Reply Bookmark Topic Watch Topic
  • New Topic

Using FileChannel instead of RandomAccessFile

 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Guys,

Can someone help me with this code. I have writen a simple ObjectStore using Random Access File (basically, for seeking, getting file pointers etc. functionality). I came across this FileChannel couple of days agao and read that it can improve IO performance a lot.

Can someone please tell me how i can convert this code to use FileChannel.
Keeping functionality same.


[ October 25, 2007: Message edited by: Bear Bibeault ]
 
Rob Spoor
Sheriff
Posts: 20817
68
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Akhil Jain:
I came across this FileChannel couple of days agao and read that it can improve IO performance a lot.

Just because it can, doesn't mean it has to.

I once tested a file copy in three different ways:
- using InputStream and OutputStream using block sized reading
- using Reader and Writer using block sized reading
- using FileChannel

Now everybody was saying that with FileChannel it was so much faster. Well I found out it that's not necessarily true. By playing around with the block size, I found out that using InputStream and OutputStream can be MUCH faster than FileChannel! Of course, use the wrong block size and it's much slower, but that just shows that performance all depends on the settings and the program.
 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
But my current program takes almost 3-4 seconds to write 1000 (One thousand) objects to a file. and i need a performance factor of atleast 20,000 per second.

How i can achieve this? Any Suggestions
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Rob: were you using the transferTo() or transferFrom() methods? As I recall, that's where the real performance improvement in FileChannel came from, for copying.

Still, it's very true that improvements are far from guaranteed, and depend on a lot of different things. When I first modified a copy routine from using streams to using FileChannel, it became a factor of ten faster on my wondows box - but the exact same code on a linux box showed no change in performance at all. And yes, I tried different block sizes to be sure that was well-tuned. NIO was new then; I guessed that maybe the linux port hadn't been optimized much yet. These things can change there performance characteristics from platform to platform, and release to release. And various other things, probably; little is guaranteed.

Akhil: what sort of objects are you talking about? How big? You might try commenting out the code that actually writes to the file. Just write to a ByteArrayOutputStream to see how fast that is. If it's too slow, then forget about writing to a file; it will just be even slower. I would try looking at what sort of data the objects represent, and see if you can find a more compact, faster format for that data. Serialization may be too slow for what you need.
 
Rob Spoor
Sheriff
Posts: 20817
68
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Jim Yingst:
Rob: were you using the transferTo() or transferFrom() methods? As I recall, that's where the real performance improvement in FileChannel came from, for copying.

Yes I was, using 0 and the original file size as positions, that was why I was a bit surprised. But like I said, duplicating or halving the block size could shift results around quite a bit.
 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I modified my program to use filechannels and here are the results.

With Normal IO operation usign RandomAccessFile
Time taken to write 1000 objects in a file is: 4406 ms

with NIO operation using filechannel
Time taken to write 1000 objects in a file is: 1547 ms

here is the code i modified using file channel.

Objects which i want to serialize are Events Objects (a wrapper which will contain the actual event Object). this line should tell it all, which i am using in testing my code by creating dummy objects.

Events event1 = new Events("Event1","Topic1",new Date(),new DummyEvent1(01,"dummyEvent1"));

Now, the requirement is to have a maechanism to write at least 20,000 object to a file, which can be later deserialized and can be viewed using Event Viewer kind of application (basically to play them back.).

Now i need your advice guys, whats the best way of doing things.?
 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
My Appologies foe earlier posting regarding the code.
It was not properly using fileChannel.

Here is the code which is using FileChannels properly and performance is more faster. (but dont miss to answer my questions in previous post)



Performance measurements are as follows:

Time taken to write 10000 objects in a file is: 1015 ms
Store Count of Object Store is: 10000
Time taken to get the count of objects in a file is : 157 ms
Object at position 250 is: Event2
Time taken to read specific object in a file at position 250 is: 15 ms
Time taken to read set of objects in a file at position 200 to 220 is: 63 ms


regards
Akhil Jain
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well, it sounds like you're getting close to the time you need, yes? If you need more, then I still think serialization with ObjectOutputStream may be limiting you. Every single Events object serialized this way needs a new ObjectOutputStream with a new ByteArrayOutputStream and a new byte[] array which gets a new serialization header written to it, plus the data you actually want. If there's another way to transmit the data in these objects (e.g. using the DataOutput methods you get from a RandomAccessFile), that may well be faster.

[Akhil]: Events event1 = new Events("Event1","Topic1",new Date(),new DummyEvent1(01,"dummyEvent1"));

OK, most of this data can be written with RAF methods like writeUTF(), writeLong(), writeInt(). The problem is that DummyEvent object. What kinds of Events do you really have here? Do they have a common interface? You may be able do do something like this:

I'm just guessing about the methods in the Events class, and the Event it wraps. This approach will work if the Event is fairly simple and consistent in the types of data it contains - but if that vaires too much, this could get messy.

Then too, there may be other alternatives to serialization which may be faster. One that comes to my mind is XStream, a simple, easy-to use tool for converting objects to a simple XML representation and back. You can get good performance from it if you reuse the XStream object used to do the xml serialization, as it caches a lot of its work internally. There may well be better alternatives; I just don't know them. XStream is pretty simple to try, though.
[ October 24, 2007: Message edited by: Jim Yingst ]
 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Jim.

I will try my hands on XStream.

Will let you knwo the results.

Regards,
Akhil Jain
 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Jim,

XStream doesn't seems to be a right tool for this.
Here is the response from XStream development team itself.

"Methinks XStream is not the right tool here. It is stream based and there's no way to access random single elements. Additionally I assume from your description of 1+2 that you have written binary data. Keep in mind that XML has overhead in general. While writing can be reasonable fast, reading is a completely different matter and depends largly on the underlaying XML parser.

- J�rg"
 
Adam Fry
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sorry to resurrect this thread, but I recently encountered this myself and felt obliged to chime in.

1.) When you are instantiating the RandomAccessFile, you are setting mode "rwd." This forces writes to be synchronous with the physical media, which is _incredibly slow_.
2.) If you look at the source for the JDK, you will see that RandomAccessFile uses the nio classes underneath it already - FileChannel and FileDescriptor.

Just use "rw" instead of "rwd" and you don't have to reinvent the wheel.

(Sorry about the name, I shamefully admit that I read the naming guidelines *after* registering and posting. Hopefully it'll be updated to reflect my compliance at some point...)
[ January 09, 2008: Message edited by: Adam Fry ]
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!