• Post Reply Bookmark Topic Watch Topic
  • New Topic

Serialize with Java NIO  RSS feed

 
Richard Gray
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am new to Java and have been trying to serialize objects to a file using ObjectInputStream and ObjectOutputStream. As a requirement, I am adding image files about 1MB as byte[] since an Image is not serializable. Once I deserialize the byte[], I convert it to BufferedImage so that I can process it such as subimages and so forth. I am attempting to serialize many objects into one file. (greater than 2 GB)

The issues that I am running into is that when I get to 1024MB, I get a OutOfMemory exception. I got the exception earlier and I increased the heap sisze -Xmx1024m.

I know this is not the way to go because I will always run into the OutOf Memory exception.

Now with Java NIO, I am thinking that I can use a memory mapped file using Channels and MappedByteBuffer.

The question is:

How do I serialize my objects into a file using this method to perform a write and a read. I am assuming that I would have to use the ObjectInputStream and ObjectOutputStream with the Channels and ByteBuffer but not sure where to begin.

I would like to use the Java NIO because it utilizes the Operating System for I/O rather than filling up my JVM Heap which results in OutOfMemory.

Could someone please give me guidance into how I can go about doing this?

I exhaustively search various forums and Googled it and could not find a soultion that makes sense to me.
 
Carey Evans
Ranch Hand
Posts: 225
Debian Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
ObjectOutputStream has some unexpected behaviour when you try to use it for large numbers of large objects. In particular, it holds onto a reference to every object you write until you call reset() or close(), which is where your 1GB of heap is going. It may not be the best way to save raw image data.

A MappedByteBuffer will also cause problems when it gets very large; on 32-bit Windows, you can only have 2GB of data per process, including Java heap and all MappedByteBuffers. You also can't guarantee when a MappedByteBuffer's memory will be released to the OS.

For my part, I would use a ZipOutputStream, or a DataOutputStream with the array length then the byte array written for each image.
 
Richard Gray
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ok, if we scrap using the ObjectOutputStream/ObjectInputSTream due to limitaions and buffering problems, and unexpected buffering issues with MappedByteBuffer, and go with using ZipOutputStream or DataOutputStream using byte arrays, that would mean we would need to convert an Object to a byte array using BufferedOutputStreams.

Then the question of reading the bytes from the file, how do we know how much bytes to read from the file to retrive the object?

As you can see, I am new to Java and have been using the APIs that Java has provided for these kinds of scenarios so I have stuck with using the ObjectOutputStreams since Java will return back the objects using readObject and write using writeObject. This makes it very intuitive.

When we need to read a number of bytes then having to form that back into an object is where I am starting to get confused and anxious.

Is there not a way in Java to serialize large amounts of Objects into a file without effecting heap? I thought this is why NIO comes into play.

 
Carey Evans
Ranch Hand
Posts: 225
Debian Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think I made some incorrect assumptions about what you were doing. So... if ObjectOutputStream is working for you with multi-gigabyte files, apart from the memory leak, just call reset() after writing each set of objects.

I dislike using ObjectOutputStream, from personal experience, because it's nearly impossible to check the file contents without using another Java program with the same classes on the classpath.

Just returning to your original question: ObjectOutputStream can only wrap an OutputStream. To write to a Channel, you can wrap the Channel with Channels.newOutputStream(), and use that. I don't think you can serialize directly to any kind of Buffer.
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!