• Post Reply Bookmark Topic Watch Topic
  • New Topic

Need help regarding serializing multiple Objects by difeerent Threads in a single fil  RSS feed

 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Guys,

Need help urgently.
I am working on a thing to serialize my user-define object to a file.
I want to use FileChannel in NIO package of JDK 1.4 efficiently, problem i am facing is that how to serialize multiple objects to a single file by multiple threads. e.g there is a user defined class called Events and this event class object needs to be serialized in a file on disk say "events.ser". and there are multiple threads which are having that "Event Object" with different state and want to serialize them in same file.finally, "events.ser" should be structured like this:

eventobject1
eventobject2
eventobject3

So that, when a seperate program wants to inspect the "event.ser" file, it should be able to deserialize all the objects persisted by different threads and should be able to analyse them using reflection or whatever.

here are my sample codes i have written so far. please help me out in this.

Main Testing Class

[ October 17, 2007: Message edited by: Jim Yingst ]
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What does it do now that is not correct?
 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
No the above code is not working as desired.... it does write to a file but when i deserialize it, it gives me StreamCurruptException and fails badly.
the deserialization code i am using is as follows:



What is the best way to serialize objects (the most efficient, keeping performance in mind), if we have to serialize around 20,000 Objects per sec to a file? and that too....a strategy which doesn't uses much of a synchronized blocks or methods (because they have their own performance bottlenecks) and allow multiple threads to persist objects in the same file?
[ October 17, 2007: Message edited by: Jim Yingst ]
 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE Java MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Your problem is that you are creating a new ObjectOutputStream in every thread to write an object. ObjectOutputStream writes a header once for every serialization process. ObjectInputStream expects the input stream to adhere by the above rule.
In your case, this header will be present before every object serialized in the stream. So, you get the exception. More details here

Originally posted by Akhil Jain:

What is the best way to serialize objects (the most efficient, keeping performance in mind), if we have to serialize around 20,000 Objects per sec to a file? and that too....a strategy which doesn't uses much of a synchronized blocks or methods (because they have their own performance bottlenecks) and allow multiple threads to persist objects in the same file?


If you have multiple threads writing to the same file then you have to synchronize the writes i.e. one thread writes to the file at a time. Otherwise the data will be corrupted. This is what channel does for you. It only allows one thread to write to a file at a time.

I think you have to write a single class that writes to the file and every other class just queues its events to this class. The writer takes one event at a time and writes to the file. I dont think this would scale very well as the number of event publishers will increase with time.

If you are writing a production level code, then i dont think this is the correct way of doing things. You can have a messaging system that can take these events and store them reliably.
The performance numbers you have put is for writing to the file or processing the events? I think the numbers are very aggressive to start with.
[ October 17, 2007: Message edited by: Nitesh Kant ]
 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I dint really got much from the thread link you pasted..
Does it mean that if multiple threads are writting to a same file, then the header should be written on once? or is it the other way round, that every thread's ObjectoutputStream should first write a header and then write its object to a file?

if you can tell me in detail that will be great, in this particular scenario, cause i tried passing the same ObjectOutputStream object to all the threads while instantiating them.
something like
....
ByteArrayOutputStream baos = new ByteArrayOutputStream();
....

Thread thread1 = new Thread(new Thread1(eventArray,baos,channel));

Please comment.
 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE Java MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Why do you want to use channels for doing this?
 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Because channel will automatically take care of Multiple Thread scenario and moreover, using synchronized startegy will have a nice perfromance hit with so many Objects being serialised and written concurrently by different threads in single file.
 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE Java MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Akhil Jain:
Because channel will automatically take care of Multiple Thread scenario and moreover, using synchronized startegy will have a nice perfromance hit with so many Objects being serialised and written concurrently by different threads in single file.

Channels does nothing but synchronization to stop concurrent updates to the same file. They do not do a synchronization on all the methods but for the methods that involve change in size of the file. This will be true in your case and so write to the file will always be one at a time.
As i told before, if multiple threads are writing to the same file then you *can not* do without synchronization.
 
Akhil Jain
Ranch Hand
Posts: 74
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Can you please give me an example of, how i can achieve the following scenario. because i am still getting StreamCurruptedException while deserializing the objects from file.

Strategy for writting multiple objects to same file with different threads (or may be single thread, which gets event objects from different threads to be serialized in a file), can be decided later on.

But the challenge i am facing right now is How efficient will be the use of FileChannels in this case and how can i test the scenario of serializing event objects to a file (approx, 5000 per sec to start with) and deserializing all objects accurately without getting StreamCurruptedException.

Any directions on this will be highly appreciated.


Thanks
Akhil
 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE Java MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Akhil Jain:
because i am still getting StreamCurruptedException while deserializing the objects from file.

What changes did you do in the code to avoid this exception?

Originally posted by Akhil Jain:

Strategy for writting multiple objects to same file with different threads (or may be single thread, which gets event objects from different threads to be serialized in a file), can be decided later on.


No, your design will depend totally on the above decision. I think you should have a single class that recieves requests to persist event objects from different threads. Using channels does not make sense to me as ObjectOutputStream does not provide a channel on the stream and putting a work around to use channels, just to get a thread-safe file writing is something non-intutive.
Yeah, there is a bug in ObjectOutputStream that does generate the StreamCorruptException if you want to append to an already exisiting file. Here is the bug parade link
As told earlier, the best and reliable way probably will be to use a messaging system. At least i will suggest that.
[ October 18, 2007: Message edited by: Nitesh Kant ]
 
Scott Bro
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Strategy for writing multiple objects to same file with different threads: one way

class that contains .write method

ReentrantLock rtl = new ReentrantLock(true);

public boolean write(String fileName) {
rtl.lock();

// code to write to file

rtl.unlock();

}
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!