Win a copy of Java Concurrency Live Lessons this week in the Threads forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Sending a large XML file and an ArrayList  RSS feed

 
Serkan Demir
Ranch Hand
Posts: 61
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi everybody,

I am currently designing a system in which large xml files (approximately 1 Gb) are parsing by SAX and values in it are filled a huge cache. I decided to seperate parsing process and caching process into two machines. In the first machine i am parsing the document and filling the values (as Strings) into an arrayList (i am not sure this is the best collection type).
My question is about the transfer of this arraylist to other node, my cache. These two nodes are in local network, what is the best way to send this huge arraylist to other node? socket communication over TCP, RMI, JMS or other alternatives? Performance, round-trip-time is important for me and parsing interval of XML files is 24 hours.

Could you make any suggestion?
Thanks lot,

Serkan
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
1. You should probably think in terms of transmitting a String[] rather than an ArrayList.
2. Why not just serialize the String[] to some shared file location and notify the other application with JMS. That way communication is asynchronous and would still work if the other application is temporarily busy or offline. If you don't delete the previous serialized String[] until the new one has been written, you remove a possible failure point.
Bill
 
Guy Allard
Ranch Hand
Posts: 776
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I agree that shared disk is the best way to handle this. Notification can be made my JMS or even cruder methods: create the output file, then create an empty trigger file.

The other side just watches for the trigger file, then processes the actual data file (with a trigger file delete of course).

If you must go with Network transfer, basic Socket IO will get you the best performance.

Guy
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!