Sending a large XML file and an ArrayList

[Serkan Demir]

Hi everybody,

I am currently designing a system in which large xml files (approximately 1 Gb) are parsing by SAX and values in it are filled a huge cache. I decided to seperate parsing process and caching process into two machines. In the first machine i am parsing the document and filling the values (as Strings) into an arrayList (i am not sure this is the best collection type).
My question is about the transfer of this arraylist to other node, my cache. These two nodes are in local network, what is the best way to send this huge arraylist to other node? socket communication over TCP, RMI, JMS or other alternatives? Performance, round-trip-time is important for me and parsing interval of XML files is 24 hours.

Could you make any suggestion?
Thanks lot,

Serkan

[William Brogden]

1. You should probably think in terms of transmitting a String[] rather than an ArrayList.
2. Why not just serialize the String[] to some shared file location and notify the other application with JMS. That way communication is asynchronous and would still work if the other application is temporarily busy or offline. If you don't delete the previous serialized String[] until the new one has been written, you remove a possible failure point.
Bill

[Guy Allard]

I agree that shared disk is the best way to handle this. Notification can be made my JMS or even cruder methods: create the output file, then create an empty trigger file.

The other side just watches for the trigger file, then processes the actual data file (with a trigger file delete of course).

If you must go with Network transfer, basic Socket IO will get you the best performance.

Guy