Bi-directional application level synchronization

[Kris Melotte]

Hello all,
We have a website that uses toplink to connect to a relational oracle database.
We want to do a bi-directional synchronization with another application. The access between both applications will happen over a leased line. The other application I'm talking about is in fact a
consolidation server that has data from several off line applications that is syncronized with the consolidation server at regular times. The offline applications are used for data entry of books.
We want to synchronize operations between the website application server and the consolidation server like:
create a new book, including images (app. server <-> cons. server)
modify a book's properties (app. server <-> cons. server)
delete a book (app. server <-> cons. server)
visitor statistics (app. server -> cons. server)
order a book (app. server <-> cons. server)
As you can see some synchronization needs to be bi-directional and some synchronization is only uni-directional.
We thought to describe the transactions in XML and put them on the disc of the website application server and to create a thread that reads from
an IN directory. A process at the consolidation server put XML files in the IN directory over a SSH connection. Our application puts XML
transactions in the OUT directory.
It's also needed to synchronize jpegs, that will be stored on disk and not in the database, the JPG's will also be put on disk. The XML transaction eveloppe describes the transaction to be done and has references to the images (jpg's) that belong to the transaction.
The transactions execution time can range from 5 sec to 3 minutes.

Is this a good solution? IS SOAP an alternative here? What are the drawbacks/advantages of such a solution?
thanks for any input.
Kris

[William Brogden]

If I understand your situation properly, your communication is all in-house so you might be able to use an XML format that is simpler than SOAP.
If your link does not have to be HTTP, the transport of messages might be a job for Java Message Service or JavaSpaces. The advantage of a messaging approach (instead of RPC) is that either server could go offline for a short period without loss of messages. In any case, synchronization will have to be supplied by your design, SOAP itself does not supply any.
Sounds like a cool project - be sure to take notes so you can write an article about it later!
Bill

[Kris Melotte]

Does this mean that we have to define a DTD or a schema for all the possible transactions?
Messages should be persistent somehow so that they are not lost when the web application server or the consolidation application is down. This was a initial requirement and that was the reason why I was thought about storing XML messages on the disk.
We can indeed choose our transport. Right now we have already uni-directional communication that works with ftp'ing propertyfiles from the consolidation server to the web application server, but this gets too complicated and unmanageable due to the lack of a "contract".
The logic on the consolidation server is all written in Visual basic and not made by our company but by another company.
This means that we and the other company have to agree on some contract (should we choose for a DTD or a schema to define the transaction formats?)
I will study a little bit on JMS and JavaSpaces because I've never worked with these technologies and do not know what the possibilities and limitations are.
best regards and thank you very much for your input.

[William Brogden]

Does this mean that we have to define a DTD or a schema for all the possible transactions?

Oh yes, you can't get away from having something like DTD / Schema or WSDL or IDL to describe all possible messages, but there are utilities to help you create them. Besides, creating the description formally helps you to think about the system architecture.
The requirement for no loss of messages puts you in the Messaging Oriented Middleware (or MOM - love that acronym) area for sure. By good fortune, that is the chapter that is publicly available on the book support site!
http://www.lanw.com/books/javasoap/
Please stay in touch - I think the forum members would be interested in what you come up with.
Bill

[Kris Melotte]

We are still thinking about wether we choose for the message based approach or the RPC approach. The other company wants prefers to use web services, they already use it in their synchronization chain.
If we go for web services, what API should we use? Apache-Soap or Axis?

[William Brogden]

Apache SOAP or AXIS?
I guess that depends on how you feel about using an API that is currently at the "alpha 3" stage of development.
If it was my problem I would join the AXIS users mailing list and monitor the traffic to see what kind of trouble people were having.
http://xml.apache.org/axis/index.html
Bill

[Mrutyunjay Hanchinal]

Hi
I am also facing a problem something similar to Kris Melotte's.
My system is like this
My clients have their offices at different locations say A,B and C.
There is a already developed application, which uses MS Access as its DB. Let me give it a name say MyApp.
This MyApp is deployed in all the three locations A, B and C
Each of the offices create some templates through the use of MyApp which stores the templates in to their DB(MS Access).
There is a central DB(MS SQL) which will be in central office.
The requirement is from time to time the data in all the locations be synchrobized and the same data also be updated in to central DB.
The DBA of this central DB will be provided with a DB admin tool which will be having the provision to scedule a process of synchronizing.
Can anybody suggest me a solution on how to go about solving this problem.
Since this is a sceduled process the messaging seems to be an overhead.
Thanks in anticipation
Mrutyunjay

[Kris Melotte]

If I understand correctly your database schema's are almost identical. You have tools that do that kind of synchronization (even between different database providers.
I know of one (commercial) tool at http://www.creasoft.com . Perhaps there are also open source implementations on the market but I've never seen them.

[v bodani]

is this problem, i mean bidirectional communication, similar to something like a '2-phase commit' ?

[Atul Apte]

Data Synchronization across application databases is an old problem. Some databases have database mirroring capabilities which will work if the database models are identical.
I prefer to look at the design patterns (as described in the Gang of Four Design Patterns book) and specialize a sub set of patterns for data synchronization.
The command pattern, singleton pattern, and visitor pattern are some of the fundamental patterns you can look at if you are developing your own data synchronization solution.
There is a better way, use an application integration framework to manage the data synchronization and future change management will be simpler. Don't just think of solving the current problem, put some effort into future change management too.. as it is bound to happen.
Atul