Servlet & file processor design

[Blake Minghelli]

I am developing an XML gateway that will handle various XML messages, posted from a client, with the following characteristics:
1. The XML messages will be sent to a controller servlet for processing
2. The servlet will delegate the processing of each request to a specific handler (POJO) depending on the message type.
3. The data within some messages will be processed during the request and then a response returned to the client
4. Some messages will contain a "file" within a CDATA element of the XML. The files are nothing more than a comma-delimited list of records. Because the file is too large to process within the request/response cycle, it will be stored on the file system and processed later. The reponse sent back to the client will simply be a "I received the file" response.
5. I have to consider things like keeping track of how much of the file has been processed so if it aborts in the middle, the app can start back up where it left off if aborts for any reason. I'll probably handle that via a "file status" table in our db.
I'm trying to figure out the best way to process the files. I have the following thoughts:
1. Create a separate app that exists outside of the web container that processes the files. But I'd like to avoid that.
2. Create a "file processing" thread that is launched during the controller servlet's initialization and continually monitors for new files to process. The advantage here would be that all of the code is running within the web container.
I would appreciate any suggestions or other solutions.
[ April 16, 2004: Message edited by: Blake Minghelli ]

[Stan James]

J2EE container specs tell you not to start or manage any threads. I think some of the newer ones are enforcing this suggestion by causing errors if you start new threads (can anybody confirm that?). I'm working with a vendor package that snds a JMS message to a Message Driven Bean to fire up another thread. I guess submitting another request via HTTP would do the same thing, but JMS is inherently asynchronous and probably cleaner. Seems like a lot of work, but it gets around the "don't start your own thread" rule.
Will your client come back later and ask for status? Do you have to filter so they only see their own status or can they see all jobs in progress? This sounds like a pretty cool little console to write.

[Blake Minghelli]

Thanks for your reply Stan. I was not aware of the "don't start your own thread" rule. We're not using EJBs though so I can't go the message driven bean route. I also thought about sending another HTTP request to handle the file processing but then I'm not sure how I could continue processing a file if the container shuts down for some reason and leaves the file unfinished.

Will your client come back later and ask for status?

Nope.

[C Kutler]

I have never done it myself but would a JMS message server and message queue work?
http://java.sun.com/products/jms/overview.html

[Nicholas Cheung]

All *value-added* functions, like multi-threading, security, transactions, are performed by the container (depends on what container you used). Thus, you noticed that your Servlets never implements Runnable, but the same Servlet can always handle multiple requests.
JMS is a good idea to handle asynchronized requests. Depends on the importance of the messages are, you may need to have some recovery scheme implemented. For example, while the message is NOT finished the processing, but the server downs, does the message still available when the server comes back up? If yes, you need to write some codes for recovery as well. In such case, WebSphereMQ is a good choice as it assures that, once the message is put into the queue, it will be available until it is finished the processing. Of course, MQ is an expensive software, and thus, you need to make a trade off.
Nick

[C Kutler]

Nick,
I have only read about JMS and never had a reason to use it myself, but it seemed to me to be a good solution for stuff I had built in the past (before Java and patterns). Is JMS too complex for simple situations. As for expense, The Sun Application Server is free and has a JMS server.

[Nicholas Cheung]

Is JMS too complex for simple situations.

JMS is not difficult in coding, as well as configuration.
In WSAD, what you need to do is to configure a message factory, with the type of queue, the port number, and maybe you need to configure 2 queues, one for incoming and the other for outgoing.
What the program does is to create the message, and put it into the outgoing queue (incoming queue of the receiver), while the other end is listening to the incoming queue (outgoing queue of the sender).

As for expense, The Sun Application Server is free and has a JMS server.

You mean SunONE server? I dont know think it is free for production machine, you'd better check with the licenses.
Nick

[C Kutler]

Good point about looking at the license. Here is what it says in the J2EE FAQ:

If you are a developer you can build an application with the J2EE 1.4 SDK and then deploy it on the Sun Java System Application Server Platform Edition 8 (included with the J2EE 1.4 SDK or available separately). No matter how many developers are on your team, all of them can use the J2EE 1.4 SDK at no charge. Once your application is ready for production, you can deploy including the Sun Java System Application Server Platform 8 Edition in production on as many servers or CPUs as you want.

I am confused by what is meant by "you can deploy including the Sun Java System Application Server Platform 8 Edition in production on as many servers or CPUs as you want" and I posted feedback on that page asking what they meant.
By the way, the name-du-jour is Sun Java System Application Server Platform Edition 8