• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • paul wheaton
  • Jeanne Boyarsky
  • Ron McLeod
Sheriffs:
  • Paul Clapham
  • Liutauras Vilda
  • Devaka Cooray
Saloon Keepers:
  • Tim Holloway
  • Roland Mueller
Bartenders:

Which architecture?

 
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

my scenario:
A lot of XML-Files delivered via a directory should be transformed into a specific ASCII-File.
Ok. the amount of the XML-Files could be huge, perhaps hundreds per second and should be transformed as fast as possible (nearly realtime).

The transforming should be easy and is not the problem, but which architecture should i use?
Because of the huge amount of transformations the app/service should be scalable and absolutely stable.
I have no clue how to implement it (webservice, servlet on an webcontainer like tomcat etc.) for best performance.

Cheers
Tom
 
author
Posts: 14112
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Sounds like just a plain Java application should be able to do the job.

For the transformation itself, you might want to consider using XSL-T.
 
Tom Bowen
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello Ilja,

you are right, the core of the app are a few classes.
But if we need more performance we want to add another pc('s) to get enough perfomance and stability (perhaps one pc fails etc.)
How could i implement the scalability and stability in an easy way?

Regards
Tom
 
Ranch Hand
Posts: 239
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
How about spawning multiple threads to do the job since this involves lots of IO?
 
Tom Bowen
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello Rajah,

ok. But how do i distribute or orchester this multiple threads on multiple machines?
Is there a suitable framework (spring etc.) or lib to do this easy and stable?

Regards
Tom
 
Rajah Nagur
Ranch Hand
Posts: 239
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by tommeltom:
Hello Rajah,

ok. But how do i distribute or orchester this multiple threads on multiple machines?
Is there a suitable framework (spring etc.) or lib to do this easy and stable?

Regards
Tom



The ideal solution for your problem would be - to have a distributed JVM which can run multi-threaded Java application transparently.
But as of now there is nothing of this sort.

Please explore the possibility of JINI or JavaSpaces solution.
 
Bartender
Posts: 1638
IntelliJ IDE MySQL Database Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi "tommeltom ",

Welcome to the JavaRanch. On your way in, you may not have noticed that we have a policy on screen names here at the ranch. It must consist of a first name, a space, and a valid last name. It must also not be fictitious.

Unfortunately, your screen names do not seem to conform with this policy. Please take a moment to change them.
You can change your screen name here
 
Tom Bowen
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
First: Thanks all

Another idea:
I make a plain java app which copies an xml-file from the given directory, processes it and store it in the output-directory.
Perhaps this process could be run on several threads on one machine.
If i had a performance problem i add another machine with the same app.

So there is a possibility of deadlocks, two or more threads are taking the same xml-file.
How could i avoid such a deadlock?

Regards
Tom
 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE MySQL Database Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

tommeltom: A lot of XML-Files delivered via a directory should be transformed into a specific ASCII-File.



I did not get this. You mean you scan a shared/local directory to get a few xml files that you have to transform?

If this is what you need then a servlet or a web service will not help you as a both of them need a client to invoke them. (This can be automated i.e. write a client that periodically polls these services, but it will be more of a hack than anything else.)
For scheduled jobs, you can use a library like quartz that can also be deployed as part of a J2EE application or in a servlet container. Thus, enabling you to use the clustering and fail over capabilities of these servers for scalability and fault taulerance.
 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE MySQL Database Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tom:
So there is a possibility of deadlocks, two or more threads are taking the same xml-file.
How could i avoid such a deadlock?


Well it is not called a deadlock.
To solve this you can use FileLock to ensure that only one process(JVM) gets hold of the file.
(I could have provided you with the code as to how to use FileLock but i thought you will learn more if you try to discover yourself.)
[ May 20, 2008: Message edited by: Nitesh Kant ]
 
Ilja Preuss
author
Posts: 14112
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Nitesh Kant:

To solve this you can use FileLock to ensure that only one process(JVM) gets hold of the file.



Note this from the javadoc:


Whether or not a lock actually prevents another program from accessing the content of the locked region is system-dependent and therefore unspecified.



An alternative is to implement your own locking system. For example, before you start to process a file somefile.xml, you could try to create a file somefile.xml.lock. Only one process will be able to actually create that file - all others will get an exception and therefore know to skip it.
 
Tom Bowen
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
As always: Thank you all, very kind.

The idea with the lock is good, but if the machine crashes and the files are locked?
Hmm, a selfimplemented locksystem could persist a machineID or something else and then after restarting the machine first looks if there are locks with this machineID and unlocks this files.

Any ideas?

Regards
Tom
 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE MySQL Database Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Its true that FileLock is os-dependent as it directly maps to the native file locking capabilities of an OS.
However, it will be useful in applications where the intended OS is known and a small test program can validate whether OS supports the same. (Windows and redhat linux supports it)

In my opinion, implementing custom lock support(using a lock file) has a few issues like what tom has mentioned i.e. system crash will leave the file non-processable by other processes. Of course, one can think of a timed lock(based on a number much greater than the expected maximum processing time) but again its some amount of work.
On the other hand, the FileLock will be released after the JVM exits/crashes. So, the additional burden of worrying about system crash and file being locked till eternity does not exist here.
 
Ilja Preuss
author
Posts: 14112
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Yes, if you can control the environment, it would make sense to first try it with the FileLock approach, and only implement a custom locking system when that fails.
 
Tom Bowen
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Many thanks guys!

Ok, i try it with the "Java Filelock" and a plain multithreaded java-app.

Once again: Many thanks!
 
Can you hear that? That's my theme music. I don't know where it comes from. Check under this tiny ad:
Smokeless wood heat with a rocket mass heater
https://woodheat.net
reply
    Bookmark Topic Watch Topic
  • New Topic