Design question : Is Java the right solution for this problem?

[joshua van-breukelen]

Hi everyone:
Background Info: Back-end loads are handled via proprietary software and accompanying customized scripts. Scripts are written using vendors own language, Hi everyone:
Background Info: Back-end loads are handled via proprietary software and accompanying customized scripts. Scripts are written using vendors own language, which is based off of the C language. Among other things, the scripts massage data from the flat files and insert/update the DB as appropriate.
Problem Set: "New" flat file will be almost 1/2 gig of data. Fields/data within the file is/are convoluted based on old legacy systems processing mechanisms. Vendor’s proprietary software may not be appropriate – may have to implement an alternative solution
Possible high-level design solution: Is it appropriate to consider Java for back-end load or batch type operations? The basic idea would be to write a Java app that does read-massage-write to DB for large amount of data. Based on the extremely large amount of data that must be read, massaged and processed is writing a Java "Processor-Loader" the way to go? Are there any open-source "Java-Loader-Engine APIs" out there?
Comments/Suggestions welcome.
Thanks,
Joshua

[Kevin Arnold]

What kind of database is it ? I have seen this sort of thing done for import into an Oracle database entirely in PL SQL. This flat file is pretty big too. File from each business partner spans 3 to 4 CDs. It does a pretty decent job of massaging and uploading close to 16GB of data overnight. You might find Java considerably slower. You can do similar stuff in Transact SQL too (MS SQL Server). Hope this helps.
Kevin

[Lasse Koskela]

Technically Java is up for the task. However, as Kevin said, Java would probably perform worse than some not-so-portable solutions.
I would consider the skill base of your company to be the primary indicator of whether Java is a good solution.
If you don't have staff competent in PL/SQL, TransactSQL or whatever language you consider using, think carefully about the real costs of gaining that competence. I wouldn't be surprised to see the project end up being late with a relatively large amount of defects if the project has been used as a test-bed for learning a new technology.
However, because Java probably performs worse, you need to compensate that with more hardware--if worse comes to worse, you need to distribute the processing, which in turn adds to complexity.

[Kevin Arnold]

Originally posted by Lasse Koskela:
I would consider the skill base of your company to be the primary indicator ......think carefully about the real costs of gaining that competence. .........the project has been used as a test-bed for learning a new technology.

Doh! I keep forgetting that one.
Excellent point