This week's book giveaway is in the JavaScript forum.
We're giving away four copies of Cross-Platform Desktop Applications: Using Node, Electron, and NW.js and have Paul Jensen on-line!
See this thread for details.
Win a copy of Cross-Platform Desktop Applications: Using Node, Electron, and NW.js this week in the JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Batch processing  RSS feed

 
Jorge Martin
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

is it possible nowadays to have high-volume (1-2 millions of records) batch processing implemented in Java that performs well? I might have to consider migrating batch written in Cobol to Java and I�m trying to figure out what kind of Java architecture could provide an acceptable performance. I would appreciate your comments/ideas on this issue.

Regards
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Java performance in a well tuned system will certainly be as good as any other approach. The easy of creating multi-threaded applications could be particularly handy. Where do the batch items come from and where do they go?
Bill
 
Jorge Martin
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Bill. The processing would involve reading database records, processing them in the server tier (where the business logic resides) and finally updating those records. Another reason for me to think of Java is that processing could also involve in a near future updating database records with XML data received from an external system.

I do know that JDBC has methods aimed at batch processing (addBatch(), executeBatch(), etc), but I don�t know how efficient they would be when dealing with high-volume data. Would you consider these methods as part of a "well tuned Java system"?

Again thanks for your reply.
Martin
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The processing would involve reading database records, processing them in the server tier (where the business logic resides) and finally updating those records. Another reason for me to think of Java is that processing could also involve in a near future updating database records with XML data received from an external system.

So you basically have a "present" problem and a "future" problem and you want the smoothest possible transition.
It sounds to me like your present processing will be limited more by the database configuration and JDBC tuning - an area I never get into. The JDBC forum here at the ranch could help you there.
Your future processing requires XML document processing - something I do alot. Java facilities/toolkits for XML processing are excellent. Separating the XML parsing and JDBC update in separate Threads would seem to offer considerable benefits since both processes do a lot of waiting.
Bill
 
Jorge Martin
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I agree Bill. However, back to your first reply, you seems to suggest that, providing that enough time is dedicated to performance tuning, business logic implemented in Java should not be an issue when considering high-volume data processing. Do you know of any real cases where this has been done with success? I�d be really helpful if I could support this idea with some real figures.
Thanks
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Jorge Martin:
I�d be really helpful if I could support this idea with some real figures.


Than you'd probably most benefit from a spike. Invest a day to code a quick and dirty solution for a simple processing in the order of magnitude you are targeting. It doesn't have to do anything usefull, it just has to show wether you can get the performance you need.
[ April 15, 2005: Message edited by: Ilja Preuss ]
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I agree with Ilja. There are so many variables involved that you could never be sure that any example is an appropriate one. Testing with your actual configuration and mock loads is much better than trying to decide if any given example is appropriate.
Bill
 
Jorge Martin
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks to both of us for your comments and ideas. Putting a simple configuration together sounds good to me as well and it�ll be something to try before making any decission on the issue of migrating.

Regards
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Just for grins, if you're trying the JDBC batch methods make it easy to configure how many inserts you put into a batch before executing it. Doing more frequent commits may help you recover from failures, but I don't know if it might make JDBC faster or slower. Let us know what you find!
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!