• Post Reply Bookmark Topic Watch Topic
  • New Topic

Batch Processing  RSS feed

 
Chris Montgomery
Ranch Hand
Posts: 141
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Realizing this could be any number of issues, I'm hoping for some insights as to where to begin...

We have a batch process that spawns several threads. Each thread makes an api call to process some rows in a database.

At first, the transactions per second are quick. Life is good.
However, over time, they begin to slow to the point where it's only one transaction per second.

If we stop the batch processing and start it back up again, things are back to super fast for a while.

Does this scream a particular problem, or could it be any number of issues?

Things I've thought of:
If we aren't releasing connections properly, then I'd expect at one point the process to fail all-together, since the connection pool has 'dried up'. This is not the case in our situation.

My next hunch is the releasing of any objects used during the life of a transaction. Perhaps we have object litter and garbage collection thinks all of these objects are still in use. If my memory is all used up, the slow down could be explained from the swapping that is occurring...

Anything else you can think of?

Thanks!
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Have you observed any other changes over time? A tool like JConsole (Java 5 and later) or JProbe or any number of profilers can tell you if you're leaking memory or starting threads or something else.
 
Chris Montgomery
Ranch Hand
Posts: 141
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
okay, profiler says everything is fine. Windows doesn't show excessive use of memory while process is running.

From a documentation point, it would appear the system is running fine.

Any other thoughts, or is this the point where someone shoud say "I'd need to see some code"...
 
Brian Mozhdehi
Ranch Hand
Posts: 81
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
some code would be helpful. It could be any number of things, including unclosed database RESOURCES such as Statements, etc. Have you checked to make sure that your code always explicitly closes all statements and all resultsets? Failure to do that, even if the connection itself is closed, can cause a burden on the database engine that could cause a slowdown.
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Maybe not much help this time, but note that Windows won't show excessive memory use. It will show something up to the max allowed by your heap and other memory sizes, but won't show how much of that you are using right now. JConsole or some of those other tools will show how much you are using, how much is free, and maybe even how much time you've spent in GC.

On our big AIX boxes we were advised to keep VM memory sizes well below physical memory size to minimize virtual memory swapping. We have 8 gig boxes running two 1 gig JVMs. I was surprised at first but I guess memory is cheap enough that this is a good optimization.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!