This makes it kind of hard for me to post a suggestion, since I don't know for sure that it will help. So you should be more careful about how you say things.Originally posted by Rama Krishna:
I am looking for help and help only please.
or else I have to post it somewhere else and get help
What do you mean by " No sense in guessing changes when there are tools that will tell you exactly where the hotspots are?"
This is rather like blackmail, too. First you freeze the discussion and then you complain...
database profiling tools will tell you if indices and other tuning are needed...
..there are JVM profiling tools available to see what your application is up to...
Yes, we have already moved to plain jdbc to migrate. I am not sure if we can use third party tools (SQL*Loader or SQL*Plus and scripts) to migrate as you suggested, because we have lots of logic involved in migration including changed table names, order, and the columns. Each of the new database tables has columns which depend on columns from various other tables. Anyways, I will surely look into those if we these tools can also do what we are doing currently.
select newObjConstructor(someParentObj.childObjaSet, someParentObj.childObjbSet..., some fields,... )
from parent someParentObj,..
where..
Originally posted by Paul Sturrock:
"Coffe Rancher"
Please check you private messages for a message from me.
Originally posted by Rama Krishna:
Thank you for your suggestion.
The problem really occurred because when I left join a parent table with more than one child tables with a one-to-many relationships ...
-Cheers
Rama
Vic Lowtney wrote:Hi all, this thread seems to be the most related to something I discovered while attempting to use hibernate 3.6 for an ETL task processing millions of rows. Since it bubbles up fairly high on Google's results for my search terms, I figured I'd post an update here in hopes that it might benefit others. First, like everyone has discovered, Hibernate is useless and dog slow for repetitive ETL tasks involving many database rows. For my task, we wer getting 1 record per second. We tried many of the suggestions here and elsewhere, e.g., disabling caching, playing around with lazy versus eager fetching, db indexing, etc., all to no avail. Hibernate remained obstinately slow. Here's what we discovered: It was very fast for the first 200-300 records it processed and then steadily declined from there. We tuned and tweaked every hibernate parameter we could find and nothing changed this behaviour. We then tried flush(), evictAll(), and clear() for every 200 rows or so. No workie. The next thing was to destroy and recreate the EntityManager every 200 rows. Despite the setup and teardown overhead of doing this, we went from 1 record per second to about 80 rows per second - much better. We then encapsulated each of batch into its own thread with its own EntityManager and multithreaded the entire process. We're now processing at about 250 rows per second and are only limited by Oracle performance now.
Conclusion: I'm not an expert on the architecture of EntityManager, but it clearly wasn't designed for ETL. It seems to be fine for runtime appserver use, but even in that scenario I will be eyeing EntityManager suspiciously from now on.
I found a beautiful pie. And a tiny ad:
Smokeless wood heat with a rocket mass heater
https://woodheat.net
|