• Post Reply Bookmark Topic Watch Topic
  • New Topic

Handling large volumes of data  RSS feed

 
Gary Jones
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
I fire a query which fetches me about 1 million records in resultset.
What approach is correct for handling and processing such a huge resultset.
getting such a huge resultset object put a heavy load on the system jvm and may ocasionally result in OutOfMemory error.Is XML appropriate?
Pl advise.
Regards,
GaryNYC
 
Anton Rechenauer
Greenhorn
Posts: 13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
do you really need 1 million rows?

cant you use aggregate functions, a limit clause and/or pagination?

i guess that producing a plain-text-xml file is "better than" creating 1 million objects (but how should this xml be processed), but without more information what you want to do with 1 million rows in a single ResultSet it's difficult to give advice
 
Gary Jones
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Of course there are turn arounds like pagination etc - but can we do it with a single hit to the database rather than reaching for the db everytime a new page is called.Even an xml stream with 1m rows will be big size.
 
Stefan Wagner
Ranch Hand
Posts: 1923
Linux Postgres Database Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
XML will increase the size drastically - won't it?

Why do you need one million rows in memory?
How big is a row?
If a row is smaller than 100 bytes, you end with less than 100 MB.
What do you want to do with that data?
[ December 11, 2004: Message edited by: Stefan Wagner ]
 
Gary Jones
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I display all the data that i fetch spanned across several pages with page number in the footer.
What i want to avoid is querying the db for data that i display on every page.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!