• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

How I can retrive 40 million records from Database?

 
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello all,

please I am trying to retrieve 40 million records from Oracle Database, but my JSF page still loading without retrieve anything.

FYI, I use SQL statement to retrieve these data.

So is there any tools help. (hadoop Apache software not supported for Windows Production, only for development)

Thank you in advance.

Sami.

 
Rancher
Posts: 618
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Because you are using JSF, I assume you want to display the information to a user on a web page. I'll skip asking why a user would want to see 40 million records. Does your app work when you try to display one record? If you post your code, maybe someone can comment on it.
 
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tom Reilly wrote:I'll skip asking why a user would want to see 40 million records.


I won't :-)

As Tom said, so many records should not be handled by the web layer because there's no way for a user to view them. They most likely should not be handled in one batch no matter what: if a record takes just 100 bytes of space then ... well, you do the math :-)
 
Sami Ahmad
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thank you all, every day my customer sending 300000 messages to his customers, So he want to know what he send before 3 month.
let's say the point not see 40 million records, bcz I can make pagging in my JSF page that will retrieve 500 every search.

but when I am trying to make search with some conditions like From (Date), To (Date), Status, etc.... the page still loading without retrieve anything.

FYI.. the page retrieve 2 million records in 1 min .

Thank you again.
 
Ulf Dittmer
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Does the search query work if you run it directly against the DB (using the DB's tools)?
 
Sami Ahmad
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thank you,
I am using Toad, it retrieve 500 row when I make search, I want to know if there tools or software store huge number of data dealing with java language, but NOT hadoop.

Sami .
 
Ulf Dittmer
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I think you should rethink your design. There is no point for the search query to return more records than will be displayed, and IMO 500 records is too many to put on a page meant for human consumption; consider displaying 50 records, or 100 at the most.
 
author
Posts: 4335
39
jQuery Eclipse IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I think for fun you should create a dummy page that just create 40 million rows, repeated without going to the database, and then show this to your customer. If they can load it without their browser crashing and be able to look through it easily, then continue.

This is more of a case of a customer not understanding what they are asking for than a technical issue.

BTW here's an article on Pagination that may be useful. Do your really think when Google says "About 10,200,000 results for your query" they have a system designed, or ever expect, users to iterate through 10+ million results? What it sounds like you need, besides pagination, is a search engine built on top of your data. Granted, in this instance, simple database indexes may suffice.
 
author & internet detective
Posts: 41860
908
Eclipse IDE VI Editor Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I agree that you should be using paging. In addition to Scott's link, see our PaginationOrPaging FAQ page.
 
Sami Ahmad
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thank you guys for your help
 
Proudly marching to the beat of a different kettle of fish... while reading this tiny ad
a bit of art, as a gift, that will fit in a stocking
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic