• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • Devaka Cooray
  • Ron McLeod
  • Jeanne Boyarsky
Sheriffs:
  • Liutauras Vilda
  • paul wheaton
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Piet Souris
  • Carey Brown
  • Tim Holloway
Bartenders:
  • Martijn Verburg
  • Frits Walraven
  • Himai Minh

Caching a Large List

 
Greenhorn
Posts: 2
jQuery Eclipse IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have a very large list(Array List) of objects - approximately 50,000 to 60,000 instances of a particular class (which in turn has approximately 9-10 instance variables including Lists, Hash Map).
I need to filter this list as per user request based on 3 parameters.
This list is the life of my application - I need it be available for usage throughout the lifetime of my application. Also keeping the performance standard in mind - I can not afford to hit Database repeatedly. Is there any way so that i can cache this list in an effective manner? I am not permitted to alter the DB schema.

If any further info is needed - please let me know.

Please help. Thanks in advance.
 
Marshal
Posts: 76866
366
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Debasish Mukherjee wrote:I have a very large list(Array List) of objects - approximately 50,000 to 60,000 . . .

that isn’t large. 60000000 would be large.

You should not have to alter the DB schema if you can create a query which will filter the database before you populate your list. That is almost certainly the fastest way to filter that List. And probably the best.
A few other suggestions:-
  • Create a new database and populate it from that List. Run a query to select according to your three parameters on that database.
  • Create a class (Foo) which encapsulates the 3 fields from that List. Populate a Map; I am not sure whether you are better with a Map<Foo, List<Something>> or a Map<Something, Foo>.
  • Create a Comparator<Something> which uses those three fields and put the contents of the List into a sorted set. You can find elements with a binary search.
  • I do not know whether any of those suggestions will help you.
     
    Campbell Ritchie
    Marshal
    Posts: 76866
    366
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    I think this question is too difficult for “beginning”, so I shall move it.
     
    Bartender
    Posts: 10780
    71
    Hibernate Eclipse IDE Ubuntu
    • Likes 1
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Debasish Mukherjee wrote:I have a very large list(Array List) of objects - approximately 50,000 to 60,000 instances of a particular class (which in turn has approximately 9-10 instance variables including Lists, Hash Map).
    I need to filter this list as per user request based on 3 parameters.
    This list is the life of my application - I need it be available for usage throughout the lifetime of my application. Also keeping the performance standard in mind - I can not afford to hit Database repeatedly. Is there any way so that i can cache this list in an effective manner? I am not permitted to alter the DB schema.


    First: If your list/table contains 50-60,000 rows, I highly doubt that database access is going to be that much of an overhead.

    Second: It's also highly likely that your database can sort and filter that list far quicker than you will ever do, because that's what they're designed to do.

    The only thing that caching the List might do is to reduce network traffic between the db and your app; however, there are several other ways of doing that, and I'd want to be absolutely sure that I was solving an actual problem - not a "perceived" one - before I wrote any extra caching layer.
    And the ONLY way to prove that is with metrics.

    W.A. Wulf wrote:More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason — including blind stupidity.



    Winston
     
    Time is the best teacher, but unfortunately, it kills all of its students - Robin Williams. tiny ad:
    the value of filler advertising in 2021
    https://coderanch.com/t/730886/filler-advertising
    reply
      Bookmark Topic Watch Topic
    • New Topic