• Post Reply Bookmark Topic Watch Topic
  • New Topic

Caching a Large List  RSS feed

 
Debasish Mukherjee
Greenhorn
Posts: 2
Eclipse IDE Java jQuery
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have a very large list(Array List) of objects - approximately 50,000 to 60,000 instances of a particular class (which in turn has approximately 9-10 instance variables including Lists, Hash Map).
I need to filter this list as per user request based on 3 parameters.
This list is the life of my application - I need it be available for usage throughout the lifetime of my application. Also keeping the performance standard in mind - I can not afford to hit Database repeatedly. Is there any way so that i can cache this list in an effective manner? I am not permitted to alter the DB schema.

If any further info is needed - please let me know.

Please help. Thanks in advance.
 
Campbell Ritchie
Marshal
Posts: 56536
172
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Debasish Mukherjee wrote:I have a very large list(Array List) of objects - approximately 50,000 to 60,000 . . .
that isn’t large. 60000000 would be large.

you should not have to alter the DB schema if you can create a query which will filter the database before you populate your list. That is almost certainly the fastest way to filter that List. And probably the best.
A few other suggestions:-
  • Create a new database and populate it from that List. Run a query to select according to your three parameters on that database.
  • Create a class (Foo) which encapsulates the 3 fields from that List. Populate a Map; I am not sure whether you are better with a Map<Foo, List<Something>> or a Map<Something, Foo>.
  • Create a Comparator<Something> which uses those three fields and put the contents of the List into a sorted set. You can find elements with a binary search.
  • I do not know whether any of those suggestions will help you.
     
    Campbell Ritchie
    Marshal
    Posts: 56536
    172
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    I think this question is too difficult for “beginning”, so I shall move it.
     
    Winston Gutkowski
    Bartender
    Posts: 10575
    66
    Eclipse IDE Hibernate Ubuntu
    • Likes 1
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Debasish Mukherjee wrote:I have a very large list(Array List) of objects - approximately 50,000 to 60,000 instances of a particular class (which in turn has approximately 9-10 instance variables including Lists, Hash Map).
    I need to filter this list as per user request based on 3 parameters.
    This list is the life of my application - I need it be available for usage throughout the lifetime of my application. Also keeping the performance standard in mind - I can not afford to hit Database repeatedly. Is there any way so that i can cache this list in an effective manner? I am not permitted to alter the DB schema.

    First: If your list/table contains 50-60,000 rows, I highly doubt that database access is going to be that much of an overhead.

    Second: It's also highly likely that your database can sort and filter that list far quicker than you will ever do, because that's what they're designed to do.

    The only thing that caching the List might do is to reduce network traffic between the db and your app; however, there are several other ways of doing that, and I'd want to be absolutely sure that I was solving an actual problem - not a "perceived" one - before I wrote any extra caching layer.
    And the ONLY way to prove that is with metrics.

    W.A. Wulf wrote:More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason — including blind stupidity.


    Winston
     
    It is sorta covered in the JavaRanch Style Guide.
    • Post Reply Bookmark Topic Watch Topic
    • New Topic
    Boost this thread!