Win a copy of Beginning Java 17 Fundamentals: Object-Oriented Programming in Java 17 this week in the Java in General forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Tim Cooke
  • Campbell Ritchie
  • Ron McLeod
  • Liutauras Vilda
  • Jeanne Boyarsky
Sheriffs:
  • Junilu Lacar
  • Rob Spoor
  • Paul Clapham
Saloon Keepers:
  • Tim Holloway
  • Tim Moores
  • Jesse Silverman
  • Stephan van Hulst
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Piet Souris
  • Frits Walraven

How to effectively manage data in-memory

 
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi.

Pretty sure this is intermediate level too, but would prefer more of the advanced peeps to give some feedback.

I have an application that needs in-memory access to a lot of data. Basically this data is a large (10k's) of EJB3 entities, which are, as you know, just POJOs.

So in essence we just have a large collection of POJOs.

Each of these needs to be accessed quickly given a set of params. E.g. Month, Year, Code, Some Other Code.

So I've plumped them all in a large Map for very quick retrieval.

i.e. myObject = map.get(month).get(year).get(code).get(someOtherCode);


Obviously this becomes a bit of a mess putting this data in the Map, keeping it up-to-date, and it looks pretty awful.
It works very well. Everything is in-memory and very fast. It just doesn't look and feel right.

What's a more OO way of doing this? Any suggestions on a better alternative which isn't so cumbersome?

I'd appreciate your feedback and idea.

Regards,

Dean.
 
Sheriff
Posts: 67620
173
Mac Mac OS X IntelliJ IDE jQuery TypeScript Java iOS
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
[quote=Dean Pullen]Pretty sure this is intermediate level too, but would prefer more of the advanced peeps to give some feedback.[/quote]Please post in the appropriate forum. I can assure you that the same people will see it.
 
author and iconoclast
Posts: 24203
44
Mac OS X Eclipse IDE Chrome
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
What you're doing is likely to be the fastest way. An alternative is to use an in-memory database. Java OODBMSs seem to be in short supply these days; it's a fad that comes and goes. But HSQLDB has an in-core relational database engine. It would give you more flexible querying than what you have now, and it might possibly be a little better on memory.
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for the replies so far.

Hm the HSQLDB option sounds interesting, I'd not thought/heard of utilising it in that matter.

I also thought of using something like JBoss Cache or similar, but haven't read too much into it. Though I suspect this wouldn't really get around the data structure problem that's already inherent.
 
Sheriff
Posts: 22579
122
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
But can a HSQLDB database contain arbitrary Java objects?

If not, you may want a two-way lookup - each row has a unique (long?) identifier, which you associate with an object using a Map<Long,Object>. That would almost certainly make it slower than the map-map-map-map solution.


If you really dislike the map-map-map-map solution only because of the messiness and up-to-date-ness, you should consider encapsulating it in its own class. The lookup method takes all of the fields you want to search for:
 
Ernest Friedman-Hill
author and iconoclast
Posts: 24203
44
Mac OS X Eclipse IDE Chrome
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Rob Prime wrote:But can a HSQLDB database contain arbitrary Java objects?



Nope, you'd have to use ORM. I didn't say it would be faster, I said it would be more flexible and maybe use less memory (assuming the unused objects are discarded.)
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Maybe Im missing something here, but can all these fields be combined into a single key?
If there is a unique object for "1939.10.code.someothercode" why not just one map?

Bill
 
Rob Spoor
Sheriff
Posts: 22579
122
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You may want to use a new class for that; splitting strings back is usually not fun.

As long as you override equals and hashCode properly it should work just as fine.
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Bill's point about the single key - I hadn't even thought of it. *smacks head*

However, sometimes we want all objects that match a certain part of the (ultimately) unique key.

E.g. all objects with a certain year and month, but not specifying 'code' and 'someothercode'.

How would you then get these specific objects from the Map?

 
Rob Spoor
Sheriff
Posts: 22579
122
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
In that case my earlier suggestion of an in-memory database combined with a map seems useful.

The database has one table that has one column for each search field, plus a unique ID (autoincremented). The map maps from those IDs to the actual object.

The lookup can specify a where-clause for all fields you want, or omit fields you don't want. In pseudo code:

Granted, it is slower when all fields are filled in. But because it is an in-memory database it may be faster when omitting more fields. With the map-map-map-map solution, you will have to iterate over the maps in some levels.
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Took me a few glances to understand what you meant, but you're basically saying to:

a) Hold a Map in memory key'd by a unique ID that is the unique ID of an element within the (in-memory) DB

b) Utilise the DB to find the ID, then retrieve from the Map

Aren't you now holding the information in both the map and the DB, thus doubling (well, increasing) memory requirements?
 
Rob Spoor
Sheriff
Posts: 22579
122
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Yes, but it is possibly the fastest solution.

If you don't need to search on all fields sometimes, you'll have to make a trade-off: either memory increases (DB + map or something similar), or speed decreases (looping through at least several maps, or even a list if all search fields are part of the POJOs).
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Ok thanks for the suggestions - all interesting reading.
 
WHAT is your favorite color? Blue, no yellow, ahhhhhhh! Tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
reply
    Bookmark Topic Watch Topic
  • New Topic