Map size

[David Spades]

Hi all,

So, my structure is like this :

Map<String, Map<String,Long>> workHour

this represents the workHour of each employee. So each employee will have a map of Map<String, Long> where String represents the date and long is the working hour at that date. So it's like
workHour :
{
"John Doe":{"2015-01-01":6,"2015-01-02":8,...etc}
"Mary":....
"Jane":....
}

each employee will have 365-366 entries (depends on leap year or not). Will this have a huge toll on memory or other aspect if I have, say..1000 employees (365000-366000 entries)
Thanks

[E Armitage]

Why not use an object oriented design and have an Employee class that holds the desired fields? That way your code to manipulate this data will be easy read and maintain. In general, you should first worry about having a good maintainable design before worrying about memory/performance.

[David Spades]

ok, but object oriented design also will have roughly the same amount of data, so it will be 1000 Employee objects with 365-366 daily entries each. unless I'm missing something here?

[Tapas Chand]

After having a good maintainable design as suggested by E Armitage, you can keep the data in database so there is no need to keep all huge data in RAM at a given point of time.
You need to get only those records to RAM which you will play with.

[David Spades]

ok, so this is for reporting and I did populate those object from database.
The aim here is to minimize DB hits without Cache FW for now (other team is still working on that).
So, is there any side effect on this approach?

[Dave Tolls]

If this is for reporting, what exactly are you doing with the data?

That's a lot of data to stick in a single report, for example.

[David Spades]

so, each assignment has locations and team members --> employees with their worklogs.
this is for attendance and billable report
so, it is a huge report prepared every mid-year for all the activities of all employees.

[Stephan van Hulst]

Do you have any good reason to prevent DB hits if the reports are generated twice a year?

[Tapas Chand]

David Spades wrote:so, it is a huge report prepared every mid-year for all the activities of all employees.

I hope you are not trying to view the whole report using JSP.In that case JVM will cry(die).
As far as I know such reports are normally generated in Oracle (atleast we use).
JSP is only used to trigger the stored procedure if trigger requires manual intervention.
Data can be exported to some kind of file from Oracle.

[Campbell Ritchie]

Why are you using a String for the date? Why do you need a date at all? You can use an array for hours worked. I believe you can use long[] as a type in generics, but you can't use long (small L).
Why are you using a Long for hours worked? Unless you expect each of your staff to work > 2³¹ hours daily? Even Ebenezer Scrooge didn't expect Bob Cratchit to work that long.

[David Spades]

Campbell Ritchie wrote:Why are you using a String for the date? Why do you need a date at all? You can use an array for hours worked. I believe you can use long[] as a type in generics, but you can't use long (small L).
Why are you using a Long for hours worked? Unless you expect each of your staff to work > 2³¹ hours daily? Even Ebenezer Scrooge didn't expect Bob Cratchit to work that long.

2 words that all developers understand all too well --> legacy code
it was stored that way since the beginning of time, I suspect from back in 2000 and now, apparently I'm it.

[Campbell Ritchie]

And you can't find who decided to use Strings for dates and shoot them?

[David Spades]

nope, since they're the ones hiring my team.
they said that using string is enough at that time and easier to manipulate using joda time. used string in anticipation of using other datetime framework.

[Campbell Ritchie]

It may be possible to copy the entire database to a new database converting the Strings to dates as you go.

Maybe shooting would be too kind. Something more lingering involving boiling oil or honey and fire ants might be more appropriate

[David Spades]

problem is, we're given a new box to put our application and the data resides in other box which is not under our control, so....
that's why I've been asking around about ehcache
https://coderanch.com/t/651517/ehcache/caching/ehcache-store-data

I'm only allowed to implement any kind of workaround so long I put it in my box and not changing anything in other boxes. oh well, if it's easy, anybody can do it right? that's why there's always obstacle, be it technical or "non-technical"

[E Armitage]

Instead of fetching records into your 'box',s memory, rather create a database instance on that box and setup replication so that your box is a slave that gets updated with data (setting this up is trivial for databases like mysql ). Then use database mechanisms to build the report and write it to disk.

[Karthick Pattabiraman]

David Spades wrote:problem is, we're given a new box to put our application and the data resides in other box which is not under our control, so....
that's why I've been asking around about ehcache
https://coderanch.com/t/651517/ehcache/caching/ehcache-store-data

I'm only allowed to implement any kind of workaround so long I put it in my box and not changing anything in other boxes. oh well, if it's easy, anybody can do it right? that's why there's always obstacle, be it technical or "non-technical"

What is the DB that you are using?

If its oracle then the data is already cached in the memory in your other box (I don't know about other DB but they should also work the same way).

When returning result set from Oracle DB you just return as ref cursor. Refcursor is pointer to the cursor that need to be executed.

And Java just execute it and gets the data. Java should not be storing the fetched data in a collection, it should be directly sending it to the destination (Report - display or a file etc).

[David Spades]

we're using mongoDB for this one and I think creating database is not feasible since tables involved are transactional.