Win a copy of The Little Book of Impediments (e-book only) this week in the Agile and Other Processes forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

[NX: UrlyBird 1.3.2] Storing data in memory

 
Larry Cha Cha
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't really know the ins and outs of how db's should work.
I have a few questions regarding the db.
1) My 1st question is what should the db be doing with all of the data from the file both when initially launched and when running.
My thoughts are to parse the file and create an ArrayList that holds all of the existing records in memory when it starts. If the user needs to search then the records are all there ready to go.
If a modification to the db needs to be done to I change the records in memory and then dump it all to a new file? or update the record in memory and the file at the same time.
2) Is storing the data as ByteBuffers in the ArrayList suitable? or would it be better to create Record objects with all of the fields as private member variables.
3) Does booking a room mean that the user selects a room as well as entering an id for the 'owner' field and that I just update this field with the owner number in it? (not allowing for 0) as this means the room is free
4) From the Data.java interface I must implement the delete comments say"// Deletes a record, making the record number and associated disk
// storage available for reuse." It sounds like I just remove this record from the file, however there is a valid data field flag that says 0x8000 implies deleted record so do I just change the records flag to this? or delete the record all together.
Thanks for all your help sorry if it's a bit long but I'm just starting out and seem to have so many questions. :/
 
Andrew Monkhouse
author and jackaroo
Marshal Commander
Pie
Posts: 12012
218
C++ Firefox Browser IntelliJ IDE Java Mac Oracle
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Larry
My thoughts are to parse the file and create an ArrayList that holds all of the existing records in memory when it starts. If the user needs to search then the records are all there ready to go

I don't think it is necessary to hold the records in memory. Having the records in memory could cause problems if you had a huge number of records.
I know some members of this forum have looked at using the NIO features to have the database mapped into memory, but this is a slightly different concept than what you are suggesting.
If the user needs to search then the records are all there ready to go.

I don't believe the slight decrease in performance by reading the records from disk when required is not enough to worry us.
Does booking a room mean that the user selects a room as well as entering an id for the 'owner' field and that I just update this field with the owner number in it? (not allowing for 0) as this means the room is free

This is how I interpret the instructions.
However my instructions state that blank means the room is free - therefore under my instructions, 0 could be a valid owner number.
4) From the Data.java interface I must implement the delete comments say It sounds like I just remove this record from the file, however there is a valid data field flag that says 0x8000 implies deleted record so do I just change the records flag to this? or delete the record all together.

I think you only have to change the flag to indicate that the record has been deleted.
Since you are dealing with fixed length fields, the space can then be reused the next time someone creates a record - any record marked as being deleted can be overwritten with the new record. This meets the requirements, and saves you the issue of how to reclaim disk space.
Regards, Andrew
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't think it is necessary to hold the records in memory. Having the records in memory could cause problems if you had a huge number of records.
True that it's not necessary, and it could cause problems. However memory is pretty cheap, and the initial DB size is several orders of magnitude below where you'd actually have a problem. And if we start considering what happens when the number of records gets large, let's also consider the possibility of an increasing number of clients, all hitting "find" simultaneaously. You can improve server performance quite a bit by having data already in memory, not reread every time there's a find() invoked. It's a tradeoff either way, but my gut feeling is that having data in memory is a pretty good way to go. It's pretty easy to switch the code around to do whichever strategy you prefer, IMO. Only one class needs tobe changed (Data.java or whatever it's called in your assignment).
 
Larry Cha Cha
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for both of your thoughts, things are starting to make more sense now, but initially it all looked quite daunting.
What have other people done regarding storing the data in memory or just reading from file each time? I have two diff. views here and am still not certain which to go for.
However my instructions state that blank means the room is free - therefore under my instructions, 0 could be a valid owner number.

Mine says blank too, but doesn't this mean that each of the 8 bytes are 0? or is it the string 'blank'?
I haven't gotten around to looking into the field yet and it will be obvious when I do I guess.
 
Philippe Maquet
Bartender
Posts: 1872
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
I use 3 main techniques to improve performance of the Data class, without keeping all records in memory :
1� I implemented a cache of last accessed records. The cache size is a property.
2� I can create in-memory indexes for any field (indexedFields is a property) and the findByCrieria() method is optimized to take smartly the existing indexes (if any) into account.
3� I maintain a list of deleted records to optimize the create() method and the RecordNotFoundException check.
Hope that helps,
Philippe
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Mine says blank too, but doesn't this mean that each of the 8 bytes are 0? or is it the string 'blank'?
This isn't entirely clear from the instructions. I'd suggest studying the DB file they provide in order to ascertain the most probable way that "no owner" will be indicated. However study the file format carefully. In my case (NX:Contractors) all fields are considered as text (chars), and the instructions allow for the possibility that a field will be all NULL chars (value 0). However in practice, space chars were used for anything "blank". I tried to write the code to allow for either possibility, as the incoming data might change in the future, and I want to accommodate the formal spec as well as the de facto standard usage. Check your instructions carefully though; they are probably different in subtle ways.
Philippe - yes, caching is an option. I'm inclined to not actually code the caching now, but write my "in memory" code so that it could be replaced with a cache. E.g. when I get a record from memory, check if it's null, and if so, read from the file, and put the result im memory. Someone later could put in more code to actually remove older entries - for now, I don't see that as necessary, but want to accommodate its possibility.
 
Larry Cha Cha
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for your suggestions Phillipe and Jim.
I like some of those caching ideas and may consider using them as I've decided not to keep the record in memory (well this week anyway . )
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic