• Post Reply Bookmark Topic Watch Topic
  • New Topic

paging with nio and memory efficiency

 
Binu K Idicula
Ranch Hand
Posts: 99
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi All,
I have been using hashmap for storing objects with their associated keys. But as you all know, it will dump core if my runtime needs are more than what JVM can allocate. Is there a way, I use NIO and overcome this problem by a cache mechanism with paging?
Thank you in advance.
regards
Binu K Idicula
 
Joe Ess
Bartender
Posts: 9362
11
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't know any easy way in Java to map a hashmap to out-of-memory storage. NIO does have memory-mapped files, but that's not going to get you the key/value relationship you need.
I'd say your options are between rolling your own database-like structure using RandomAccessFile (which can introduce some non-trivial issues like what to do with updates/deletes) or using an embedded database.
 
Binu K Idicula
Ranch Hand
Posts: 99
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you for the quick reply.

May be I could use, mykey -> filepointer as a hashmp instead of storing mykey -> myjavaobject. Here myjavaobject is a String( an XML String) which has a defined overall maximum length. This might generate some file holes, but may be better than my current hashmap.

Although the hashmap entries can still cause problem, but entire program memory handling may become efficient.

If I use RAF, I might not get all benefits of NIO. ( Is my assumption correct?)

Thank you in advance.
Binu K Idicula
 
Joe Ess
Bartender
Posts: 9362
11
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Binu K Idicula:
If I use RAF, I might not get all benefits of NIO. ( Is my assumption correct?)


What specific benefits are you thinking of? NIO introduced many previously-unavailable features to the API. Some of these features make significant performance enhancements possible. At the time, the regular java.io classes were rewritten to take advantage of NIO where possible. Chances are, unless you can name a specific feature of NIO that you require (i.e. non-blocking sockets for a server app), you will not be missing out using regular IO.
 
Binu K Idicula
Ranch Hand
Posts: 99
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Joe Ess,
I have only one requirement here. I donot want entire objects to be loaded into hashmap. Only a few are accessed mostly. If I use a hashmap for storing object key and object as value, it will fail to grow beyond a limit. If I use nio based memory mapped file as my cache which stores entire object in file, I hope since nio is having paging, the capacity of the program will increase greatly. In hashmap I could store file pointer instead of entire object. non-blocking io as I understand gets me better performance as well.
I will try this.
regards
Binu K Idicula
 
Binu K Idicula
Ranch Hand
Posts: 99
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
can anyone give me pointers to third party tools which provides caching mechanism with paging?

regards
Binu K Idicula
 
Joe Ess
Bartender
Posts: 9362
11
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Binu K Idicula:
In hashmap I could store file pointer instead of entire object.

This plan sounds very much like the RandomAccessFile-based database I linked to above. Did you read it?
The "increased capacity" of your program has nothing to do with NIO and everything to do with moving data out of memory. This is actually the opposite of caching, which is, generally speaking, keeping data in memory for quick access.
[ April 09, 2008: Message edited by: Joe Ess ]
 
Binu K Idicula
Ranch Hand
Posts: 99
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, caching is not the right word. But I would say paging would fit here. Anyway "Random Access File" based database seems to be logical for the solution. I would cross check the performance hit.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!