• Post Reply Bookmark Topic Watch Topic
  • New Topic

I need to serialize a 2gb file  RSS feed

 
Nirmal Mekala Kumar
Ranch Hand
Posts: 63
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have a big file that is 2gb and might even exceed. Will there be ne probs if i serialize it. Can u suggest ne other ideas.
 
Ernest Friedman-Hill
author and iconoclast
Sheriff
Posts: 24217
38
Chrome Eclipse IDE Mac OS X
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What do you mean exactly by "serializing a file?" In Java, serialization is something you do to an object, and a file isn't an object. What do you hope to achieve?
 
David O'Meara
Rancher
Posts: 13459
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ni!
 
Nirmal Mekala Kumar
Ranch Hand
Posts: 63
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
sorry for the ambiguity ... i meant tht, can jvm handle such huge objects in the memory. What i need is a Collection (a hashmap or hashtable) that maps a file code to its URL. this collection should store information of atleast 4 million documents.
 
David O'Meara
Rancher
Posts: 13459
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I appologise for my previous reply, I was just amusing myself.

I was trying to indicate that we prefer users to UseRealWords and try to avoid abbreviations like ne, ur, tht and so on. We don't charge by the letter!

Dave
 
David O'Meara
Rancher
Posts: 13459
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If you're just storing file codes and URLs you don't really need the file in memory, do you? This should make the memory requirements vastly smaller.
 
Ernest Friedman-Hill
author and iconoclast
Sheriff
Posts: 24217
38
Chrome Eclipse IDE Mac OS X
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sun's 32-bit JVMs can hold somewhat less than 2GB worth of data in memory. Further, because characters are held as 16-bit Unicode values, a disk file's size might actually double when read in as a Java String. So serialization issues aside, you may have problems just assembling this data structure in the first place, unless you have 64-bit hardware.

Consider putting the data into a relational database (making sure, of course, that it can handle that volume of data) and manipulating it from Java without loading it all in.
 
Nirmal Mekala Kumar
Ranch Hand
Posts: 63
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you guys. It helped a lot. I guess I was too ambitious to increase the performance and wanted to eliminate the database connection overhead.
...and David I will take care not to use abbreviations again. I guess you noticed it already in this post ;-)
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!