This week's book giveaway is in the JavaScript forum.
We're giving away four copies of Cross-Platform Desktop Applications: Using Node, Electron, and NW.js and have Paul Jensen on-line!
See this thread for details.
Win a copy of Cross-Platform Desktop Applications: Using Node, Electron, and NW.js this week in the JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Serialize many objects to disk..is it possible?  RSS feed

 
Api Thanon
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dear All,
I am writing an app that reads about 1,000,000 records. Each record is transformed to an object and i am applying some transformations to those objects. The final step is to serialize these objects to disk.
I have tried the following :
public class Test {
public static void main(String args[]) {
Test t=new Test();
int capacity=2000000;

GenericObject gen;
try {
FileOutputStream out = new FileOutputStream("theTime");
ObjectOutputStream s = new ObjectOutputStream(out);
for (int i=0;i<capacity;i++) {
gen=new GenericObject();
gen.setValues(i,false,i);
s.writeObject(gen);
}
s.close();
} catch (Exception e) {e.printStackTrace();}
}
..where GenericObject is simply an object that has two ints and a boolean as member variables and the setValues() method basically populates each object with values. The problem is that i am getting a memory exception with that implementation (currently using -Xms200M) is there any workaround?
Thank you in advance!
 
Joe Ess
Bartender
Posts: 9425
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You read in a million records THEN write them to disk? That's your memory problem right there. Perhaps you should go a little easier on the heap and read one record in at a time, process it, write it to disk, then read the next. If you have performance concerns, process blocks of a hundred at a time, just don't do a million!
 
Adrian Yan
Ranch Hand
Posts: 688
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Also, correct me if I'm wrong, I don't think you can serialize int or boolean, since they are primitive. You prolly need to wrap them in Integer, Boolean, etc. Other than that, follow Joe's advise, do 100 at a time, slowly increase that until you find the optimal number of objects you can process.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!