• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • paul wheaton
  • Paul Clapham
  • Ron McLeod
Sheriffs:
  • Jeanne Boyarsky
  • Liutauras Vilda
Saloon Keepers:
  • Tim Holloway
  • Carey Brown
  • Roland Mueller
  • Piet Souris
Bartenders:

Heap size increases when writing XML to the File

 
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am converting an object (ArrayList) in XML and then storing that XML as a file on local machine. When there are large number of data, the XML takes too long to write in a File or doesnt write at all (gives some exception).

xmlSource = xmlConverter.convert(object); // object here is no.of ArrayLists..

FileWriter fw = new FileWriter(f);
fw.write(xmlConverter.getXML());

fw.write() takes too much time. Is there any other way which doesnt eat up so much memory ???
 
Rancher
Posts: 13459
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
wrap it in a BufferedWriter is the first step

PrintWriter out
= new PrintWriter(new BufferedWriter(new FileWriter("foo.out")));
 
batuk chatuk
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
thanks!!!

File f = new File("foo.xml");
PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter(f)));
pw.write(xmlConverter.getXML(obj));

Using BW does speed up the process by almost 40%, although the PC remains in a numb state during that time..but it does work..

I m afraid this solution is not gonna be enough as there are around 100k entries on Production env at max.. ( I checked above sol. on 50000 records..)

Is there any possibility that i split the records and then in chunks i add them in "same file". (dont want to create more than 1 XML for one page..)

Thanks again...
 
batuk chatuk
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Its Done!!!

File writing is happening totally with all data at a time in above code..
So this is the answer to better and memory friendly code...

xmlSource = xmlConverter.convert(obj);
BufferedReader bufferedReader = new BufferedReader(xmlSource.getReader());
PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter(f)));
int count = 0;
int buffer=1024;
char[] c1=new char[buffer];
while((count = bufferedReader.read(c1, 0, buffer)) != -1){
pw.write(c1, 0, count);
}
pw.flush();
pw.close();
 
batuk chatuk
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
For 100K records or more the above code didnt work on my machine.. ( i had kept JVM Heap size to Max. 1300K.. )
The reason was the XMLConverter was not able to convert that much data into XML So i wrote the code to divide all the ArrayLists in chunks and then convert them and write them into the same file..

FileWriter fw = new FileWriter(new File("foo.xml , true);

the flag in the constructor is for Appending the new data to same file..
and so file can be generated without any trouble..

Best Regards
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic