• Post Reply Bookmark Topic Watch Topic
  • New Topic

OutOfMemoryError while Data above 16 MB  RSS feed

 
Nikhil Vasaikar
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
I am reading from a muliplte resultset in nested loops and writing to a file. This is a new XML file that i am creating using FileOutputStream and then writing to it. I can write upto 16MB of data and beyond that it gives me an OutOFMemory Error. The sample write statement is as follows
fos.write(("<ITDW>" + mainRS.getString(5) + "</ITDW>").getBytes());
I would appreciate suggestion to write a file upto 100MB which is my requirement.
Thank You
 
Adrian Yan
Ranch Hand
Posts: 688
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How much memeory does your system has? This is really strange, cause the stream doesn't keep the entire file, it writes eveything in its buffer and discard them. Can you post more than just that lines of code? Also, what are the size your resultset?
 
Nikhil Vasaikar
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
System Memory - 128MB
Three ResultSets-
Second and Third have Child records of First RS
At failure records in teh resultset were
First RS - 4000
Second RS - 8000
Third RS - 70000
Max file size so far created - 16MB
 
Adrian Yan
Ranch Hand
Posts: 688
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
can you give me abit more code? looks like some objects are being handle properly.
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ummm... is your result for column 5 really, really long? Try breaking this up to see exactly which operation is throwing the error:

While we're at it, wrap the FileOutputStream in a Writer so you don't have to create these extra objects. If str is really huge, then msg and bytes are also huge - there's no need for the second two to ever be created if you're careful:

Of course, replace UTF-8 with whatever encoding you wish to use for this file - or omit encoding to use system default. (Whihch varies from machine to machine, so beware sharing files unless you know what encoding is to be used.)
If the str string is not really big, then these changes may not make much difference, and your problem is somewhere else. But it's worth checking...
[ October 10, 2003: Message edited by: Jim Yingst ]
 
Nikhil Vasaikar
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Here's the code snipplet that is doing the IO
FileOutputStream fos = null;

try
{
parentRS = getParentData();
System.out.println("Got parentRS RS");
child1RS = getChild1Data();
System.out.println("Got child1RS RS");
child2RS = getChild2Data();
System.out.println("Got child2RS RS");

/******** Generate the XML Document ************/
fos = new FileOutputStream(filename);
fos.write(("<?xml version=\"1.0\" encoding=\"UTF-8\"?>").getBytes());
fos.write("<DATA>".getBytes());
while (parentRS.next())
{
String itId = parentRS.getString(1);
fos.write(("<A V=\"" + parentRS.getString(1) + "\">").getBytes());
fos.write(("<B>" + parentRS.getString(2) + "</B>").getBytes());
fos.write(("<C>" + parentRS.getString(3) + "</C>").getBytes());
fos.write(("<D>" + parentRS.getString(4) + "</D>").getBytes());
fos.write(("<E>" + parentRS.getString(5) + "</E>").getBytes());

while(child1RS.next())
{
if(!itId.equals(child1RS.getString(1)))
{
child1RS.previous();
break;
}
fos.write("<W>".getBytes());

fos.write(("<G>" + child1RS.getString(2) + "</G>").getBytes());
fos.write(("<H>" + child1RS.getString(3) + "</H>").getBytes());
fos.write(("<I>" + child1RS.getString(4) + "</I>").getBytes());
fos.write(("<J>" + child1RS.getString(5) + "</J>").getBytes());
fos.write(("<K>" + child1RS.getString(6) + "</K>").getBytes());
fos.write(("<L>" + child1RS.getString(7) + "</L>").getBytes());
fos.write(("<M>" + child1RS.getString(8) + "</M>").getBytes());

fos.write("</W>".getBytes());
}


while(child2RS.next())
{
if(!itId.equals(child2RS.getString(1)))
{
child2RS.previous();
break;
}
fos.write("<N>".getBytes());

fos.write(("<O>" + child2RS.getString(2) + "</O>").getBytes());
fos.write(("<P>" + child2RS.getString(3) + "</P>").getBytes());
fos.write("<Q>".getBytes());
fos.write(("<R>" + child2RS.getString(4) + "</R>").getBytes());
fos.write("<S>".getBytes());
fos.write(("<T>" + child2RS.getString(5) + "</T>").getBytes());
fos.write(("<U>" + child2RS.getString(6) + "</U>").getBytes());
fos.write(("<V>" + child2RS.getString(7) + "</V>").getBytes());
fos.write(("<W>" + child2RS.getString(8) + "</W>").getBytes());
fos.write(("<X>" + child2RS.getString(9) + "</X>").getBytes());
fos.write(("<Y>" + child2RS.getString(10) + "</Y>").getBytes());
fos.write(("<Z>" + child2RS.getString(11) + "</Z>").getBytes());
fos.write(("<UP>" + child2RS.getString(12) + "</UP>").getBytes());
fos.write("</S>".getBytes());
fos.write("</Q>".getBytes());

fos.write("</N>".getBytes());
}
fos.write("</A>".getBytes());
}
fos.write("</DATA>".getBytes());

}
finally
{
if ( fos!=null ) fos.close();

parent.cleanUp(parentRS);
child1.cleanUp(child1RS);
child2.cleanUp(child2RS);

ConnectionManager.closeConnection();
}
 
Nikhil Vasaikar
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Is there a better efficient way to write to a file than what i have coded in the above post?
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Did you read my previous post?
 
Adrian Yan
Ranch Hand
Posts: 688
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, nikhil, you need to look at Jim's post. Outofmemiry error usually doesn't happen when you do a write. So just double-check your resultset, and use a buffered IO stream.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!