Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

OOM  RSS feed

 
Mary Cole
Ranch Hand
Posts: 362
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

Am using HttpClient to make a connection to a remote URL to get the file as byte[]. Am getting OOM error when the file size exceeds 10 MB. Is there any alternate way to avoid this OOM



I think coz of the large amount of data in memory, OOM occurs....any suggestions pls..

Thanks in advance
[ April 25, 2007: Message edited by: Mary Cole ]
 
Vlado Zajac
Ranch Hand
Posts: 245
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Don't read all data into memory, if it is not necessary.
(Use getResponseBodyAsStream().)

Or increase maximum heap size for JVM with -Xmx parameter.
 
Mary Cole
Ranch Hand
Posts: 362
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for your reply. Even if I use getResponseBodyAsStream, am getting the same error. Can you pls elaborate on Don't read all data into memory, if it is not necessary
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It looks like the byte[] array "data" contains the entire content of the file. If the file is too big, then that array will be too big. So if there's any way to avoid putting all the data in a single byte[] at one time, do it. For example, you might grab a little bit of the stream at a time, and write the data from the stream to a file:

Here we still use a byte[] array, but its size is limited to 4096. We just keep filling it and writing it as many times as necessary.

It really depends on what you need to do with the data. Often it's possible to process the data in small chunks, as above. Other times it's much easier to have all the data in memory at once. But for large files, try to avoid putting all the data in memory at once.
[ April 26, 2007: Message edited by: Jim Yingst ]
 
Mary Cole
Ranch Hand
Posts: 362
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for your reply. I got the response using the getResponseBodyAsStream,
now the problem is when I try to insert in the postgres DB, it gives the OOM again. I have written the response in a temp file and using that file as a stream to insert to DB


Am seting as -



can you pls help me whats the alternative soln
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you sure this is where the OOME occurs? Either here or when you get to ps.execute(), I guess? If the OOME doesn't have a stack trace (they often don't), you can add logging statements to figure out just where the execution was when the OOME occurs.

If the problem is indeed coming from the PreparedStatement, it seems like this may be a bug or inherent limitation in the SQL driver. You might try checking to see if other drivers are available - a more recent version perhaps? Check the postgresql site for bug reports, or try asking on one of the postgresql mailing lists.

What is the SQL data type of column 4? Perhaps it's something that is not supposed to receive so much data. Though I think that would create a problem on the database side, not on the client side.

Have you tried simply raising the max heap memory with -Xmx as Vlado suggested?

I note that if the goal is to get this data into a PreparedStatement, you can probably skip the intermediate step of writing the data to a file and then reading the file. Just plug the response stream directly into the PreparedStatement:

This probably won't solve your OOME, but it might. And it should be simpler and faster, which is something.
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
http://archives.postgresql.org/pgsql-jdbc/2002-08/msg00115.php

Sounds like you might try using setBlob() rather than setBinaryStream().

If you can confirm that you are certain the OOME comes during the postgresql calls, we might move this thread to the JDBC forum for additional ideas.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!