It looks like the byte array "data" contains the entire content of the file. If the file is too big, then that array will be too big. So if there's any way to avoid putting all the data in a single byte at one time, do it. For example, you might grab a little bit of the stream at a time, and write the data from the stream to a file:
Here we still use a byte array, but its size is limited to 4096. We just keep filling it and writing it as many times as necessary.
It really depends on what you need to do with the data. Often it's possible to process the data in small chunks, as above. Other times it's much easier to have all the data in memory at once. But for large files, try to avoid putting all the data in memory at once. [ April 26, 2007: Message edited by: Jim Yingst ]
Thanks for your reply. I got the response using the getResponseBodyAsStream, now the problem is when I try to insert in the postgres DB, it gives the OOM again. I have written the response in a temp file and using that file as a stream to insert to DB
Are you sure this is where the OOME occurs? Either here or when you get to ps.execute(), I guess? If the OOME doesn't have a stack trace (they often don't), you can add logging statements to figure out just where the execution was when the OOME occurs.
If the problem is indeed coming from the PreparedStatement, it seems like this may be a bug or inherent limitation in the SQL driver. You might try checking to see if other drivers are available - a more recent version perhaps? Check the postgresql site for bug reports, or try asking on one of the postgresql mailing lists.
What is the SQL data type of column 4? Perhaps it's something that is not supposed to receive so much data. Though I think that would create a problem on the database side, not on the client side.
Have you tried simply raising the max heap memory with -Xmx as Vlado suggested?
I note that if the goal is to get this data into a PreparedStatement, you can probably skip the intermediate step of writing the data to a file and then reading the file. Just plug the response stream directly into the PreparedStatement:
This probably won't solve your OOME, but it might. And it should be simpler and faster, which is something.