• Post Reply Bookmark Topic Watch Topic
  • New Topic

Posting large files using HttpURLConnection  RSS feed

 
Michael Bronshteyn
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am trying to post very large file to the web server using the following code:
// get file to copy to server
in = new FileInputStream( contentFile );
// get HttpURLConnection
conn = ( HttpURLConnection )m_url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setUseCaches( false );
// set request propertieis
setRequestProperties( conn, requestProps );
conn.connect();
OutputStream out = conn.getOutputStream();
// setup loop to read content of the file
// and write to the output stream
byte[] data = new byte[ 4096 ];
int bytesRead = 0;
while(( bytesRead = in.read( data )) != -1 ){
out.write(data, 0, bytesRead );
out.flush();
}
out.close();
retStatus = conn.getResponseCode();
The problem is that this code attempts to read entire file into memory and than sends it to the server. Is it a way to send file by reading a chunk into memory and sending it over through OutputStream?
Thanks a lot.
 
Peter den Haan
author
Ranch Hand
Posts: 3252
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What makes you think the entire file is read in memory? As far as I can make out, the HttpURLConnection uses a plain BufferedOutputStream wrapped around the socket's OutputStream under the hood.
- Peter
 
Michael Bronshteyn
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Peter.
Thanks a lot for your reply.
I have established connection with remote debugger to the remote servlet. The breakpoint has been set in doPost method. I have another session with the debugger in the code which posts using HttpURLConnection. I can see that remote sites trips the breakpoint only after I am done with the loop and call "getResponseCode" method.
The reason I noticed this problem I am trying to post 90M file to remote site. While this code works fine for smaller files, this file causes "OutOfMemoryException". I have ran the code which posts using hprof switch ( -Xrunhprof ) and it showed a lot of heap allocations in the traces reading file in the loop.
 
Peter den Haan
author
Ranch Hand
Posts: 3252
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yikes. What JDK are you using? The JDK version I (don't tell anyone) decompiled before posting my reply is 1.4.1 -- did not do a full analysis though, so there can still be a problem hiding somewhere.
You could always try the Apache Jakarta HttpClient library. At least it's open source and you can mend things if they break
- Peter
 
Michael Bronshteyn
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Peter,
I am using jdk 1.3.l ( one of the system requirements ). Thanks a lot for suggestion to use Jakarta library. I am going to give it try.
Thanks a lot,
Mike.
 
Raj Kethu
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Mike,

Have you got the solution for this. I am also facing exactly the same probelm and its very important part of our application. If you found the solution. please paste the code or ideas here.

Thanks,
Raj
 
Raj Kethu
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I investigated on this problem and found the solution.

Problem:
I have a Huge file that should goto server but, that should go chunk by chunk not at once(because, the file is 250 mb in size, i dont want to store that in memory). I used default JDK java.io and nio but, it did not flush to the server until I close the stream (or you get the 200 (OK) from the server.

Solution:
The default JDK stream does not flush to servlets (or server programme) even if you use stream.flush in your programme. But, if you use the following apis you can overcome this problem of flushing.

1) Servlets - Servlet clent can flush every chunk everytime.
2) HTTPClient - It works same as servlets but, in case if you are not using servlets (In my case, i cant use servlets because, I should not contact webserver). This works same as servlets.

Hope this helps to someone.
 
Michael Bronshteyn
Ranch Hand
Posts: 85
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Using Commons HttpClient solved the problem. Thus you can try the same approach. It should work for you.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!