• Post Reply Bookmark Topic Watch Topic
  • New Topic

Dropbox "java.lang.outofmemoryerror java heap space" when trying to upload a large file.  RSS feed

 
pankaj kumar t
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
HI, i'm new to this form,
i was getting an outofmemoryerror while trying to upload a large file to dropbox via java dropbox sdk.
Here is my scenario :
Server A contains a file (1GB) --> Server B opens a stream and passes that stream to dropbox api in java
Here is my code :


Could anyone solve my issue thank you.
 
Dave Tolls
Rancher
Posts: 2914
36
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Can you post the full stack trace?
 
pankaj kumar t
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
org.springframework.web.servlet.DispatcherServlet.triggerAfterCompletionWithError(DispatcherServlet.java:1259)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:945)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:855)
javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)
javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
root cause

java.lang.OutOfMemoryError: Java heap space
java.util.Arrays.copyOf(Arrays.java:2271)
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:52)
com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:63)
com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:34)
com.dropbox.core.v2.DbxUploadStyleBuilder.run(DbxUploadStyleBuilder.java:30)
org.spring.ourchat.home.uploadTODropBox(home.java:66)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:855)
javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)
javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
 
Campbell Ritchie
Marshal
Posts: 55793
164
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Don't know, but welcome to the Ranch
 
Paul Clapham
Sheriff
Posts: 22526
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
This part of the stack trace:

java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:52)


tells me that the Dropbox code is copying something to a byte array. Which suggests it's trying to copy your upload file into memory before uploading it to Dropbox. I could be wrong about that because all I see is the stack trace, but my first sentence is fact and the second sentence is a speculative conclusion based on that fact. But since an out-of-memory condition is occurring, that makes my conclusion much more likely.

Clearly that isn't scalable, because your code only has a fixed amount of memory available so it restricts the size of the file you can upload. And it's surely not necessary to do it that way. But it's not unknown for even well-financed corporations to distribute untested software written by people who don't really know what they are doing.

But again, I might be wrong. If this SDK is open-source then you could look at the code yourself and see if it's really making that error. If not, you could perhaps contact the place you got it from and ask what's going on.
 
Dave Tolls
Rancher
Posts: 2914
36
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Unless the code has changed between v1 and v2 (and the cached v2 version in Google looks the same for the copy code) then it uses a 16k buffer.
IOUtil.java

Maybe there's a config thing that's set incorrectly?
I notice the DbxClientV2 takes a config object.
 
pankaj kumar t
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I really don't know what to do right now , i will have to do trail and error to resolve this issue.
 
Tony Docherty
Bartender
Posts: 3268
82
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
No don't do trial and error, that approach rarely works and even if you do get your file to upload you may find your code is extremely brittle. Investigate the problem, find out why the exception is occurring and then plan your approach to overcoming the problem.

Dave and Paul have given you some things to look at, take their advice and investigate those areas.
 
pankaj kumar t
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Paul Clapham wrote:This part of the stack trace:

java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:52)


tells me that the Dropbox code is copying something to a byte array. Which suggests it's trying to copy your upload file into memory before uploading it to Dropbox. I could be wrong about that because all I see is the stack trace, but my first sentence is fact and the second sentence is a speculative conclusion based on that fact. But since an out-of-memory condition is occurring, that makes my conclusion much more likely.

Clearly that isn't scalable, because your code only has a fixed amount of memory available so it restricts the size of the file you can upload. And it's surely not necessary to do it that way. But it's not unknown for even well-financed corporations to distribute untested software written by people who don't really know what they are doing.

But again, I might be wrong. If this SDK is open-source then you could look at the code yourself and see if it's really making that error. If not, you could perhaps contact the place you got it from and ask what's going on.

How can i avoid loading a total file into memory and pass that stream to dropbox and flush after some 'x' bytes.
can please you give me code sample.
Thank you :)
 
Paul Clapham
Sheriff
Posts: 22526
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
pankaj kumar t wrote:How can i avoid loading a total file into memory and pass that stream to dropbox and flush after some 'x' bytes.


I don't know. As I said, my answer was speculation and I suggested some things you could do. Am I right in guessing you haven't done any of those things? Okay, let's try the first one together. Is the Dropbox code open source? If so, give me a link to the source code and I'll help you in finding where to look.
 
pankaj kumar t
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Paul Clapham wrote:
pankaj kumar t wrote:How can i avoid loading a total file into memory and pass that stream to dropbox and flush after some 'x' bytes.


I don't know. As I said, my answer was speculation and I suggested some things you could do. Am I right in guessing you haven't done any of those things? Okay, let's try the first one together. Is the Dropbox code open source? If so, give me a link to the source code and I'll help you in finding where to look.


Hi paul thanks for the help
here is the github link https://github.com/dropbox/dropbox-sdk-java for dropbox.
 
Dave Tolls
Rancher
Posts: 2914
36
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So this is the IOUtil class that does the transferring.
This is called by the DbxUploadStyleBuilder, with an in and an out stream.

Looking at that version of copyStreamToStream, line 31, we see that it uses the default buffer size, which is 16 k.
So that can't be the issue.

However, look at the output stream being used.
That's a ByteArrayOutputStream.

So, it's not writing to an external place, it's writing to an array internal to your app.
So each 16k block of data is being added to the array held in this class. That's what's happening in the grow() method.

So where is this stream coming from?
Looks like it's from the uploader supplied, which is called on line 30 of the DbxUploadStyleBuilder.

This is supplied by that call to uploadBuilder, which is based (I believe) in the config.

So...what is the "dropbox/java-tutorial" client identifier?
Is it, by any chance, a config for simple testing that doesn't actually hit dropbox?

That would be where I would look.
 
pankaj kumar t
Greenhorn
Posts: 6
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dave Tolls wrote:So this is the IOUtil class that does the transferring.
This is called by the DbxUploadStyleBuilder, with an in and an out stream.

Looking at that version of copyStreamToStream, line 31, we see that it uses the default buffer size, which is 16 k.
So that can't be the issue.

However, look at the output stream being used.
That's a ByteArrayOutputStream.

So, it's not writing to an external place, it's writing to an array internal to your app.
So each 16k block of data is being added to the array held in this class. That's what's happening in the grow() method.

So where is this stream coming from?
Looks like it's from the uploader supplied, which is called on line 30 of the DbxUploadStyleBuilder.

This is supplied by that call to uploadBuilder, which is based (I believe) in the config.

So...what is the "dropbox/java-tutorial" client identifier?
Is it, by any chance, a config for simple testing that doesn't actually hit dropbox?

That would be where I would look.


Hi Dave finally i got it working using new api

here is the code

 
Paul Clapham
Sheriff
Posts: 22526
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for coming back to let us know about your fix, Pankaj!
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!