Win a copy of Java Concurrency Live Lessons this week in the Threads forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

FOP - Out of memory  RSS feed

 
Jeppe Sommer
Ranch Hand
Posts: 270
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi.

I am trying to use FOP to get an e-mail attachment returned as byte array.

It seems that I am running out of memory when I create the Fop object:
Fop fop = fopFactory.newFop(MimeConstants.MIME_PDF, foUserAgent, out);

I am pretty sure it has something to do with the ByteArrayOutputStream.

Any help would be appreciated?



Server log:
 
Ulf Dittmer
Rancher
Posts: 42970
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How big is the attachment? How much memory does the servlet container have allocated? FOP needs a lot.
 
Jeppe Sommer
Ranch Hand
Posts: 270
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The file is only 12 kb.

The server have 2560 MB and Xmx512m.

I have tried to save the attachment as a file on disk.

The error now come here:
transformer.transform( src, result );



Server log:
 
Tim Holloway
Bartender
Posts: 18531
61
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Oh yuck. PermGen space.

PermGen memory isn't the same thing as regular heap memory. For Tomcat, the default PermGen space is "only" 128MB, but if you're using an ORM or other systems that do a lot of static objects, it's often advisable to double that.

If you are running Tomcat, search the Tomcat forum for more on this particular issue.
 
Richard Tookey
Bartender
Posts: 1166
17
Java Linux Netbeans IDE
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I find it worrying that this 'out of memory' is happening inside a Servlet. If the Servlet handles many simultaneous requests then it is possible that two or more FOP transformation will be active at the same time and if you are short of memory with just one request then, obviously, you are going to be worse off with two or more simultaneous requests. You may have to consider having a process, maybe external to the Servlet container, which queues FOP transformations to make sure that you limit the number of active transformations. Of course this could mean that your average response time will increase but that must be better than an 'out of memory' condition.
 
Tim Holloway
Bartender
Posts: 18531
61
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Richard Tookey wrote:I find it worrying that this 'out of memory' is happening inside a Servlet. If the Servlet handles many simultaneous requests then it is possible that two or more FOP transformation will be active at the same time and if you are short of memory with just one request then, obviously, you are going to be worse off with two or more simultaneous requests. You may have to consider having a process, maybe external to the Servlet container, which queues FOP transformations to make sure that you limit the number of active transformations. Of course this could mean that your average response time will increase but that must be better than an 'out of memory' condition.


Depending on the size and complexity of the request, running FOP inside a servlet is not an unrealistic thing to do. If you want a web request to generate a PDF via FOP, ultimately, it's going to be a servlet receiving the request anyway. The apache Cocoon webapp framework relies on FOP for its PDF functionality, for example.

The issue here isn't "running out of memory", it's running out of PermGen memory. PermGen space isn't the same thing as general space. It's allocated differently, and, as I mentioned before, it's not as big as the regular heap memory pool. Nor can adjusting the heap limits help, since there's a completely separate parameter that sets the PermGen size.
 
Richard Tookey
Bartender
Posts: 1166
17
Java Linux Netbeans IDE
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tim Holloway wrote:
The issue here isn't "running out of memory", it's running out of PermGen memory. PermGen space isn't the same thing as general space. It's allocated differently, and, as I mentioned before, it's not as big as the regular heap memory pool. Nor can adjusting the heap limits help, since there's a completely separate parameter that sets the PermGen size.


OK it is not running out of general memory but 'PermSpace' but surely this doesn't change the fact that FOP is consuming a too much of the container's resources for just one request and simultaneous requests will make it far worse. Many year ago when I first used FOP in earnest the FOP transformation took so much memory (JDK1.4 so not PermGen involved) that the FOP processing was done in a separate computer.
 
Tim Holloway
Bartender
Posts: 18531
61
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Actually, we have no idea how much general heap memory this particular transformation is using. All we truly know is that it wants more PermGen space. And lacking actual breakdowns, I can't even guarantee that FOP is what's eating PermGen. I've never had FOP blow out that particular resource, although ORM-based apps are a perennial nuisance in that regard for me. So FOP+Hibernate could be more a Hibernate problem than a FOP problem, as an example.

I would definitely monitor system resource usage, however. There may be FOP-related issues. Although a "lot" of memory is something whose absolute size changes from year to year. Still, in order to measure how much heap memory is being eaten - and whether it's worth the expense to redesign for an external processor - first we need to resolve the immediate problem. That can be done with a simple VM setting, which is a lot faster and easier than re-architecting the app for gains which may or may not materialize.
 
Richard Tookey
Bartender
Posts: 1166
17
Java Linux Netbeans IDE
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tim Holloway wrote:, which is a lot faster and easier than re-architecting the app for gains which may or may not materialize.


I have voiced my concerns about FOP memory resource usage and have not suggested a "re-architecting". Having worked on a project that employed an IBM high volume web site consultant I know that establishing a client usage pattern is paramount in deciding on an architecture. Knowing the usage pattern allows one to model various architectures to establish resources required. My experience is that FOP is (or at least was) a very resource hungry process and may be bad news when used in a web site. I know little about the more recent garbage collectors and bow to your experience but I wanted the OP to be aware of possible resource shortage problems. I don't want to get into an argument over this.
 
Jeppe Sommer
Ranch Hand
Posts: 270
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for your comments.

We solved our memory issue by simply add:

-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
-XX:MaxPermSize=256m

For some reasons that we don't know, the MaxPermSize was reseted and too low.

When this is said our traffic is still low so I am not sure about your discussion.

I will keep in mind about the simultaneous active transformations...
 
Tim Holloway
Bartender
Posts: 18531
61
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
MaxPermSize defaults to 128m. 256m will almost always do if you need more, although I did have to do a bump up to 384m on a complex system this morning.

Fortunately, PermGen space is not used for most memory needs and it's pretty much all shared memory, so adding more users is unlikely to require massive amounts of PermGen.
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!