There was an error opening this document. The file is damaged and could not be repaired.
 It is unavoidable that the fop engine processing consumes heap space. There are some notes out there purporting that page-sequence will consume heap space without releasing... until it is switched to another. Hence, there is a urban wisdom saying something like that you have to try not use the same page-sequence master for a whole lot of pages. This is worth considering. Maybe you can try consistently use a fo:page-sequence for even page and another, amid the same in terms of fo:flow or else, for odd page etc... I say this taking into consideration that you've already done adding option -Xms and -Xmx up to Giga Bytes...
 I have tried to emulate the problem with 2K+ pages with single fo:page-sequence and fop can handle adding -Xms512m order of magnitude. But, the page is relatively simple without graphic and user generated fonts. But it is conceivable page size continuously increase until breaking all possible resource size of the machine... Hence, maybe the memory model of fop could be redesign one-day by those in that specialized field.
 If you are using fop-1.0 (it seems not available in fop-0.95), there is a ConserveMemoryPolicy setting for FOUserAgent, something like this:
 It is a certainty that however large resource fop can handle, it would be exhausted by arbitrarily large data set and fo document. Hence, I am not able to help beyond this physical limit.
CAUTION! Do not touch the blades on your neck propeller while they are active. Tiny ad:
Two software engineers solve most of the world's problems in one K&R sized bookhttps://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton