PDFBox throws OutOfMemory error

[Konstantinos Vasileiou]

Hi guys,
I am using PDFBox to parse sequentially a large number of PDF files, get their text, and write in another document. What I have done is creating a method

  public static void writePDFtoWriter(File pdfFile, Writer writer)   

that accepts as arguments the name of the file and the Writer to the new file that contains the concatenation of the textual data.

The error I get, when the parsing reaches one specific file, is:

 Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2734)
	at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
	at java.util.ArrayList.add(ArrayList.java:351)
	at org.pdfbox.pdfparser.PDFStreamParser.parse(PDFStreamParser.java:117)
	at org.pdfbox.cos.COSStream.getStreamTokens(COSStream.java:133)
	at org.pdfbox.util.PDFStreamEngine.processSubStream(PDFStreamEngine.java:202)
	at org.pdfbox.util.operator.Invoke.process(Invoke.java:96)
	at org.pdfbox.util.PDFStreamEngine.processOperator(PDFStreamEngine.java:452)
	at org.pdfbox.util.PDFStreamEngine.processSubStream(PDFStreamEngine.java:215)
	at org.pdfbox.util.PDFStreamEngine.processStream(PDFStreamEngine.java:174)
	at org.pdfbox.util.PDFTextStripper.processPage(PDFTextStripper.java:336)
	at org.pdfbox.util.PDFTextStripper.processPages(PDFTextStripper.java:259)
	at org.pdfbox.util.PDFTextStripper.writeText(PDFTextStripper.java:216)
	at profileBuilder.Downloader.writePDFtoWriter(Downloader.java:134)
	at profileBuilder.Downloader.printTRECDocument(Downloader.java:67)
	at main.MainTasks.executeTasks(MainTasks.java:110)
	at main.MainGUI$2$1.run(MainGUI.java:117)   

Any ideas what is going wrong?

[Konstantinos Vasileiou]

To add some more details to my previous post:
Increasing the heap size does not help at all - I tried up to 512Mb getting the same error at this particular pdf.
I made a simple prototype application like this:

   PDDocument doc = null;
		try {
			
			doc = PDDocument.load("1.pdf");
			PDFTextStripper stripper = new PDFTextStripper();
			System.out.println("Encrypted? " + doc.isEncrypted());
			System.out.println("to string? " + doc.toString());
			System.out.println("total pages? " + doc.getNumberOfPages());
			System.out.println("text: "+stripper.getText(doc));
			if(doc.isEncrypted()){
		try {
			doc.decrypt("");
		} catch (CryptographyException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (InvalidPasswordException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}///blah blah...
				 

and the system throws exactly the same error at this damned file.

[Jarred Olson]

How big is the PDF?

I've never used the PDFBox, but I have used itext (http://www.lowagie.com/iText/) and that worked pretty well for me but I was generating PDFs and not reading them.

[Konstantinos Vasileiou]

The PDF is not very big (1.8Mb) and PDFBox works fine with much larger files.

[Jarred Olson]

Again, I've never used PDFBox so I'm not sure if you can do this or not (I know you can do it with java.io.*) but you might want to try reading it in line by line to try and keep your heap size down.

[Konstantinos Vasileiou]

Jarred Olson wrote:Again, I've never used PDFBox so I'm not sure if you can do this or not (I know you can do it with java.io.*) but you might want to try reading it in line by line to try and keep your heap size down.

I am sure you can do this... at least I do not know such a method of PDFBox. The getText() method extracts all the text at once, but as I can guess from the description of the error message, PDFBox also uses the structure of the pdf document, so I do not know if parsing line by line can exist, similarly to a "flat" I/O stream.