Catching InputStream from a Process

[Prashant Sehgal]

Hi,

I am facing a problem that some of you may have also seen in the past. I am trying to catch the InputStream from a Process which is simply

 Runtime rt = Runtime.getRuntime(); Process pr = rt.exec( "tail -5000000 A_Very_Large_File.txt" ); 

The problem is that the input stream coming from this process it too fast for my

BufferedReader br = new BufferedReader(new InputStreamReader(pr.getInputStream())); 

to handle.

As a result my Buffers get filled up and the system throws OutOfMemory errors. How would you handle such a situation? Is there a way in Java with which one can pause and read the input stream from a process upto a point, process it, then resume reading beyond that point?

Also, can I change my UNIX command (or maybe write a shell script and invoke that schell script from my java program) in anyway such that I can get the input stream in managable chunks?

Thanks for reading. Hope you share your thoughts.

Regards,
Prashant.

[Stan James]

Wow, interesting problem. Is your Java program unable to keep up because you are doing time consuming procesing on the data you read? Moving it to another thread won't help much if you can't keep up and the buffers explode. But maybe you could trim down your algorithm, or just write to disk and process it later.

What if you don't use buffered reader, just an input stream reader? The Unix process may think it has a very slow console but it should wait on the console rather than blowing up your buffers.

What does the error tell you? I've never had this happen. Is it pretty clear that the JVM is giving up because of buffer overflow?

[Stefan Wagner]

Do you know the maximum-size for the very_large_file?
Perhaps the -X flags for the JVM can help in that case?

[Prashant Sehgal]

Thanks for your thoughts guys.

If I dont buffer, the bare InputStreamReader will blow up even faster. So that wont sole the problem.

Also altering the heap is not really a "solution". It's just a "band-aid solution".

Is there some way (like something one does in Audio Streams) so that I could potentially pause the input stream coming from the process? Or (since I am working in UNIX) can I used named pipes (mknod or mkfifo) to route the streams through? Would that help?

This is a really interesting problem. Hope that more people can join in and help me in finding a way out.

Regards,
Prashant.
[ September 23, 2004: Message edited by: Prashant Sehgal ]

[Prashant Sehgal]

Anyone?

[Joe Ess]

How about running "tail -5000000 A_Very_Large_File.txt | less"?