• Post Reply Bookmark Topic Watch Topic
  • New Topic

Catching InputStream from a Process

 
Prashant Sehgal
Ranch Hand
Posts: 56
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

I am facing a problem that some of you may have also seen in the past. I am trying to catch the InputStream from a Process which is simply

The problem is that the input stream coming from this process it too fast for my to handle.

As a result my Buffers get filled up and the system throws OutOfMemory errors. How would you handle such a situation? Is there a way in Java with which one can pause and read the input stream from a process upto a point, process it, then resume reading beyond that point?

Also, can I change my UNIX command (or maybe write a shell script and invoke that schell script from my java program) in anyway such that I can get the input stream in managable chunks?

Thanks for reading. Hope you share your thoughts.

Regards,
Prashant.
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Wow, interesting problem. Is your Java program unable to keep up because you are doing time consuming procesing on the data you read? Moving it to another thread won't help much if you can't keep up and the buffers explode. But maybe you could trim down your algorithm, or just write to disk and process it later.

What if you don't use buffered reader, just an input stream reader? The Unix process may think it has a very slow console but it should wait on the console rather than blowing up your buffers.

What does the error tell you? I've never had this happen. Is it pretty clear that the JVM is giving up because of buffer overflow?
 
Stefan Wagner
Ranch Hand
Posts: 1923
Linux Postgres Database Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Do you know the maximum-size for the very_large_file?
Perhaps the -X flags for the JVM can help in that case?
 
Prashant Sehgal
Ranch Hand
Posts: 56
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for your thoughts guys.

If I dont buffer, the bare InputStreamReader will blow up even faster. So that wont sole the problem.

Also altering the heap is not really a "solution". It's just a "band-aid solution".

Is there some way (like something one does in Audio Streams) so that I could potentially pause the input stream coming from the process? Or (since I am working in UNIX) can I used named pipes (mknod or mkfifo) to route the streams through? Would that help?

This is a really interesting problem. Hope that more people can join in and help me in finding a way out.

Regards,
Prashant.
[ September 23, 2004: Message edited by: Prashant Sehgal ]
 
Prashant Sehgal
Ranch Hand
Posts: 56
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Anyone?
 
Joe Ess
Bartender
Posts: 9361
11
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How about running "tail -5000000 A_Very_Large_File.txt | less"?
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!