• Post Reply Bookmark Topic Watch Topic
  • New Topic

How do I pipe I/O between two processes?  RSS feed

 
Brian McGuinness
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I want to create a pipeline where I create processes with ProcessBuilder and then pipe the standard output from each process into the standard input of the next one. I see that by default, ProcessBuilder directs standard input, standard output, and standard error to pipes, but the ProcessBuilder's redirection methods only let me redirect I/O to *files*, there are no methods provided to redirect I/O to PipedInputStreams or PipedOutputStreams. On the other hand, a Process can return its I/O streams, but only as InputStreams or OutputStreams, not PipedInputStreams or PipedOutputStreams. And plain InputStreams and OutputStreams don't provide a connect() method to connect them. So how do you connect these?

The Java process library seems to be very poorly designed. I have already discovered that ProcessBuilder.inheritIO() doesn't work properly if the I/O streams of the parent process have been redirected. Now I have the above frustration with pipelines.
 
Tony Docherty
Bartender
Posts: 3264
81
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There's an article (with code) on how to achieve this. See http://www.certpal.com/blogs/2010/11/using-a-pipedinputstream-and-pipedoutputstream/
 
Brian McGuinness
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks, but that reference, like other articles I have seen, just indicates how to establish pipes between two threads of a Java program. I want to set up a pipeline between separate programs (I am experimenting with writing a shell in Java), so I need to use ProcessBuilder, not Runnable. The problem is that ProcessBuilder only provides methods to redirect to files, and Process only returns InputStreams and OutputStreams, not PipedInputStreams and PipedOutputStreams, which you could connect. If ProcessBuilder was designed properly, it would support redirection to streams, so you could redirect to file streams, string streams, piped streams, or anything else you wanted rather than just to files. I don't see how to get around these limitations and establish the connections that I want between the processes. There should be a way to do this, but it's hard to find.
 
Rob Spoor
Sheriff
Posts: 21048
85
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Just create a thread that copies data from one Process' InputStream and to the other Process' OutputStream (it's annoying that a process' output is available through an InputStream and vice versa...).
 
Brian McGuinness
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ok. This seems like a rather kludgey approach. I was thinking that there had to be a better way to do this, but it was probably buried deep in the documentation somewhere. But apparently that is not the case. I will write a simple pipe manager and use that in a thread to connect each pair of processes.

Thanks for the suggestion.
 
Paul Clapham
Sheriff
Posts: 22374
42
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I may well be wrong, having never used ProcessBuilder, but from reading the documentation it's possible this might work:


 
Brian McGuinness
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I tried this:



When I ran it, it just sat there for a moment and I got no output.

Meanwhile, I created a pipe connector for my shell:



I tried setting up a pipeline with this, starting with the last process and moving backward to the first, so each process would block while waiting for input from the previous process:



with



When I tried entering pipeline commands and executing them, the first process completed but the others froze, according to the diagnostic messages from my program. I used



to wait for the pipeline to complete. For the line "ls -l | sort" I got:



and I got no output from the pipeline. When I tried "cat ../cantrips.txt | sort" to display a decent sized text file, I got


and again there was no output, but this time when I exited the shell (my Java program) the sorted lines from my text file displayed on the screen. So the file actually had passed through the pipeline to sort and been sorted, but it then got hung up somehow and didn't display until the shell exited.
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Brian McGuinness wrote:Ok. This seems like a rather kludgey approach.

Not quite sure why you think so. A PipedInputStream IS an InputStream, and the same for output, so all you need to do is to subclass (or wrap) a Process to create "piped" versions, and have your shell use those instead.

It'll sure highlight any "illegal communication".

Now whether you want them buffered or not (especially if they're separate programs) is a whole new question; and I'd think very carefully before you decide...

Winston
 
Brian McGuinness
Greenhorn
Posts: 10
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The reason it seems kludgey is that Java has the capabilities necessary to make pipes work, so they should provide a simpler way of creating the pipe connections that I need. For example, providing methods in ProcessBuilder to redirect streams, rather than just files, would solve the problem. It shouldn't be necessary to resort to workarounds.

In any event, I added code to my pipe connector class to close the I/O streams after copying was finished, and now my pipes seem to work properly. Apparently, the output wasn't being flushed, and that kept the threads from terminating. So now I have:



Thanks very much for your suggestions and feedback.
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Brian McGuinness wrote:The reason it seems kludgey is that Java has the capabilities necessary to make pipes work, so they should provide a simpler way of creating the pipe connections that I need. For example, providing methods in ProcessBuilder to redirect streams, rather than just files, would solve the problem. It shouldn't be necessary to resort to workarounds.

But you're not re-directing, you're piping - or rather you're using a pipe as a "connection" (except that you're not).

In any event, I added code to my pipe connector class to close the I/O streams after copying was finished, and now my pipes seem to work properly. Apparently, the output wasn't being flushed, and that kept the threads from terminating. So now I have:

And this is my point. You're NOT actually using pipes (or a Pipe) at all; you're "kludging" one yourself from Streams - and I hate to say, but not the most efficiently.

I think, if I was doing this, I'd actually wrap (or extend) a Process to use 'Piped' versions for its input and/or output streams and then simply chain them myself directly.
And even if I did do what you're doing, I wouldn't try on every iteration, and I certainly wouldn't "swallow" an exception, viz:As I indicated, I'm pretty sure the flush() is redundant (or could perhaps be moved outside the loop, immediately before the close()), but I'm not absolutely sure, since more than one Thread is involved. If it is an issue, then it's just one more reason to use PipedInputStream/PipedOutputStream, because they are designed to be used with separate Threads.

And just a minor style point: Underscores in field names are generally frowned on in Java - especially as "prefixes" - as they smack of "C". You can use them, but you may see a few raised eyebrows from older bods like me.

HIH

Winston
 
Tony Docherty
Bartender
Posts: 3264
81
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Winston Gutkowski wrote:I think, if I was doing this, I'd actually wrap (or extend) a Process to use 'Piped' versions for its input and/or output streams and then simply chain them myself directly.
And even if I did do what you're doing, I wouldn't try on every iteration, and I certainly wouldn't "swallow" an exception, viz:

I agree with the points you make but I'm not sure about closing the streams in the transmit method - if an exception is thrown during the read/write phase the streams will never be closed. I'd suggest moving the stream closure to a finally clause in the run method or use try-with-resources, again in the run method.
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tony Docherty wrote:I agree with the points you make but I'm not sure about closing the streams in the transmit method - if an exception is thrown during the read/write phase the streams will never be closed. I'd suggest moving the stream closure to a finally clause in the run method or use try-with-resources, again in the run method.

Yup, much better. Keep forgetting about finally, mainly because I hardly ever use try...

Winston
 
Brian McGuinness
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ok. For now I will clean up the pipe connector class as you have suggested.

I will have to experiment with extending the Process class and see what happens.

I find the leading underscores useful for identifying member names since I don't have to search all over the code to find where a variable was defined; I know right away when it's a member, which is defined at the top of the class.

Thanks for your help.
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Brian McGuinness wrote:Ok. For now I will clean up the pipe connector class as you have suggested.

And don't forget Tony's point. It's much better than what I wrote. Indeed, try-with-resources is probably the more "up-to-date" way to do it, because then you don't need to worry about closing; and you may not need the extra method either.

I will have to experiment with extending the Process class and see what happens.

I think you'll be pleasantly surprised. You'll probably save yourself an extra Thread for each "pipe" too.

Alternatively: have you considered using ProcessBuilder.Redirect.PIPE? I have a feeling that that may be what you were looking for in the first place, but I'm not very familiar with ProcessBuilder.

I find the leading underscores useful for identifying member names since I don't have to search all over the code to find where a variable was defined; I know right away when it's a member, which is defined at the top of the class.

Fine. Just be aware that it's not "standard" (sounds like a convention from another language) and, in places that are strict about coding practises, it might not be allowed.

Winston
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!