• Post Reply Bookmark Topic Watch Topic
  • New Topic

'Gobbling up' data from an OS process's output and error streams - what if not all returned at once?

 
F Turner
Ranch Hand
Posts: 30
1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello,

My program runs an operating system process, and then waits for it to finish before continuing. The output stream is redirected to a file, and there is code to 'gobble up' the contents of the error stream (actually it is cached to be output if there is an error), as I know that this could otherwise cause the thread calling and waiting on the OS process to block. My current code looks like this:

This works fine, but I am concerned about what might happen if the error message 'isn't output all at once'; i.e. the following situation:
  • An error message is output by the OS process to the buffer from which 'reader' reads.
  • This output is read off the buffer such that readLine() returns null.
  • The condition for the while loop is no longer met and the execution of the code advances past this error gobbling loop.
  • The OS process outputs another error message, which either goes unread or fills the buffer, causing the program thread to block.


  • Could this happen? Thanks for any help
     
    Tony Docherty
    Saloon Keeper
    Posts: 3138
    72
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    AFAIK readline() will only return a null when the end of the stream is reached which only occurs when there is no possibility of further data ie when the data source terminates/disconnects.
     
    F Turner
    Ranch Hand
    Posts: 30
    1
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Hi Tony,

    That would be very useful if true. Do you know of any documentation which confirms it?

    Also, would I have problems with line 2 of the code? That couldn't possibly block the Java thread could it? I assume that, once everything has been set up as in line 2, the writing of the information to the file has nothing to do with the JVM.

    And as another question on the side, do I need to close the file (i.e. release file system hooks) when I am writing to a file using an operating system command? I wouldn't have thought so but I'm not certain.

    Thank you
     
    Campbell Ritchie
    Marshal
    Posts: 52531
    119
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    F Turner wrote:. . .  Do you know of any documentation which confirms it? . . .
    Have you tried the API documentation?
     
    F Turner
    Ranch Hand
    Posts: 30
    1
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Hi,

    I did look at that, I just found their statement 'null if the end of the stream has been reached' a bit ambiguous, as I didn't really know what was meant by 'reaching the end of a stream': having read all of the data in that stream, or the data source terminating/disconnecting as Tony said :/
     
    Campbell Ritchie
    Marshal
    Posts: 52531
    119
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    I presume you get null for your line if either happens. If you look at the classic article about processes by Michael Daconta, which you have doubtless read already, you will see Daconta uses
    while( (line = reader.readLine()) != null) ...
    Daconta was writing before the introduction of Scanner, which you can use instead of a buffered reader. It would require the loop to look like this:-The Scanner will not return null. In fact, you can only get null out of a Scanner (as far as I know) with the methods called findXXX. If you go beyond the end of the Stream, it will throw an Exception instead.
     
    Tony Docherty
    Saloon Keeper
    Posts: 3138
    72
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Campbell Ritchie wrote:I presume you get null for your line if either happens.

    If you get a null once you have "read all of the data in that stream" then the call to readline() will never block waiting for more data so I don't think you will get a null in this case. Of course, if the API documentation isn't clear, the only way to be sure is to write a short test program to see what does happen.
     
    Campbell Ritchie
    Marshal
    Posts: 52531
    119
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Don't know about the stream disconnecting before a socket is properly closed (or similar). I may have been mistaken yesterday. Maybe nothing will happen to the buffered reader, but some other code would throw some sort of IOException. I am sure you will continue to get non‑null input until the streams close, but the reader will block waiting for input between successive lines. Since the readers are in their own Threads, that blocking is not going to cause you any problems.
    The loop as described will run all right and terminate; don't worry about non‑termination. You will either have line equal to null, or you will suffer an Exception. Some Processes return nothing at all from the streams; if you use the loop to add the Strings returned to a List, and later print the List, you sometimes get [] displayed, an empty List. In that case the buffered reader waits until the Process has completed, at which point its Streams both close and readLine returns null and your loops (and their Threads) terminate.
     
    F Turner
    Ranch Hand
    Posts: 30
    1
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Hi, thanks for all of the comments and help. This is basically what you just said Campbell, but I tried an experiment as Tony suggested. This was the code:

     
    F Turner
    Ranch Hand
    Posts: 30
    1
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Sorry, I clicked submit instead of preview...

    My findings were just as described: that BufferedReader blocks on readLine() until there is more input to be received (looking at the API code seems to confirm this, though I only looked at it briefly), and only when the stream is closed does the while loop in ConcurrentReader exit - i.e. the BufferedReader does wait for input and doesn't move on from the 'collecting' loop as soon as it has exhausted the buffer.

    Does this mean that src.close() in Program.main() calls notify/notifyAll() in the background?
     
    Tony Docherty
    Saloon Keeper
    Posts: 3138
    72
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Thanks for updating the thread and confirming the answer - have a cow.

    F Turner wrote:Does this mean that src.close() in Program.main() calls notify/notifyAll() in the background?

    It actually calls the associated PipedInputStream's receivedLast() method which sets the closedByWriter flag and then calls notifyAll().
     
    F Turner
    Ranch Hand
    Posts: 30
    1
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Thanks for the cow! I don't know what type of InputStream object will be returned in my actual program via myProc.getErrorStream(), but I'll just have to believe that that is the standard way that Java has been designed (i.e. block while there is no input but the possibility for more, only return null when there cannot be any more).

    Would you mind addressing my second and third questions in the second of my posts in this thread?
     
    Campbell Ritchie
    Marshal
    Posts: 52531
    119
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    F Turner wrote:. . . Does this mean that src.close() in Program.main() calls notify/notifyAll() in the background?
    Probably not. The thread corresponding to each stream from the process will terminate when the process completes; you have never called wait() on anything, so who needs to call notify()/notifyAll()?
     
    Campbell Ritchie
    Marshal
    Posts: 52531
    119
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    F Turner wrote:. . . do I need to close the file . . .
    Yes. Closing a file reader or file writer will close all associated resources and flush the stream (writer only), so closing the file is part of the process of closing the file writer or buffered writer. That means closing a buffered writer flushes all streams and closes the file and closes everything else necessary. By the way: you will probably find try with resources a better way to close it.
     
    Campbell Ritchie
    Marshal
    Posts: 52531
    119
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Don't know about redirection of the output stream to a file, but it doesn't say anything against it in the API documentation.

    Anybody else?
     
    Tony Docherty
    Saloon Keeper
    Posts: 3138
    72
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Campbell Ritchie wrote:
    F Turner wrote:. . . Does this mean that src.close() in Program.main() calls notify/notifyAll() in the background?
    Probably not. The thread corresponding to each stream from the process will terminate when the process completes; you have never called wait() on anything, so who needs to call notify()/notifyAll()?

    PipedInputStream uses Threads and calls wait() on them hence the need for PipedIputStream to call notify()/notifyAll() when it receives notification that the stream has ended.
     
    F Turner
    Ranch Hand
    Posts: 30
    1
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Hi Campbell,

    Just to clarify: I am running an operating system process and redirecting its output to a file. I was wondering if I somehow have to close this connection, but as far as I can see it is actually nothing to do with my program or the JVM, once it [the OS process] has been started.
    I create a new File object in this line of code (line 2):

    But do not create any writers or streams to write to it.

    My thoughts were that, when you execute a command in a shell, such as:

    The OS does all of the file handling for you, so I suspected that the case would be similar here. Am I correct in thinking this?
     
    F Turner
    Ranch Hand
    Posts: 30
    1
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    (where ' > myFile.txt' is redirection to a file)
     
    Tony Docherty
    Saloon Keeper
    Posts: 3138
    72
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    F Turner wrote:
    Also, would I have problems with line 2 of the code? That couldn't possibly block the Java thread could it? I assume that, once everything has been set up as in line 2, the writing of the information to the file has nothing to do with the JVM.

    To be honest I've never used this technique so can't give a definitive answer but I would have thought that it would block if the write buffer to the file filled up. Now with writing to a local file system this is highly unlikely but if the file is on a remote machine then there is more chance of this happening. Having said that the blocking would only be temporarily ie until there was space in the write buffer to write the next piece of data.

    As to whether or not the JVM is involved in the writing process I have no idea and I'm not sure if it evens makes any difference other than if the JVM crashed mid process.

    F Turner wrote:And as another question on the side, do I need to close the file (i.e. release file system hooks) when I am writing to a file using an operating system command? I wouldn't have thought so but I'm not certain.
    Thank you

    If you are referring to closing the File object you have passed into the redirect then no you don't close it (you can't close a File object), if you are referring to any streams you open to a File then as Campbell has already said yes you must close them and I'd suggest you follow his advice of looking at "try with resources".
     
    Campbell Ritchie
    Marshal
    Posts: 52531
    119
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Tony Docherty wrote:. . . PipedInputStream . . . calls wait() . . .
    Thank you. I never knew that.
     
    • Post Reply Bookmark Topic Watch Topic
    • New Topic
    Boost this thread!