BufferedReader read in real time.

[Bod McLeon]

I run a python script from Java, and the using a BufferedReader, I read the input. Up until now it was running perfectly. All of a sudden, it seems in certain functions of my python script, it is no longer giving me any output until the script has finished.

Here a shortened version of my java code:

public static void main(String[] args) throws IOException {
		
		String[] command = new String[] {"python", "example.py"};
		
		Runtime rt = Runtime.getRuntime();
		Process process = rt.exec(command);
		
		
		BufferedReader input = new BufferedReader(new InputStreamReader(process.getInputStream()));
        String s = null; 
        System.out.println("-----------------", Level.INFO);
        try {
        	while((s = input.readLine()) != null){
        		System.out.println(s);
        	}

        	input.close();
        } catch (IOException e) {
        	e.printStackTrace();
        }
}

And the python code can simply be:
print("line1") for i in range(0,10000000):    pass print("line 4")
It's a very simplified version of what is happening in one of my python functions.
What I would expect the output in Java to be is:

line 1
..... // a very long wait
line 4

In "real time" - so when the python prints a message, it appears in Java as quick as possible. What actually happens is that nothing is printed until the for loop has finished running. Not even the "line 1" will appear even though it is before the for loop. Correct me if i'm wrong, but because it's a buffered reader is (blocking?) buffering the input. Normally I would use a .flush() method but the buffered reader doesn't have that

How can I change this so as soon as python prints a message, Java will print that message as well?

[Campbell Ritchie]

Runtime#exec() isn't a beginning topic, so I shall move you to our “general” forum.
What does line 11 mean? Does it actually compile?
I think (not certain) part of your problem might be that you are executing, reading, and writing all on the same thread. The thread might not start output until after the execution has finished. Or the three printing calls might be interfering with one another. Why didn't you use the waitFor() method?
Please find “When Runtime.exec() won't” by Michael Daconta (try here), which explains the pitfalls in exec(). It is old but still relevant. A ProcessBuilder may make the exercise simpler.

[Ron McLeod]

Most likely the output is being buffered. Try running python with the option for unbuffered output:

String[] command = new String[] { "python", "-u", "example.py" };

[Carey Brown]

BufferedReader also performs its own buffering. "readLine()" doesn't return until it sees a new-line. Is your python program outputing a new-line with each print?

[Bod McLeon]

Campbell Ritchie wrote:Runtime#exec() isn't a beginning topic, so I shall move you to our “general” forum.
What does line 11 mean? Does it actually compile?
I think (not certain) part of your problem might be that you are executing, reading, and writing all on the same thread. The thread might not start output until after the execution has finished. Or the three printing calls might be interfering with one another. Why didn't you use the waitFor() method?
Please find “When Runtime.exec() won't” by Michael Daconta (try here), which explains the pitfalls in exec(). It is old but still relevant. A ProcessBuilder may make the exercise simpler.

Line 11 is just a typo - normally I have a logger there and I just forgot to remove the Level parameter.

The first thing I wanted to do was to swap to ProcessBuilder as suggested. While doing so, I came across these examples. Unfortunately, their non-blocking example isn't what I needed.
You mention that it could be because I'm reading, writing and executing all on the same thread. How would I not do that?

public static void main(String[] args) throws IOException {
		
		ProcessBuilder processBuilder = new ProcessBuilder();

        processBuilder.command("python", "example.py");
        Process process = processBuilder.start();
        
        System.out.println("-----------------");
        
		watch(process);
	}
	
	private static void watch(final Process process) {
	    new Thread() {
	        public void run() {
	            BufferedReader input = new BufferedReader(new InputStreamReader(process.getInputStream()));
	            String line = null; 
	            try {
	                while ((line = input.readLine()) != null) {
	                    System.out.println(line);
	                }
	            } catch (IOException | InterruptedException e) {
	                e.printStackTrace();
	            }
	        }
	    }.start();
	}

In this code I have separated the executing from the reading and writing but sadly I am still getting the "buffered" output.
You also mentioned waitFor(). Where would this go?

[Bod McLeon]

Ron McLeod wrote:Most likely the output is being buffered. Try running python with the option for unbuffered output:

+String[] command = new String[] { "python", "-u", "example.py" };

That has fixed it!

Carey Brown wrote:BufferedReader also performs its own buffering. "readLine()" doesn't return until it sees a new-line. Is your python program outputing a new-line with each print?

I don't think it was. It was printing 1 message, going into a for loop and then printing another line after completing the loop. 99% of the time nothing would be printed by the for loop.

I'm very surprised that was the fix. I was under the impression that it was the 'readLine' causing the buffering but also it's surprising because it has worked without that argument for a very long time.

Either way, it seems to be working now so thank you!

[Campbell Ritchie]

Bod McLeon wrote:. . ., it seems to be working now so thank you!

Success