Runtime getRuntime() exec(cmd[]) - How does this work?

[uniojn qoifazy]

hi all,

string path ='/testdata/testfile/ft-v05.2010-05-09.190301+0800';
String[] cmdlinux = new String[3];
		cmdlinux[0] = "/bin/sh";
		cmdlinux[1] = "-c";
		cmdlinux[2] = "/usr/bin/flow-print -f 5 <"+path;

 try {  
            Runtime rt = Runtime.getRuntime();
			Process proc = rt.exec(cmdlinux);
			InputStream stderr = proc.getErrorStream();
			InputStreamReader isr = new InputStreamReader(stderr);

			BufferedReader br = new BufferedReader(new InputStreamReader(proc
					.getInputStream()));
            String line = null;  
            while ((line = br.readLine()) != null) {  
                System.out.println(line);  
            }  
        } catch (IOException e) {  
            e.printStackTrace();  
        } 

it's just my code , but I can not seem to get it to work. Do I need to do something else or is there a better way.
but if i don't add "-f 5 " to cmdlinux[2] = "/usr/bin/flow-print -f 5 <"+path; , it's will work successful ,
anyone can tell me why ? how to add the parameters "-f 5 " ?

thanks,
unioj

[Charles Hargrave]

Hello,

I don't have access to a UNIX server at the moment so I can't test your code but I saw something that stands out to me.

You are not consuming (reading) any content from the stderr output stream. If your process's stderr or stdout streams fill up with content, they will lock up your process; this causes problems for a LOT of people when they execute native code through Java.

In short, you need to create threads that will read the content from the stdout and stderr streams after you execute the command (rt.exec() line). Read this article and see page 4 for a good example about it (it's the StreamGobbler class).


As for how you set up the cmdlinux[], I'm not sure if the redirect (<) will cause you trouble in there or not. I usually don't run native programs through Java on a UNIX system but it might be worth it to put those commands into a shell script and make your Java code execute the shell script.

CNH

[Chuck Buche]

Using the java exec() you cannot re-direct stdio as you can from the command line. The command line interpreter (shell) handles I/O redirection. Using the Process object in Java, you need to handle redirection yourself. Here is one possible implementation:

                String path = "/testdata/testfile/ft-v05.2010-05-09.190301+0800";  
		String [] cmdlinux = new String [] {
			"/bin/sh",
			"-c",
			"/usr/bin/flow-print",
			"-f",
			"5"
		};
		try {    
			Runtime rt = Runtime.getRuntime();  
			Process proc = rt.exec(cmdlinux);  
			InputStream stderr = proc.getErrorStream();  
			InputStreamReader isr = new InputStreamReader(stderr);  

			BufferedReader br = new BufferedReader(new InputStreamReader(proc  
					.getInputStream()));  

			// copy content of "path" to stdin:

			OutputStreamWriter w = new OutputStreamWriter(proc.getOutputStream());
			BufferedReader r = new BufferedReader(new FileReader(path));

			int ch;
			while( (ch = r.read()) != -1 ) {
				w.write(ch);
			}
			w.close();
			r.close();

			// copy content of stdout to our stdout:
			String line = null;    

			while ((line = br.readLine()) != null) {    
				System.out.println(line);    
			} 
			br.close();
			
			int returnCode = proc.waitFor();
			
			if( returnCode != 0 ) {
				// error message
			}

		} catch (Exception e) {    
			e.printStackTrace();    
		}

[Jeff Verdegan]

Stupid quote and code tags don't seem to be working...

Using the java exec() you cannot re-direct stdio as you can from the command line. The command line interpreter (shell) handles I/O redirection.

He is using the shell.

Using the Process object in Java, you need to handle redirection yourself. Here is one possible implementation:

                String path = "/testdata/testfile/ft-v05.2010-05-09.190301+0800";  
		String [] cmdlinux = new String [] {
			"/bin/sh",
			"-c",
			"/usr/bin/flow-print",
			"-f",
			"5"

No, you need "print -f 5" to be a single arg as he has it in his OP.

[uniojn qoifazy]

hi Chuck Buche ,
i have follow your code to modify my code ,

import java.io.*;
public class flowtest {

	/**
	 * @param args
	 */
	public static void main(String[] args) {
		String path = "/testdata/data/ft-v05.2011-07-27.000001+0800";
		String [] cmdlinux = new String [] {
		"/bin/sh",
		"-c",
		"/usr/bin/flow-print",
		"-f",
		"5"
		};
		try {
		Runtime rt = Runtime.getRuntime();
		Process proc = rt.exec(cmdlinux);
		InputStream stderr = proc.getErrorStream();
		InputStreamReader isr = new InputStreamReader(stderr);

		BufferedReader br = new BufferedReader(new InputStreamReader(proc
		.getInputStream()));

		// copy content of "path" to stdin:

		OutputStreamWriter w = new OutputStreamWriter(proc.getOutputStream());
		BufferedReader r = new BufferedReader(new FileReader(path));

		int ch;
		while( (ch = r.read()) != -1 ) {
		w.write(ch);
		}
		w.close();
		r.close();

		// copy content of stdout to our stdout:
		String line = null;

		while ((line = br.readLine()) != null) {
		System.out.println(line);
		}
		br.close();

		int returnCode = proc.waitFor();

		if( returnCode != 0 ) {
		// error message
		}

		} catch (Exception e) {
		e.printStackTrace();
		} 

	}

}

but it's can't work and get some error msg as follows :
java.io.IOException: Broken pipe
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:297)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:121)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:124)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:112)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:194)
at flowtest.main(flowtest.java:35)


please anyone can help me ?

thanks ,

[Charles Hargrave]

At the risk of asking a silly question, is there any flow-print command line option for specifying an input file? Is it really required that you use the redirect?

If you have to pipe the file content into the command, there's probably a Java class that can handle that (never used it myself though).

I still think the easiest option would be to put the UNIX commands inside a UNIX shell script and have it accept a parameter for the input file's name (to be used in the UNIX command).

Oh well, it's late. If I think of anything else, I'll post again.
CNH

[uniojn qoifazy]

Charles Hargrave wrote:At the risk of asking a silly question, is there any flow-print command line option for specifying an input file? Is it really required that you use the redirect?

If you have to pipe the file content into the command, there's probably a Java class that can handle that (never used it myself though).

I still think the easiest option would be to put the UNIX commands inside a UNIX shell script and have it accept a parameter for the input file's name (to be used in the UNIX command).

Oh well, it's late. If I think of anything else, I'll post again.
CNH

hi Charles Hargrave ,
it's need to use the redirect (<) to input file ,
and

String[] cmdlinux = new String[3];  
        cmdlinux[0] = "/bin/sh";  
        cmdlinux[1] = "-c";  
        cmdlinux[2] = "/usr/bin/flow-print  <"+path;  

it's work in java , but when i add "-f 5"

String[] cmdlinux = new String[3];  
        cmdlinux[0] = "/bin/sh";  
        cmdlinux[1] = "-c";  
        cmdlinux[2] = "/usr/bin/flow-print -f 5 <"+path;  

it's failed ,
so , i think Java class that can handle redirect (<) , but i don't know why can't add more parameter (-f 5 ) ??

[Jeff Verdegan]

Does the command work with the -f 5 directly on the command line without Java?

Are you gobbling stdout and stderr as suggested here? http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

The following works fine for me.

public class P {
  public static void main(String[] args) throws Exception {
    String[] cmdlinux = new String[3];  
    cmdlinux[0] = "/bin/sh";  
    cmdlinux[1] = "-c";  
    cmdlinux[2] = "/bin/ls -l -t -r /tmp > /tmp/zzz";
    Runtime rt = Runtime.getRuntime();  
    Process proc = rt.exec(cmdlinux);  
  }
}

[Charles Hargrave]

uniojn qoifazy wrote:
hi Charles Hargrave ,
it's need to use the redirect (<) to input file ,
and

String[] cmdlinux = new String[3];  
        cmdlinux[0] = "/bin/sh";  
        cmdlinux[1] = "-c";  
        cmdlinux[2] = "/usr/bin/flow-print  <"+path;  

it's work in java , but when i add "-f 5" it's failed ,
so , i think Java class that can handle redirect (<) , but i don't know why can't add more parameter (-f 5 ) ??

I'm guessing that the redirect into the '-f 5' is treated differently through Java. You might want to try restructuring the command to see if you can work around it. I have no idea what the '-f 5' does for the 'flow-print' command.

String[] cmdlinux = new String[4];    
cmdlinux[0] = "/bin/sh";
cmdlinux[1] = "-c";
cmdlinux[2] = "cat " + path + "|";
cmdlinux[3] = "/usr/bin/flow-print -f 5";

As Jeff mentioned in his last reply, about gobbling up the stderr and stdout streams, I strongly recommend implementing the things in that article - especially for production code. Even though that Java article is old, it's been remarkably accurate for the past 10 or so years. If the 'flow-print' command's '-f' option generates a lot of screen output, it can fill up the the stderr and stdout streams and cause the process to lock up. The size of the stdout and stderr buffers is different for each OS.

Also another observation: if this is for a Java 1.7 environment, its ProcessBuilder class (a Process building class) has changed and has added methods to redirect output streams (stderr, stdout) and input streams (stdin) to File objects. For Java 1.7, to 'gobble' up streams (read them but discard the content), you might be able to create a File instance that points to "/dev/null" and use the ProcessBuilder instance's 'redirectOutput(File)' and 'redirectError(File)' methods. It's just an idea - I have not tested it.

CNH

[Jeff Verdegan]

Charles Hargrave wrote:

uniojn qoifazy wrote:
hi Charles Hargrave ,
it's need to use the redirect (<) to input file ,
and

String[] cmdlinux = new String[3];  
        cmdlinux[0] = "/bin/sh";  
        cmdlinux[1] = "-c";  
        cmdlinux[2] = "/usr/bin/flow-print  <"+path;  

it's work in java , but when i add "-f 5" it's failed ,
so , i think Java class that can handle redirect (<) , but i don't know why can't add more parameter (-f 5 ) ??

I'm guessing that the redirect into the '-f 5' is treated differently through Java.

No.

We're not redirecting into -f 5. Those args are completely unrelated to the redirection. We're redirecting our input to come from the file specified by the "path" variable.

Java doesn't know or care anything about those args or the redirection. It just passes that all on to the /bin/sh command.

The structure looks correct, and since my similar /bin/ls command worked fine, and since his command works find without the -f 5, I have to guess that the -f 5 is either invalid to start with (hence my suggestion to try it directly on the command line) or else using -f 5 produces enough output on stdout or stderr to make it block.

[Charles Hargrave]

Jeff Verdegan wrote:

No.

We're not redirecting into -f 5. Those args are completely unrelated to the redirection. We're redirecting our input to come from the file specified by the "path" variable.

Java doesn't know or care anything about those args or the redirection. It just passes that all on to the /bin/sh command.

The structure looks correct, and since my similar /bin/ls command worked fine, and since his command works find without the -f 5, I have to guess that the -f 5 is either invalid to start with (hence my suggestion to try it directly on the command line) or else using -f 5 produces enough output on stdout or stderr to make it block.

Re: 'redirecting into -f 5', I don't know why I said that; I guess I was still waking up. Thanks for pointing that out so I wouldn't cause the OP to go down the wrong path.

As for the stdout and stderr streams, I agree. I suspect the '-f 5' causes 'flow-print' to write a lot of output and lock up the process (probably the stderr stream). I've worked with Java executing other programs a lot in the past and ignoring the stdout and stderr streams will cause you a lot of misery with 'chatty' programs.

[uniojn qoifazy]

hi Charles Hargrave
i have try to using your method , but still can't get the command results by java
for "-f 5" is mean the output data type and the option is 5 from flow-print
i try to using other option for -f , and the output data is less than "-f 5 ",
so , i think the problem is like you said , generates a lot of screen output, it can fill up the the stderr and stdout streams and cause the process to lock up.
am i wrong ? do you have any idea to solve the problem ?

[Jeff Verdegan]

uniojn qoifazy wrote:
so , i think the problem is like you said , generates a lot of screen output, it can fill up the the stderr and stdout streams and cause the process to lock up.
am i wrong ? do you have any idea to solve the problem ?

As I already stated, try this: http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

[Charles Hargrave]

Jeff Verdegan wrote:

uniojn qoifazy wrote:
so , i think the problem is like you said , generates a lot of screen output, it can fill up the the stderr and stdout streams and cause the process to lock up.
am i wrong ? do you have any idea to solve the problem ?

As I already stated, try this: http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

I completely agree with Jeff, and yes, if the stderr and stdout streams fill up, it will cause the process to lock up.

Please read the article that Jeff put in his last post. We have both mentioned that article in our previous replies; that's how important it is to solving your problem. It explains the problem much better than I can and it is still very accurate and useful for more than 10 years since it was written. The only update it needs is to mention the ProcessBuilder class (it was added to the JDK a few years later).

If you are not using Java 1.7:
You will need to write a thread class to read and clear the data from the stdout and stderr streams. Look at Listing 4.5 on page 4 of the article that Jeff just posted. Look at the StreamGobbler class in that listing; that will read and clear data from any output stream you give it.

Now look at how the GoodWindowsExec class uses the StreamGobbler class in the main() at lines 60 - 70; that is very important to solving your problem.

If you are using Java 1.7:
The updates to the ProcessBuilder class can help you create a Process object with stream gobbling very easily. You could use ProcessBuilder's redirectError(File file) and redirectOutput(File file) methods; they redirect the stderr and stdout output streams to files. If you don't want to save the output to a file, specify the file names as '/dev/null' in UNIX (Linux) or 'NUL' in Windows so the output will be thrown away. With all of this detail, you should be able to fix your code.

CNH

[Jeff Verdegan]

Charles Hargrave wrote:If you don't want to save the output to a file, [...] 'NUL' in Windows

Cool. Never knew that. (Of course, it's never been an issue, since when I'm on Windows I work in cygwin as much as possible, but I'll hang onto that one just in case. )

With all of this detail, you should be able to fix your code.

Or at the very least, eliminate one more variable and narrow the focus of your investigation.