Runtime.exec() hell. Any opinions? explanations?

[Andrew Shafer]

Calling all Gurus. . .
Here is my problem.
I have two C/C++ executables that I want to be able to make calls to from Runtime.exec() on Redhat linux.
I don't know much about their inner workings, but running them from the command line you give the executables two arguments; an input file and an output file. The programs do there stuff with a few messages coming back to standard output and the results should goto the output files. Both executables run fine from the command line.
Here is where the hell starts. . .
I make my little String cmd[] and line up the arguments in a nice little package and feed it to Runtime.exec(). One of the programs does what it is supposed to with one small oddity, input file goes in, output file comes out, mostly as expected.
Though the programs have identical formats for running from the command line, the other program does not behave. The output file specified is a blank file, the process.getInputStream() does not give anything back, and the process returns an exitValue() of 0.
Both the programs run perfect from the command line, and both take nearly identical input format to run. Initially I thought the problem might have something to do with the fact that I was trying to use the Runtime.exec from a servlet container (Tomcat), but then I wrote a test class that tries to run the programs from a "normal" java application. Same results. (Here was where I found an oddity with the first program, in the test class the getInputStream is giving me all the output from standard output, but in the JSP, I only get the last line. It might have something to do with buffered writers, but I don't really care as I am primarily interested in the contents of the output files.) The one program runs great, the other one doesn't. Same behavior, empty file, exitValue = 0.
The programmer that wrote them doesn't know what might be different between the two programs that would cause this, and I have been beating myself senseless trying to figure it out.
If anyone has any clue that can stop my madness, please offer it.

String[] cmd = {exe, indir + filename, outdir + outfile};
  
Process p = Runtime.getRuntime().exec(cmd);

That's what I'm doing for both, with the filenames dynamically determined, but I get the same behavior when I hard code the filenames.

[Sean MacLean]

I wonder (since I see that you are specifying the directories by appending them to the file name) if if would be better to use this
exec(String[] cmdarray, String[] envp, File dir)
"Executes the specified command and arguments in a separate process with the specified environment and working directory."
You can simply pass null to the second argument if you want. Are both exec commands operating on the same files. Perhaps there is a blocking issue?. If you reverse the calls does the first one always work or it always the same exec call that works (regardless of the order you call them in)? Just some thoughts - I'm no expert here.
Sean

[Andrew Shafer]

Sean,
I don't see the version of .exec that can specify the directory in the javadoc, but I'll go look into it.
Here is a little more information if that would help.
The two executables take text files as input and generate text files as output. One of the executables calculates molecular thermodynamic properties and the other one calculates kinetic properties of chemical reactions. The idea is to be able to upload the input files from an html form, request the calculation and then display the output, using servlets/JSP. I thought the problem might be related to the servlet container so I wrote a class to test it and the results are the same.
The executables, the input files and output files are all in seperate directories with this structure.
directory/executable
directory/infiles/file.dat
directory/outfiles/file.out
The executable lets you direct the output and input from the command line. I just adopted this stucture cause it seemed to make sense. The behavior is the same no matter where I specify the input or output. The thermo always works and the rate program always doesn't, regardless of order.
If you think any of the other bartenders or sheriffs might have any idea or are particularly knowledgeable about this area, let them know about my problem.
Much Obliged

[Andrew Shafer]

Specifying the directory for the process results in the same behavior.

[Sean MacLean]

Uhmm. Hopefully someone else has the answer. In the meantime I know there is a really good runtime.exec() article at JavaWorld. I've read it a while ago but the site seems to be down right now. I think this is the url
www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html
I has a lot of examples of handling sticky situations. If the site doesn't come back online for a while, try a google search on
"Runtime.exec() wait" and read the cached version. Good luck.
Sean

[Andrew Shafer]

Thanks again Sean,
I actually read that article the day before I posted here.
I've gotten another couple of ideas, maybe someone will have a tip or two about this.
One idea is to use Runtime.exec() to create a process that opens a shell, then run the executables through the shell. I tried messing with it, and I think it might be a go, but I haven't got it working yet.
The new problem:
Runtime.exec("/bin/bash -c '/directory/executable /directory/infiles/data.dat /directory/outfiles/data.out'")
I get a process gives an err saying there is an unexpected end of file. If I just try the executable in the shell, I get the standard output saying that I haven't specified the input file. I think I just need to figure out how the string thing is getting fed to the shell and I might be in business.
Another approach that might work is to open the shell with Runtime.exec(), and then feed the rest through process.getOutputStream. I was unable to find examples of anyone doing this, though it seems like it should work in theory.
The other idea I came up with is to write a seperate shell script that handles the execution and see where I can get with that.
Do people just not read this forum?
I know I spend most my time in Meaningless Drivel

[Andrew Shafer]

I have been messing around running it from a shell and still no love.
If I type
sh -c '/directory/Executable /directory/infiles/data.dat'
I get the expected behavior of the OUTPUT being generated in an OUTPUT file in the directory I run the shell from.
Now if I try Runtime.exec with one big string escaping the '
("sh -c \'/directory/Executable /directory/infiles/data.dat\'")
Then I get an error:
ERR>/directory/infiles/data.dat': -c: line 1: unexpected EOF while looking for matching `''
ERR>/directory/infiles/sct.dat': -c: line 2: syntax error: unexpected end of file
And if I try to feed it in as an array
{"/bin/sh", "-c", "/directory/Executable /directory/infiles/data.dat"}
Then I get this error:
ERR>/bin/sh: /directory/Executable /directory/infiles/data.dat: No such file or directory
If I try to feed .exec an array with 4 strings separating the string with the executable from the string for the input file OR if I just feed one big string to .exec with everything but the input file, I get this:
OUT>Please give the name of input file
Which is the expected behavior if you try to execute the file without an input.
I feel like I'm so close and yet so far.
Alas. . .

[John Bateman]

Hi
Could your problem be just a case of permissions. When the java runtime executes your app, what user does it run as? Once you know this, check to make sure that 'user' has the proper permission to write to your output file.
Just a thought.
You know how many times I couldn't find the darn forest because the tree were in my way!!

------------------
SOURCE CODE should be SURROUNDED by "code" tags.
Click here for an example
[This message has been edited by John Bateman (edited August 23, 2001).]

[Andrew Shafer]

I thought permissions might be an issue, so I tried running it as root, same thing.
The executables and directories have the same permissions for both applications, why does one work and the other gives grief?

[Sean MacLean]

What about running the thermo program twice (in place of the rate program)? Does the second one fail? Do the file permissions or group on the files change after running the thermo (so the rate won't work)? What happens if you run both but direct them to different files/locations completely? I'm starting to get the feeling that all is not well with the rate program (even though they both work from the command line). Have you figured out a shell script that will run them both yet?
Sean

[Andrew Shafer]

Sean,
The programs are totally independent, I'm just baffled cause one works and one doesn't. You can run the thermo program over and over with different input files and you get perfectly expected results everytime. The rate program never works. It doesn't have anything to do with the order they are run in, typically I'm trying to run the rate program over and over trying to figure out what is wrong.
I wrote a cute little shell script that runs the rate program with a hard coded input file, just to see if I could get it to run. The script runs great from the command line, but when I tried to run it from Runtime.exec(), the shell comes back with an error, Segmentation Fault (core dump) yada yada. I also have been unable to get Runtime.exec() to call a shell first and then run the executables with arguments. I can open a shell and execute programs, as long as they don't need any arguments, but if they do there seems to be a problem.
What would be wrong with the C program that I would get such anomalous behavior when I try to run it from java. I'm beginning to think there is a problem with the way Runtime.exec() is implemented. At the very least, I should be able to open a shell and then run an executable with arguments, even if there is something fishy with the rate program. Don't you think?
I'm going to try recompiling the Rate program on some other boxes and NT to see what kind of behavior I get.
Wish me luck

[Andrew Shafer]

I tried it on a few different Linux boxes with different versions of the JDK and architecture and I'm getting the same behavior.
After reading the Bug Parade from sun, it appears Runtime.exec has some problems.
I suspect there is some sort of weird memory/stream thing going on when you create the Process with Runtime.exec().
I'm going to try NT and see what I get.

[Andrew Shafer]

Runs perfect on NT.
(Which really pisses me off by the way.)