T2K/T5K type of Solaris servers do not receive unix commands from Java

[Sang Pavi]

It is being consistently observed that on T2k/T5k type of Solaris servers (e.g. T5220), the UNIX commands sent from Java using the Runtime class APIs are not being reveived by Solaris. It does not happen always, only occasionally. But when it happens, the java process hang at process.waitFor(). We are using Java1.6. What is the solution or workaround for this?

[Rob Spoor]

I'd read the Javaworld article "When Runtime.exec() won't" first.

[Sang Pavi]

I read the Java World article and the behavior we are seeing is not because of any of the known pitfalls. Also the warning in the Runtime API that speaks about the limited buffer size is also not applicable in this case because we do see it working sometimes. All of a sudden Runtime.exec gets stuck. We have also observed that the problem is not dependent on the command we pass through the Runtime.exec API. It happens for any trivial command.

For more information on the issue, when we execute a 'pstack' on the process id, we see this most of the time and the 'truss' output shows the java process to be in a sleeping state:
# pstack 20650
ff2c52d8 lwp_park (0, 0, 0)
ff2bf350 cond_wait_queue (24dd90, 24dd78, 0, 0, 0, 0) + 28
ff2bf770 cond_wait_common (24dd90, 24dd78, 0, 0, 0, 0) + 294
ff2bf874 cond_wait (24dd90, 24dd78, 0, 0, 0, 0) + 10
ff2bf8b0 pthread_cond_wait (24dd90, 24dd78, ff2f3700, 0, f337da00, ecf00000) + 8
eedddacc kernel_delete_session (24dd60, 24dd58, 0, 1, 3, 1c63c) + 114
eeddd7dc kernel_delete_all_sessions (0, 1, 0, 1c8b4, 0, eedfa000) + 98
eedd1eb4 cleanup_library (0, 1, 0, eedfa000, 28180, 3) + 3c
eedd1f1c kernel_fini (0, 1, 0, eedfa000, 28120, eedfa4a4) + 44
eede89e0 _fini (ff3f40fc, ff3f5a70, 2b3f4, 0, ff3f4910, 821) + 4
ff3c54b4 call_fini (ff3f40fc, f43518f0, eede89dc, ff3f42f0, ff3f42a8, ff3f4910) + cc
ff3cfdc0 remove_hdl (f43518f0, f1c4e2bc, 0, 4000, ff300a54, 4821) + ac8
ff3ca408 dlclose_intn (f4351978, ff3f4910, ff3f40fc, 2a520, ff3ca4bc, 0) + 24
ff3ca4e8 dlclose (f4351978, 0, f4350b88, 0, 1c00, 1) + 24
eee33904 pkcs11_slottable_delete (1, f9e2c0, 46ec90, eee46bb0, 0, 1) + 138
eee2e42c pkcs11_fini (eee46b8c, 1, eee2e074, eee46000, 17c18, eee46b84) + 4c
ff241f38 _postfork_child_handler (1d18, ff2f3700, 1c00, 4, f337da00, ff2f3700) + 30
ff2b74bc fork (0, 41, 0, f1c4e744, ff2f3700, f337da00) + 144
fe8a8ee4 Java_java_lang_UNIXProcess_forkAndExec (ffffffff, 0, fee0c0, fe8c4000, 4, 2) + 6d0
f940bc20 * java/lang/UNIXProcess.forkAndExec([B[BI[BI[BZLjava/io/FileDescriptor;Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;)I+-20552
f940bbc4 * java/lang/UNIXProcess.forkAndExec([B[BI[BI[BZLjava/io/FileDescriptor;Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;)I+0
f94058b8 * java/lang/UNIXProcess.<init>([B[BI[BI[BZ)V+62
f9405764 * java/lang/ProcessImpl.start([Ljava/lang/String;Ljava/util/Map;Ljava/lang/String;Z)Ljava/lang/Process;+182
f9405874 * java/lang/ProcessBuilder.start()Ljava/lang/Process;+112
f9405874 * java/lang/Runtime.exec([Ljava/lang/String;[Ljava/lang/String;Ljava/io/File;)Ljava/lang/Process;+16

[Rob Spoor]

Sang Pavi wrote:I read the Java World article and the behavior we are seeing is not because of any of the known pitfalls. Also the warning in the Runtime API that speaks about the limited buffer size is also not applicable in this case because we do see it working sometimes. All of a sudden Runtime.exec gets stuck. We have also observed that the problem is not dependent on the command we pass through the Runtime.exec API. It happens for any trivial command.

For exactly the same input (exactly the same files etc), does the hanging occur randomly? I've had the hanging problem once where certain input files would cause no problems while others did. After following the advice from that article my code was working with all input files.

[Sang Pavi]

Yes Rob, the problem happens randomly for exactly the same input!