Java Process PID

[prasad agarwal]

Hi everybody,

One of our systems is still the AS400 with Websphere 6.0.2 running (JDK 1.4.2).
We are anyway in the process the migrating those apps onto Websphere 8.5.5.

These days the Websphere on AS400 is crashing because of very high % ASP.
I prevent the operating system from crashing by stopping and starting Websphere which means there is some run away thread or deadlock happening.
Creating thread dump or javacore on new Websphere is quite easy but on old Websphere, the only way I found was to use wsadmin which is failing with " java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: com/ibm/jvm/Dump". It would be great if anybody could help me on why this is failing?
Maybe the JVM is too old to have anything like javacore dump?

I also thought of going into QSHELL and do the kill -3 <PID> but how do I get the pid for Java process ?
Any help would be highly appreciated.
Thanks again

[Henry Wong]

prasad agarwal wrote:
I also thought of going into QSHELL and do the kill -3 <PID> but how do I get the pid for Java process ?

For Unix based OSes (Linux, Solaris, etc.), you can use the "ps" command. For Windows, you can use the "tasklist" command.

Henry

[prasad agarwal]

Henry, Thanks.
I did try that yesterday with " ps -ef | grep java " but unfortunately that did not give any result ?
I gave this command at the root directory. I believe it does not really matter, right?

[Jude Niroshan]

This would help you.

[Tim Holloway]

Um, the most important word here is "AS/400".

The IBM AS/400, now referred to as the iSeries is a small mainframe. It has a unique hardware architecture (the Power series chips are descended from it) and a totally unique OS, unrelated even to other IBM OS's. It's also about the only OS I know of that has its own built-in DBMS.

The QSHELL provides a POSIX-style shell, but the underlying OS is NOT a Unix-style OS, and I don't know how deep the similarities go - originally the iSeries OS was pretty opaque - not even assembly language was available outside of IBM's own internal development groups.

It can be hard to locate the PID even on native Unix-style OS's, since the program name for the process is usually simply "java". Usually I use the "ps -ax" command and grep for server-specific data on the command line (this may require extra "ps" options to avoid a truncated display. For Tomcat, for example, the giveway for me is the word "catalina".

For Websphere, I don't remember, but if you can get QSHELL to display the process command lines and you can't find "websphere" in the command, look for similar terms such as "http".

If that doesn't work, probably have to hit the IBM docs.

[prasad agarwal]

Thanks Jude, I will try that.

On the other hand, I am trying to find 'core.jar' on the AS400/iSeries machine so that I can do something like 'com.ibm.jvm.Dump.JavaDump()' but I am not finding any as of now. I guess it should be there somewhere?

If anybody knows where I can find that? nothing under Java400 folder ! I also did not find under WebSphere folder either...
We are running - IBM WebSphere Application Server - Express, 6.0.2.11

We are in the process of migrating to WAS 8.5.5. but it might take a couple of months or so.

Thanks again

[prasad agarwal]

Hi Tim,

Thanks for your response.
I was trying different ways to get the javacore/dump and it appears like the wsadmin/Jacl command worked.
I mean I set the jvm and then ran '$AdminControl invoke $jvm dumpThreads' it worked without throwing any error and took me back to wsadmin command line.
It didn't tell me where the javacore is ! Now my problem is finding where it is... damn !

The below is from my wsadmin and you see after I run the command, it just takes me for entry of next command.

> set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]
WASX7029I: For help, enter: "$Help help"
wsadmin>WebSphere:name=JVM,process=server1,platform=dynamicproxy,node=okjuvju
s,j2eeType=JVM,J2EEServer=server1,version=6.0.2.11,type=JVM,mbeanIdentifier=J
VM,cell=okjuvjus
wsadmin>
> $AdminControl invoke $jvm dumpThreads

wsadmin>

[prasad agarwal]

I used the UNIX find (find . -mtime 1) and finally found the Javacore file - javacore.268136.QEJBSVR.SERVER1.20151223104626986.txt
I next supplied this javacore to the IBM Thread and monitor Dump Analyzer and I come across another problem !!

The Dump analyzer says - "Cannot find any thread dumps in javacore.268136.QEJBSVR.SERVER1.20151223104626986.txt"

I don't know why this is happening ?? Any help or suggestion of any other javacore/dump analyzer is appreciated.
Thanks

[prasad agarwal]

I found the process ID and gave the kill -3 (or QUIT) <pid>.

From documentation, this is supposed to generate the threadDump but it just killed the job.... I mean killed the Websphere server !

I need to quickly go and restart Websphere to get things back to normal.

Is this really supposed to happen this way? or something went wrong?

Thanks for your help. I appreciate

[Claude Moore]

As Tim noted, IBM As400 has its own unique architecture and OS, despite the fact that Qshell gives you a taste of a Linux like shell, I would try to analyze the issue using specific AS400 OS tools and commands.Websphere on As400 is quite a different beast with respect to Websphere on other platforms, generally speaking you should read the specific documentation for iSeries version and follow specific features for such platform. Sounds honestly strange that a program or process dies on an AS400 without any OS trace and log - as400 usually is very detailed with logging and trace. You may try to investigate the issue having a look at job logs; when WAS is running, try to monitor it with wrkactjob commands and similar ( I am not an As400 expert, but I used to deal with AS400 as DBMS so I learnt a bunch of basic commands). If I am not wrong, WAS on as400 runs in its own subsystem (QEJB ?) you should ask some As400 guy in your staff to help you in sizing and monitoring it. Sorry, but my knowledge on as400 ends here...

[prasad agarwal]

Hi Claude, Thanks for your reply.
Yes WAS on as400 runs on its own subsystem (Qwas6 for WAS 6 and Qwas855 for WAS 8.5.5).
I actually got the process ID using the subsystem name like " ps -s Qwas6 " gave me the PID. I thought I was very close in generating javacore or heapdump, but it ended killing the process without any trace as far as I could see.
I quickly restarted WAS 6 and everything was back to normal.

I will still research to see what happened when I get time.... since we are anyway in the process of moving to WAS 8.5.5 on new AS400 which has a very easy way of generating javacore directly from IBM Admin Console.

[Claude Moore]

I had a quick look at kill documentation for iSeries googling a bit, not sure if it is updated. It seems that to send a signal to a process (in your case, you want to send the signal '3') you should use 'kill -n 3 PID', not just 'kill -3 PID' . Did you try this syntax ? Maybe there are differences between 'regular' Linux commands and their qshell counterparts.

[prasad agarwal]

Hi Claude, Thanks again
I never tried 'kill -n 3 PID'. I am definitely going to give it a shot and see if that does the job. I really appreciate your help.

[Claude Moore]

Hope it helped. Let's know if it works (if you want, of course )

[prasad agarwal]

Claude, I have not yet tried it.
When the problem occurs again, I will give it a shot.
I believe sometimes I need javacore because of a deadlock or runaway job !

Good thing is ,it has not happened recently.
When it happens, I will try this command.