Swing GUI crashes (using JNI)

[Divya Marwaha]

Hi,

I am working on java swing application(GUI) using JNI code to communicate with native C code. The Swing application launches properly and works fine.
The GUI is used to start some customized system level tests(io,memory,cpu) and show their progress. The tests have to be left running at-least overnight to get the
results. However, if I check the next morning, GUI crashes and throws following message.

Any pointers on source of the issue will be greatly appreciated.

Java version: java 1.5 / java 1.6
OS: Solaris 10.

Thanks,
Divya
PS: Also posted on stackoverflow for more insight. http://stackoverflow.com/questions/3008378/swing-gui-using-jni-crashes

=============MESSAGES==================
# uname -a
SunOS Generic_127127-11 sun4v sparc SUNW,
#
# #
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0xff268924, pid=9473, tid=272
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_14-b03 mixed mode)
# Problematic frame:
# C [libc.so.1+0x68924] strstr+0x20
#
# An error report file with more information is saved as hs_err_pid9473.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

=============================

Another machine:

# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xff231fd0, pid=1406, tid=180
#
# JRE version: 6.0_18-b07
# Java VM: Java HotSpot(TM) Server VM (16.0-b13 mixed mode solaris-sparc )
# Problematic frame:
# C [libc.so.1+0x31fd0] strcpy+0x70
#
# An error report file with more information is saved as:
# /usr/sunvts/bin/hs_err_pid1406.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

[Maneesh Godbole]

Did you check out the contents of the hs_err_pidxxx logs? If I am not mistaken, they will be dumped in your home directory, though not so sure.

[Divya Marwaha]

Yes I do see the log.
The trace shows the calls all the way to my JNI code and then points to libc.so.1

But what exactly should I be looking for ? Could it be an issue in my code or is it JVM code issue?

[Maneesh Godbole]

No idea. Sorry.
Since this problem is not related to Swing, I will move this tread to the Java in General forum for you.
Hopefully somebody will be able to throw some light on this issue.

[pete stein]

Cross-posted at java-forums.org

http://www.java-forums.org/awt-swing/29682-swing-gui-crashes-using-jni.html

[Henry Wong]

Well, it is a seg fault ... meaning core dump.

If the stack trace shows that it is occuring in the JVM (and not related to the JNI call), then I recommend submitting the core dump to Sun/Oracle as a bug report. I also recommend upgrading to the latest underscore release too.

If the stack trace shows that it is due to the JNI call, then you have the core dump to track it down. Fire up gdb on the core dump, and try to isolate the issue. This should be treated as no different than hunting down any other C/C++ bug.

Henry

[Ernest Friedman-Hill]

Looks like some pretty classic C programming issues. Segvs like these

# C [libc.so.1+0x31fd0] strcpy+0x70

or

# C [libc.so.1+0x68924] strstr+0x20


are very likely just due to passing dangling pointers around in the native code.

How much do you know about JNI programming? It can be pretty tricky dealing with thread-related memory ownership issues, which I suspect are quite thick in the kind of program you're describing. Maybe you could give us an idea of the scope of your native code, and maybe show us a representative sample.

[Divya Marwaha]

Native code base is pretty huge and complex. To begin with, I will check for dangling pointers. Thanks.

[Divya Marwaha]

Below is what I see in stack trace. It doesn't appear to be JVM issue.
Looks like the problem is happening in native function call
vts_connect() and vts_user_cred(), Is that so???


Stack: [0xa2980000,0xa2a00000), sp=0xa29feb58, free space=506k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.1+0x68924] strstr+0x20
C [libjniapi.so+0x1d80] vts_connect+0x134
C [libjniapi.so+0x19e4] __1cQvts_authenticate6F_i_+0x30
C [libjniapi.so+0x1ad4] Java_com_deg_agent_JNILink_vtsCommand+0xc
j com.deg.agent.JNILink.vtsCommand(Ljava/lang/String;)Ljava/lang/String;+0
j com.deg.agent.JNILink.vtsCommand(Ljava/lang/String;)Ljava/lang/String;+0
j com.deg.agent.JNILink.executeVTSKCommand(Ljava/lang/String;)Ljava/lang/String;+7


Stack: [0xac200000,0xac280000], sp=0xac27e698, free space=1f9ff231fd0k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.1+0x31fd0] strcpy+0x70
C [libjniapi.so+0x2c60] vts_user_cred+0x94
C [libjniapi.so+0x2720]
C [libjniapi.so+0x1d64] vts_connect+0x118
C [libjniapi.so+0x19e4] __1cQvts_authenticate6F_i_+0x30
C [libjniapi.so+0x1ad4] Java_com_deg_agent_JNILink_vtsCommand+0xc
j com.deg.agent.JNILink.vtsCommand(Ljava/lang/String;)Ljava/lang/String;+15
j com.deg.agent.JNILink.vtsCommand(Ljava/lang/String;)Ljava/lang/String;+0

Ernest Friedman-Hill wrote:Looks like some pretty classic C programming issues. Segvs like these

# C [libc.so.1+0x31fd0] strcpy+0x70

or

# C [libc.so.1+0x68924] strstr+0x20


are very likely just due to passing dangling pointers around in the native code.

How much do you know about JNI programming? It can be pretty tricky dealing with thread-related memory ownership issues, which I suspect are quite thick in the kind of program you're describing. Maybe you could give us an idea of the scope of your native code, and maybe show us a representative sample.

[Rob Spoor]

Divya Marwaha wrote:Stack: [0xa2980000,0xa2a00000), sp=0xa29feb58, free space=506k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.1+0x68924] strstr+0x20
C [libjniapi.so+0x1d80] vts_connect+0x134
C [libjniapi.so+0x19e4] __1cQvts_authenticate6F_i_+0x30
C [libjniapi.so+0x1ad4] Java_com_deg_agent_JNILink_vtsCommand+0xc
j com.deg.agent.JNILink.vtsCommand(Ljava/lang/String;)Ljava/lang/String;+0
j com.deg.agent.JNILink.vtsCommand(Ljava/lang/String;)Ljava/lang/String;+0
j com.deg.agent.JNILink.executeVTSKCommand(Ljava/lang/String;)Ljava/lang/String;+7

This is what I see in stack trace. It doesn't appear to be JVM issue.
Looks like the problem is happening in native function call vts_connect(), Is that so???

It appears so. That's indirectly called by class com.deg.agent.JNILink, method String vtsCommand(String). Maybe you're passing a null string and it doesn't check for it.

[Divya Marwaha]

But once the tests are started from the swing GUI, the progress of tests is shown properly. After 24+ hours run, at some random time the SEGV crash is reported. While tests are running no changes can be made from the GUI.

Rob Prime wrote:

Divya Marwaha wrote:Stack: [0xa2980000,0xa2a00000), sp=0xa29feb58, free space=506k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.1+0x68924] strstr+0x20
C [libjniapi.so+0x1d80] vts_connect+0x134
C [libjniapi.so+0x19e4] __1cQvts_authenticate6F_i_+0x30
C [libjniapi.so+0x1ad4] Java_com_deg_agent_JNILink_vtsCommand+0xc
j com.deg.agent.JNILink.vtsCommand(Ljava/lang/String;)Ljava/lang/String;+0
j com.deg.agent.JNILink.vtsCommand(Ljava/lang/String;)Ljava/lang/String;+0
j com.deg.agent.JNILink.executeVTSKCommand(Ljava/lang/String;)Ljava/lang/String;+7

This is what I see in stack trace. It doesn't appear to be JVM issue.
Looks like the problem is happening in native function call vts_connect(), Is that so???

It appears so. That's indirectly called by class com.deg.agent.JNILink, method String vtsCommand(String). Maybe you're passing a null string and it doesn't check for it.

[Ernest Friedman-Hill]

Indeed. 999,999 times out of a million, your code might work fine -- but it still may not be coded correctly to deal with thread ownership of references or with the possible data movement that can occur due to garbage collection. That's why I said JNI can be tricky.

[Divya Marwaha]

I made some string related changes in C files (native layer for string manipulation operations)
The error is still appearing after 24 + hours of running the application. and stack is pointing to same strcpy location.

Code flow is :
Swing UI
|
java code
|
JNI C file (jniapi.cc)
|
native layer

The Swing UI sends command to native layer and native layer returns a response which is displayed by UI.

code from jniapi.cc is also below which makes calls to native functions.
################

extern int debug ;
extern int verbose ;
int vts_authenticate() {
  
    static char hostname[10];
    sprintf(hostname, "localhost"); // connect to vtsk on the localhost
    int op_mode = Sunopmode;          
   // use vts_connect() command to authenticate the user.
    int returnValue = vts_connect(hostname, NULL, op_mode);
    
    static char result[50];
    sprintf(result, "Unknown error. Error code - %d.", returnValue);
    switch(returnValue) {
	case VTS_HOSTDOWN:
	case VTS_NOHOST:
            sprintf(result, "VTSK is not responding.");
            printf("%s\n", result);
            exit(-1);
	    break;
	case VTS_PASSWD:
            sprintf(result, "Password is required.");
	    break;
	case VTS_INVPASSWD:
            sprintf(result, "Invalid password.");
	    break;
 	case VTS_DENIED:
	    sprintf(result, "\nPermission denied by the vtsk.");
            printf("%s\n", result);
            exit(-1);
           
	    break;
	case VTS_NOTCOMPATIBLE:
	    sprintf(result, "Incompatible  version.");
            printf("%s\n", result);
            exit(-1);
	    break;
	case VTS_OP_MODE:
	    sprintf(result, "Incompatible operation mode.");
            printf("%s\n", result);
            exit(-1);
	    break;
	case VTS_OK:
	    sprintf(result, "Success");
	    break;
	case VTS_BADDB:
            char *errMsg = get_last_error_message();
	    snprintf(result, 50, errMsg);
            printf("%s\n", result);
            exit(-1);
	    break;
    }

    return returnValue;
}

JNIEXPORT jstring JNICALL Java_com_deg_agent_JNILink_vtsCommand
  (JNIEnv* env, jobject jobj, jstring commandString) {
    jboolean iscopy;
    static char errorMsg[68];
    int ret = vts_authenticate();
    if(ret != 0)
        sprintf(errorMsg, "vts_connect failed \nCommunication between vtskJB & vtsk failed. vtsk is not responding.");
    char *command = (char *)
        env->GetStringUTFChars(commandString, &iscopy);
    long timeout = 180; // timeout value for rpc call to vtsk : 3 minutes
    char *returnValue = vts_cmd(command, timeout);
    
    if (returnValue == NULL) {	/* rpc failure */ 
        sprintf(errorMsg, "Communication between vtskJB & vtsk failed. vtsk is not responding.");
        printf("%s\n", errorMsg);
        return env->NewStringUTF(errorMsg);
    }
    return env->NewStringUTF(returnValue);
}

################

What sort of JNI optimization can be introduced?

Thanks,
Divya

Ernest Friedman-Hill wrote:Indeed. 999,999 times out of a million, your code might work fine -- but it still may not be coded correctly to deal with thread ownership of references or with the possible data movement that can occur due to garbage collection. That's why I said JNI can be tricky.

[Rob Spoor]

Please UseCodeTags. It will work even for C code (although the highlighting may be incorrect, but at least the indentation will be there).

[Ernest Friedman-Hill]

Can we see vts_connect? If we saw the call to strstr, then we could figure out where its arguments came from, and that would let us trace where they might possibly go wrong.

[Rob Spoor]

One unrelated note: you're calling GetStringUTFChars but your missing the ReleaseStringUTFChars call.

[Divya Marwaha]

Stack trace for jvm dump mostly points to strcpy (and sometimes strstr, strcat)

C [libc.so.1+0x31fd0] strcpy+0x70
C [libjniapi.so+0x2c60] vts_user_cred+0x94
C [libjniapi.so+0x2720]
C [libjniapi.so+0x1d64] vts_connect+0x118
C [libjniapi.so+0x19e4] __1cQvts_authenticate6F_i_+0x30
C [libjniapi.so+0x1ad4] Java_com_sun_deg_sunvts_agent_JNILink_vtsCommand+0xc
j com.sun.deg.sunvts.agent.JNILink.vtsCommand(Ljava/lang/String;)Ljava/lang/String;+15

I am including important fn def.

int vts_connect(char *host, char* passwd,int op_mode)
{
#ifdef          RPC_BASED
//	char 	*tmp;
	int	ret;
	get_ui_version();

	if (gethostbyname(host) == (struct hostent *)NULL) {
		/*(void) printf("fail gethostbyname\n");*/
		return(VTS_NOHOST);
	}

	
		{ struct timeval rtvs;
          rtvs.tv_sec = -1; rtvs.tv_usec = 0;
          clnt_control(client, CLSET_RETRY_TIMEOUT, (char*)&rtvs);
	}


        if ((ret = vts_secure_cmd(passwd,op_mode)) != VTS_OK )
		return( ret );

	if ( version_match() < 0 ) return(VTS_NOTCOMPATIBLE);
	if ( op_mode_match(op_mode) < 0 ) return(VTS_OP_MODE);
	return(VTS_OK);
#else
    if ((msgqidd = msgget(ftok(VTS_PID, 'U'), MSGGET_FLAG)) == -1) {
      perror("ui: msgget failed");
      return(VTS_HOSTDOWN);
    }
    else {
      (void) printf("%d\n", msgqidd);
      return(VTS_OK);
    }
#endif          RPC_BASED
}

static int vts_secure_cmd( char* passwd, int mode)
{

	enum 	clnt_stat 	stat;
	int    i;
	int    ret;
	char*  pt;
	char*  upt;
	struct timeval	tvs;
	char   user_cred[2048]; //CHANGED IT TO 2048, EARLIER IT WAS 512
	char   reply[256] = {0}; /* The partial initialization is for Lint */
#ifdef __linux
	int *clnt_res;
#endif       
        vts_user_cred( user_cred, passwd, mode );
	tvs.tv_sec =  180;/*638047 - 30 before */
	tvs.tv_usec = 0;
	upt = user_cred;
	pt = reply;
	stat = RPC_TIMEDOUT;	
	
	if (stat != RPC_SUCCESS) {
		clnt_perror(client,"clnt_call() failed");
	    	return (VTS_HOSTDOWN);
	}


}


[b]FILE 2[/b]
static char cmd[1024];
void vts_user_cred( char* user_cred, char* user_pass ,int op_mode)
{
	pid_t pid;
        char hostname[MAXHOSTNAMELEN+1]; 
        static struct user_info ustr;
	int  i;
size_t len=2048;
         
	pid = getpid();
        (void) sysinfo(SI_HOSTNAME,hostname, MAXHOSTNAMELEN); 
        (void) sprintf(cmd,"%s_%d-%d",hostname,pid,op_mode);
	(void) strlcpy( user_cred, cmd, len ); //CHANGED IT FROM STRCPY TO STRLCPY but still seeing same crash for strcpy

        if ( ustr.num_groups > 0 )
          for ( i=0; (i < ustr.num_groups); i++)
            free( ustr.grp_names[i] );

	(void)usr_info( &ustr );
        (void) strcat( user_cred, " " );
        (void) strcat( user_cred, "<" );
        (void) strcat( user_cred, ustr.user_name );
        (void) strcat( user_cred, ">" );
        (void) strcat( user_cred, "<" );
        for ( i=0; (i < ustr.num_groups); i++) {
		if (ustr.grp_names[i] == NULL) break;
         	if ( i > 0 ) (void) strcat( user_cred, ",");
           	(void) strcat( user_cred, ustr.grp_names[i] );
        }
        (void) strcat( user_cred, ">" );
	
	if ( user_pass != NULL ) {
          (void) strcat( user_cred, "<" );
	  (void) strcat( user_cred, user_pass);
          (void) strcat( user_cred, ">" );
	}

}

Ernest Friedman-Hill wrote:Can we see vts_connect? If we saw the call to strstr, then we could figure out where its arguments came from, and that would let us trace where they might possibly go wrong.

[Divya Marwaha]

Another thing is , when I started running the java GUI app, free disk space was about 27% and later at the time of crash I saw free space of only about 8% (using df -k on solaris sparc box). Also, If one uses terminal based UI (TTYUI using C n C ++ code) then no sort of error in seen. Only when swing based UI is used with native code, jvm dumps..why is it so?? Can it be something to do with jvm heap size or similar issue? If the problem is in native code alone then should it not get reported in some form with other UIs as well which interact with native Code? I am just wondering...expert thoughts will be enlightening for me...

[Ernest Friedman-Hill]

Well, boy howdy. There are all kinds of things that might happen here, depending on what the data looks like. One that jumps out at me is this:

static struct user_info ustr;

That "ustr" is going to be shared by all calls to vts_user_cred. If the code we're looking at is called by the JVM in multiple threads simultaneously, then the contents of that structure could become inconsistent -- i.e., from one thread's perspective, ustr.num_groups could be wrong about how many grp_names there are, and you could easily be passing a dangling or garbage pointer directly into strcat, or freeing a garbage or dangling pointer, etc.

[Divya Marwaha]

Thanks all for your valuable and prompt inputs.
strcpy was done from larger size buffer (1024) to 512 size user_cred variable...
Even after making the change the reason I wasn't seeing the crash was because
all the related binaries were not copied on the testing box properly....my bad
Anyhow,Thanks again.
-Divya