• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Application deployed on JBoss 6.1 crashes

 
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All,

I have currently hosted my java application on jboss-6.1.0.Final along with jdk1.7.0_01 set up on a linux server with 32GB of RAM.

Almost daily(once or more) the JBoss server crashes with the following error message in the /var/log/messages file.

Earlier this was working fine with jboss-4.2.2.GA and jdk1.5.0_15, before I had upgraded to the above configuration.

Nov 28 13:37:21 <server host name> kernel: java[14296]: segfault at 0000000047f81ff8 rip 00002aaaab20a364 rsp 0000000047f82020 error 6
Nov 30 11:45:25 <server host name> kernel: java[1762]: segfault at 0000000048d8fff0 rip 00002aaaaafd40c4 rsp 0000000048d90000 error 6
Nov 30 12:30:40 <server host name> kernel: java[12904]: segfault at 000000004b8baff8 rip 0000003699a082d9 rsp 000000004b8bb000 error 6
Nov 30 15:39:16 <server host name> kernel: java[15304]: segfault at 0000000045557fe8 rip 00002aaaab20a385 rsp 0000000045557fe0 error 6

I have given around 10GB memory to the JVM in JBoss and it does not use more than 4-5GB during high load. There are no logs in the server.log file when the server crashes. The only solution is to start the JBoss server after which it works fine.

I have investigated the issue and have not been able to resolve it till now so please guide me to the solution for this above problem.

Thank you,

Regards,
Varun
 
Sheriff
Posts: 10445
227
IntelliJ IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
That's a JDK crash. You seem to be using JDK 1.7. Try downgrading to JDK 1.6 and see if that works fine. By the way, what kind of application is deployed on that server? Does it include any JNI code?

 
Varun Agarwal
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I had first tried to deploy the same application using JDK 1.6 and JBoss 5.1 but faced the same problem.

My application is using JSPs, Struts, Spring MVC, EJB 3 and Oracle as the technologies. No JNI code present in the application.

Could the JVM options in run.sh be the cause? The additional JVM options added by me are:-

JAVA_OPTS="-Dfile.path=/home/project -Xms6g -Xmx12g -XX:PermSize=256m -XX:MaxPermSize=1g -server -Duser.timezone=America/New_York -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -XX:-DisableExplicitGC -XX:+UseStringCache -XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+UseCompressedStrings -Djboss.service.binding.set=ports-01 -Dorg.jboss.resolver.warning=true -Djava.util.Arrays.useLegacyMergeSort=true $JAVA
 
author
Posts: 5856
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Try removing the aggresiveOpts, that option is there mainly to run a benchmark and can often cause instabilities in normal programs.
 
Varun Agarwal
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I had actually removed the following options and it has been over 24 hours since the server crashed but due to the weekend I need to wait till at least Tue/Wed to confirm that one of these options was causing the crash.

-server -XX:-DisableExplicitGC -XX:+UseStringCache -XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+UseCompressedStrings -Dorg.jboss.resolver.warning=true

Modified -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 to 1800000
 
Peter Johnson
author
Posts: 5856
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Actually, you should set the gcintervals to really high values, not lower values. Even the original vakues are not high enough. (Ask me how I know...) But if you disable explicitgc, then you don't have to set the gcintervals at all.
 
Varun Agarwal
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Following are my current JVM OPTIONS
-----------------------------------------

JAVA_OPTS="-Dsems.file.path=/home/project -Xms4g -Xmx12g -XX:PermSize=256m -XX:MaxPermSize=1g -Duser.timezone=America/New_York -Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000 -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Djboss.service.binding.set=ports-01 -Djava.util.Arrays.useLegacyMergeSort=true -XX:ErrorFile=./hs_err_pid_.log $JAVA_OPTS"

Even with the above jvm options the JBoss crashed again, below mentioned is the log for it

/var/log/messages:Dec 5 14:22:29 <server host name> kernel: java[20833]: segfault at 0000000048789ff8 rip 00002aaaab20a364 rsp 000000004878a020 error 6
/var/log/messages:Dec 5 14:34:05 <server host name> kernel: java[7358]: segfault at 0000000049395ff8 rip 0000003699a082d2 rsp 0000000049396000 error 6
 
Peter Johnson
author
Posts: 5856
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hmm, so you have a set of JAVA_OPTS that works,m and a set that doesn't. I think you need to find out which option is causing the crash and then submit a bug report to Oracle to fix the problem. And until they fix it, avoid that option! After all, a segfault in the JVM is beyond the control of JBoss AS.
 
Varun Agarwal
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
No Peter, none of the java options work, even the new java options that were set faced the segfault error.
I think I have no other option but to downgrade my JVM to 5.

I also tried to record various metrics with respect to the memory and thread usage details using MBeans but none of those metrics indicate any problem.
 
Jaikiran Pai
Sheriff
Posts: 10445
227
IntelliJ IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Which exact version of Java 1.6 is this? It may not be related to any JAVA_OPTS but might be some bug in the JVM. If there's any upgrades available on the Oracle download sites, try using that.
 
Varun Agarwal
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
It is not JDK 1.6 but is JDK 7u1.

Initially I had tried JBoss 5 with JDK 1.6 update 26 but that too gave me the same problem.

My next step is to try JBoss 6.1 with JDK 1.6 update 29
 
Jaikiran Pai
Sheriff
Posts: 10445
227
IntelliJ IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The other thing that you should do is try and isolate this crash to some specific usecase in your application. See if you have some pattern when this crashes. Also check whether the crash generates and dump file in JBOSS_HOME/bin folder.
 
Varun Agarwal
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have tried to gather a lot of statistics but am unable to replicate this problem.
 
Varun Agarwal
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am using the bouncycastle jars in my application due to which I had to do some workarounds in my JDK configuration, please suggest if this could be a reason for the crash

Workaround done were taken from http://www.randombugs.com/java/javalangsecurityexception-jce-authenticate-provider-bc.html
 
Varun Agarwal
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I downgraded my JDK to 1.6 update 11 and it is working fine with the application with NO crashes.
 
Jaikiran Pai
Sheriff
Posts: 10445
227
IntelliJ IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Varun Agarwal wrote:I downgraded my JDK to 1.6 update 11



You might want to upgrade to the latest update (30 right now) because I remember there's atleast one very serious bug which has been fixed in the recent update (26 from what I remember).
 
Varun Agarwal
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I had tried with JDK 1.6 update 29 but that too did not work. I have not tried JDK 6 update 30 and cannot do it now as cannot test it on the live server as we are unable to replicate the crash otherwise.
 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

we have got the same problem.
Our Server: RHEL Linux 5.7, Java 1.6.0-30, Tomcat 6.0.35.

Java/Tomcat crashes occasionally with segfault error 6.

Any news here?
 
Varun Agarwal
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
you can try downgrading your JDK to JDK 6 update 11 and see if that work similar to what worked for me.
 
Peter Johnson
author
Posts: 5856
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
JDK 1.6.0_11 is a little old, so I would not go with that. We have noticed issues with 1.6.0_29 and 1.6.0_30, so several of our projects revereted to 1.6.0_26 which seems to be working without issue.
 
Could you hold this puppy for a sec? I need to adjust this tiny ad:
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic