• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Tomcat shuts down randomly

 
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I'm using Tomcat to run Jenkins. For the last few days, i'm noticing that Jenkins goes down anytime. I could not find anything in the Tomcat log. There are many articles about troubleshooting such issues and i went through many of them. One of them suggested to check JVM log for clues. I did see the log file but i'm not sure about what the exact cause is. Since i could not find any OutOfMemoryError so i'm not sure whether i need to add some memory to Tomcat. Please suggest.

Server version: Apache Tomcat/7.0.2
Server built: Aug 4 2010 12:23:47
Server number: 7.0.2.0
OS Name: Linux
OS Version: 3.14.20-20.44.amzn1.x86_64
Architecture: amd64
JVM Version: 1.7.0_65-mockbuild_2014_09_18_00_28-b00
JVM Vendor: Oracle Corporation

Here is the JVM log:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f2d286512c5, pid=28177, tid=139831173089024
#
# JRE version: OpenJDK Runtime Environment (7.0_65-b17) (build 1.7.0_65-mockbuild_2014_09_18_00_28-b00)
# Java VM: OpenJDK 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0x1482c5] __strlen_sse2_pminub+0x35
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x00007f2c5ce92000 JavaThread "Computer.threadPoolForRemoting [#655]" daemon [_thread_blocked, id=16602, stack(0x00007f2cf82fd000,0x00007f2cf83fe000)]
0x00007f2c38003000 JavaThread "Computer.threadPoolForRemoting [#654]" daemon [_thread_blocked, id=16601, stack(0x00007f2ce90f1000,0x00007f2ce91f2000)]
0x00007f2c5cf7e800 JavaThread "Proc.executor [#969]" daemon [_thread_blocked, id=16599, stack(0x00007f2cfb1f2000,0x00007f2cfb2f3000)]
0x00007f2c5d77e800 JavaThread "/usr/bin/git -c core.askpass=true ls-remote -h git@git.XYZ.com:XYZ: stderr copier" [_thread_in_native, id=16598, stack(0x00007f2cf81fc000,0x00007f2cf82fd000)]
...
...

Heap
PSYoungGen total 678912K, used 390987K [0x00000007aaa80000, 0x00000007d5580000, 0x0000000800000000)
eden space 658432K, 57% used [0x00000007aaa80000,0x00000007c1e32e40,0x00000007d2d80000)
from space 20480K, 50% used [0x00000007d2d80000,0x00000007d37a0010,0x00000007d4180000)
to space 20480K, 0% used [0x00000007d4180000,0x00000007d4180000,0x00000007d5580000)
ParOldGen total 1398272K, used 185368K [0x0000000700000000, 0x0000000755580000, 0x00000007aaa80000)
object space 1398272K, 13% used [0x0000000700000000,0x000000070b506090,0x0000000755580000)
PSPermGen total 132608K, used 115513K [0x00000006c0000000, 0x00000006c8180000, 0x0000000700000000)
object space 132608K, 87% used [0x00000006c0000000,0x00000006c70ce758,0x00000006c8180000)


GC Heap History (10 events):
...
...
Event: 59356.217 GC heap before
{Heap before GC invocations=1240 (full 18):
PSYoungGen total 676864K, used 661824K [0x00000007aaa80000, 0x00000007d5580000, 0x0000000800000000)
eden space 655360K, 100% used [0x00000007aaa80000,0x00000007d2a80000,0x00000007d2a80000)
from space 21504K, 30% used [0x00000007d2a80000,0x00000007d30d0010,0x00000007d3f80000)
to space 20480K, 0% used [0x00000007d4180000,0x00000007d4180000,0x00000007d5580000)
ParOldGen total 1398272K, used 180244K [0x0000000700000000, 0x0000000755580000, 0x00000007aaa80000)
object space 1398272K, 12% used [0x0000000700000000,0x000000070b005080,0x0000000755580000)
PSPermGen total 132608K, used 115511K [0x00000006c0000000, 0x00000006c8180000, 0x0000000700000000)
object space 132608K, 87% used [0x00000006c0000000,0x00000006c70cdfa8,0x00000006c8180000)
Event: 59356.225 GC heap after
Heap after GC invocations=1240 (full 18):
PSYoungGen total 678912K, used 1024K [0x00000007aaa80000, 0x00000007d5580000, 0x0000000800000000)
eden space 658432K, 0% used [0x00000007aaa80000,0x00000007aaa80000,0x00000007d2d80000)
from space 20480K, 5% used [0x00000007d4180000,0x00000007d4280020,0x00000007d5580000)
to space 20480K, 0% used [0x00000007d2d80000,0x00000007d2d80000,0x00000007d4180000)
ParOldGen total 1398272K, used 185336K [0x0000000700000000, 0x0000000755580000, 0x00000007aaa80000)
object space 1398272K, 13% used [0x0000000700000000,0x000000070b4fe090,0x0000000755580000)
PSPermGen total 132608K, used 115511K [0x00000006c0000000, 0x00000006c8180000, 0x0000000700000000)
object space 132608K, 87% used [0x00000006c0000000,0x00000006c70cdfa8,0x00000006c8180000)
}
Event: 60382.794 GC heap before
{Heap before GC invocations=1241 (full 18):
PSYoungGen total 678912K, used 659456K [0x00000007aaa80000, 0x00000007d5580000, 0x0000000800000000)
eden space 658432K, 100% used [0x00000007aaa80000,0x00000007d2d80000,0x00000007d2d80000)
from space 20480K, 5% used [0x00000007d4180000,0x00000007d4280020,0x00000007d5580000)
to space 20480K, 0% used [0x00000007d2d80000,0x00000007d2d80000,0x00000007d4180000)
ParOldGen total 1398272K, used 185336K [0x0000000700000000, 0x0000000755580000, 0x00000007aaa80000)
object space 1398272K, 13% used [0x0000000700000000,0x000000070b4fe090,0x0000000755580000)
PSPermGen total 132608K, used 115511K [0x00000006c0000000, 0x00000006c8180000, 0x0000000700000000)
object space 132608K, 87% used [0x00000006c0000000,0x00000006c70cdfa8,0x00000006c8180000)
Event: 60382.802 GC heap after
Heap after GC invocations=1241 (full 18):
PSYoungGen total 678912K, used 10368K [0x00000007aaa80000, 0x00000007d5580000, 0x0000000800000000)
eden space 658432K, 0% used [0x00000007aaa80000,0x00000007aaa80000,0x00000007d2d80000)
from space 20480K, 50% used [0x00000007d2d80000,0x00000007d37a0010,0x00000007d4180000)
to space 20480K, 0% used [0x00000007d4180000,0x00000007d4180000,0x00000007d5580000)
ParOldGen total 1398272K, used 185368K [0x0000000700000000, 0x0000000755580000, 0x00000007aaa80000)
object space 1398272K, 13% used [0x0000000700000000,0x000000070b506090,0x0000000755580000)
PSPermGen total 132608K, used 115511K [0x00000006c0000000, 0x00000006c8180000, 0x0000000700000000)
object space 132608K, 87% used [0x00000006c0000000,0x00000006c70cdfa8,0x00000006c8180000)
}

Deoptimization events (10 events):
Event: 37770.210 Thread 0x00007f2cb01ab800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f2d1d9eea3c method=java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;)V @ 22
Event: 37793.304 Thread 0x00007f2c5cf22000 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x00007f2d1de867a8 method=java.util.AbstractSet.removeAll(Ljava/util/Collection;)Z @ 51
Event: 41148.828 Thread 0x00007f2c5cde6800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f2d1e557e20 method=org.apache.velocity.runtime.parser.node.ASTMethod.execute(Ljava/lang/Object;Lorg/apache/velocity/context/InternalContextAdapter;)Ljava/lang/Object; @ 124
...
...

Internal exceptions (10 events):
Event: 60200.167 Thread 0x00007f2c5d613800 Threw 0x00000007ccd571a8 at /builddir/build/BUILD/java-1.7.0-openjdk/openjdk/hotspot/src/share/vm/prims/jni.cpp:721
Event: 60200.167 Thread 0x00007f2c5d613800 Threw 0x00000007ccd5db60 at /builddir/build/BUILD/java-1.7.0-openjdk/openjdk/hotspot/src/share/vm/prims/jni.cpp:721
...
...

Events (10 events):
Event: 61096.236 Thread 0x00007f2c5c0b1000 Thread exited: 0x00007f2c5c0b1000
Event: 61096.237 Executing VM operation: RevokeBias
Event: 61096.238 Thread 0x00007f2c5c0b1000 Thread added: 0x00007f2c5c0b1000
...
...

Dynamic libraries:
00400000-00401000 r-xp 00000000 ca:01 136731 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/bin/java
00600000-00601000 rw-p 00000000 ca:01 136731 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/bin/java
006f0000-00732000 rw-p 00000000 00:00 0 [heap]
6c0000000-6c8180000 rw-p 00000000 00:00 0
6c8180000-6cbc00000 ---p 00000000 00:00 0
...
...

VM Arguments:
jvm_args: -Djava.util.logging.config.file=/home/rel-user/jen-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms2048m -Xmx4096m -XX:MaxPermSize=1024m -Dfile.encoding=UTF8 -Dhudson.plugins.openid.impl.TeamsExtensiondisable=true -Djava.endorsed.dirs=/home/rel-user/jen-tomcat/endorsed -Dcatalina.base=/home/rel-user/jen-tomcat -Dcatalina.home=/home/rel-user/jen-tomcat -Djava.io.tmpdir=/home/rel-user/jen-tomcat/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/lib/jvm/jre
PATH=/home/ec2-user/perl/5.18.2/bin:/home/ec2-user/perl/5.18.2/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/bin
SHELL=/bin/bash

--------------- S Y S T E M ---------------

OS:Linux
uname:Linux 3.14.20-20.44.amzn1.x86_64 #1 SMP Mon Oct 6 22:52:46 UTC 2014 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 120223, NOFILE 4096, AS infinity
load average:2.20 3.37 3.55


Memory: 4k page, physical 15404752k(1735452k free), swap 8191996k(8171656k free)

vm_info: OpenJDK 64-Bit Server VM (24.65-b04) for linux-amd64 JRE (1.7.0_65-b17), built on Sep 18 2014 00:37:14 by "mockbuild" with gcc 4.8.2 20140120 (Red Hat 4.8.2-16)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I've removed certain portions that i thought are not relevant. In case you want more details, i can furnish it.
 
Saloon Keeper
Posts: 27763
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
A segfault indicates a bug in the JVM itself. Mere out-of-memory is not sufficient to trigger that.

I recommend that you download a Java 8 JDK and run Jenkins under that, instead. Using another Java 7 version may work - either newer or older than the one you're presently running with, but Java 8 gets rid of the annoying PermGen aspect that has plagued Tomcat since about Tomcat version 5.
 
G Chhabra
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:A segfault indicates a bug in the JVM itself. Mere out-of-memory is not sufficient to trigger that.

I recommend that you download a Java 8 JDK and run Jenkins under that, instead. Using another Java 7 version may work - either newer or older than the one you're presently running with, but Java 8 gets rid of the annoying PermGen aspect that has plagued Tomcat since about Tomcat version 5.



Thanks for the suggestion Tim! I just checked Java 8 features' link.

We are using the following configuration in Tomcat:



Going by this link, looks like i might still have to use MaxMetaspaceSize (instead of MaxPermSize). May be i'll first remove MaxPermSize altogether and see how it goes. Also, it seems Xms & Xmx are still required, correct? If yes, then going forward, i'm planning to keep both Xms & Xmx same.

I will upgrade it to Java 8 and see how it goes. I'll monitor it for sometime and then update this thread. By the way, i was just wondering why we started getting these shutdown issues lately. We did not change Java since last one year.
 
Tim Holloway
Saloon Keeper
Posts: 27763
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If you keep MaxPermsize, the JVM will simply report that it's obsolete and ignore it. I haven't yet had need to override MaxMetaspaceSize, but I'm still migrating myself.

The Xms and Xmx parameters determine the initial and maximum amount of RAM that the JVM will request from the OS for general object storage, so they allow the JVM to use OS resources politely and wisely. I don't expect those 2 options to go away any time soon.
 
Don't get me started about those stupid light bulbs.
reply
    Bookmark Topic Watch Topic
  • New Topic