• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Tomcat Crashes with Oracle Super Cluster

 
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello,

Tomcat version 7x crashes randomly when used with Oracle Super Cluster servers.
There is a big OS Binary dump generated after this. OS: Solaris 11 SPARC based 64-bit system.
No Thread dumps/Heap Dumps/JRE dumps generated for analysis. catalina.out ends abruptly without errors.
The system has Java JDK 7 installed.

The issue appeared suddenly after switching from blade servers to oracle super cluster servers.

How should we set JVM parameters in start.sh or configuration of this tomcat so that it starts generating heap dumps or thread dumps for such memory crashes?

What could be the cause of such a random issue?



--
Regards,
tomcatadmindev
 
Marie Augustus
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Can someone please help with resolving the crash issue?
 
Saloon Keeper
Posts: 27752
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Tomcat doesn't typically "crash" as such. However, the JVM that's running Tomcat may crash. In most cases a JVM dumpfile will be saved in the Tomcat home directory.

If something REALLY bad happened, then the normal JVM dump process itself might get caught up in the crash, in which case a system core dump should be obtainable (if the OS was properly configured).

In either event, typically the crash is specific to that particular JVM version and running under a different version of the JVM may help. That's because JVM's aren't supposed to crash at all, therefore you're not looking at a bug in the webapp or in Tomcat, but rather it's a bug in the JVM. Which hopefully doesn't exist in an alternate version (or brand) of JVM.
 
Marie Augustus
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for the info Tim.
In this case JVM dump is not getting generated.
How can we configure tomcat for generating JVM dumps on crash?

However, i am getting a system binary core dump of around 3GB.
Are there any methods by which we can analyze this system machine language dump to find out cause of the crash?
Tomcat abruptly stops after the crash and there are no abnormal error messages recorded under catalina,localhost or other logs.

--
Regards,
Marie
 
Tim Holloway
Saloon Keeper
Posts: 27752
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, if you're into binary debugging, you COULD analyze the coredump, but since you don't have the source code of the JVM, you'd have to extensively reverse-engineer it. Which, thanks to the DMCA is probably illegal these days, unlike when I was in college and learned much of what I know about compilers and OS's from hand-disassembly of core dumps.

So the core dump file is mostly only useful if you can send it to Oracle support. Since you got a binary core dump, don't expect to find a JVM dump, since the mere fact that you got a system dump means that the JVM failed so hard that it couldn't instigate its own diagnostics.

Vendor support being what it is these days, your quickest solution is probably to find a different JVM to run it under. If you're using jRocket, switch to the stock Sun JVM. Or vice versa. You might even consider trying IBM's J9 JVM.
 
Marie Augustus
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am using Sun Java SE JDK version 1.7.0_79 .
Could using a higher version from the same vendor be helpful?
Im afraid, changing the vendor wont be possible.

Also, the issue started to appear only after system was switched from blade servers to oracle super cluster servers.
Wonder how this is a contributing factor.

.profile options have the following memory parameters allocated
CATALINA_OPTS="-Xms1024m -Xmx8192m"

Is there anyway root cause of the crash be found?
If the crash is due bug in JVM, can changing JVM memory parameters prove helpful?

--
Regards,
Marie


 
Tim Holloway
Saloon Keeper
Posts: 27752
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
It doesn't matter what parameters you set or what product you are running. JVM's are not supposed to crash, and they are definitely not supposed to crash so hard thay can't take JVM dumps.

You don't necessarily need a newer JVM. Sometimes, in fact, the bug comes in and runs through several JVM releases before it gets fixed and thus an older JVM will do the job as well or better. As long as it doesn't expose any security concerns.

If you are constrained in the options you may use to work around the problem, all I can do is suggest that you contact Oracle Support.
 
Marie Augustus
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have already contacted oracle application support with core system dump. As tomcat is a third party software, supporting any issues related to crash would require a separate Java SE license support.
I was thinking of a possibility that maybe until and unless we explicitly specify java iptions of generating a heap dump in catalina file, JVM crash dump doesnt get generated. Please clarify of this could be one possible scenario.
 
Tim Holloway
Saloon Keeper
Posts: 27752
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I'm pretty sure that in case of a JVM crash a heap dump generates by default if it can, based on junk I have seen show up in my TOMCAT_HOME directories. You can consult the Oracle Sun documentation just to be sure. The heap dump mechanism is Sun-specific, therefore so are any controlling options. Which means that only Oracle is responsible for documenting those options, which should all begin with "-X", just like other vendor-specific extended options do.

Tomcat doesn't run under Java Standard Edition, and assuming you have a JDK support license, I'd doubt their word about the need for a separate SE licence. But then again, this is Oracle we're talking about.
 
Marie Augustus
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Tim! Really appreciate your help.
 
Marie Augustus
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Tim,

We tried many options but nothing worked as of now.
Finally i am going to try to move to JDK8 as the last resort.
Can you please suggest which is the most stable version of JDK8 that can be used?

--
Regards,
Marie
 
Tim Holloway
Saloon Keeper
Posts: 27752
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I'm afraid that I don't monitor individual JVM releases closely enough to know which ones are more or less stable. And of course "stable" is a relative term, since a JVM release that works fine for me might crash on you and [i]vice versa.[/]

On the other hand, JVM releases have historically not been so flakey that you cannot find a "safe" version in 2 or 3 random tries. And I would hope that Oracle intends to adhere to that level of quality.

Beyond that, about all I can say is make sure you have enough memory.
 
Marie Augustus
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks!
I will be moving the system to latest JDK8 version.
Will update if this resolves the issue.

--
Regards,
Marie.
 
reply
    Bookmark Topic Watch Topic
  • New Topic