• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

OutOfMemory error but application continues

 
vijayashankar dmoorthy
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

I have faced OutOfMemory error in my application. But the thing is my application keeps on running though it has thrown OutOfMemeory. After some two minutes later, JVM has quit and pid file got genearted throwing java.lang.OutOfMemoryError: requested 746 bytes for jbyte in /BUILD_AREA/jdk6_21/hotspot/src/share/vm/prims/jni.cpp. Out of swap space?.

Is it possible for JVM to continue running even after throwing OutOfMemory error ?
Also can you give me some direction to investigate on this further?


I am pasting the excerpt from log to get a better understanding.


1) There are two out of memory errors. One at 2011.12.09 16.04.09:446 and another at 2011.12.09 16.04.40:818

2011.12.09 16.04.09:446 664849490 WARNING {HTTP@9800-168}RULEZ runtime exception [transcoding]
java.lang.OutOfMemoryError
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:238)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:135)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105)
at java.util.jar.JarVerifier$VerifierStream.read(JarVerifier.java:428)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:75)
at javax.mail.Session.loadProvidersFromStream(Session.java:932)
at javax.mail.Session.access$000(Session.java:174)
at javax.mail.Session$1.load(Session.java:870)
at javax.mail.Session.loadResource(Session.java:1084)
at javax.mail.Session.loadProviders(Session.java:889)
at javax.mail.Session.<init>(Session.java:210)
at javax.mail.Session.getInstance(Session.java:249)
.
.
.[Application continues here]
.
.
.
2011.12.09 16.04.40:818 664882092 WARNING {HTTP@9800-168}RULEZ runtime exception [transcoding]
java.lang.OutOfMemoryError
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:238)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:135)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105)
at java.util.jar.JarVerifier$VerifierStream.read(JarVerifier.java:428)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:75)
at javax.mail.Session.loadProvidersFromStream(Session.java:932)
at javax.mail.Session.access$000(Session.java:174)
at javax.mail.Session$1.load(Session.java:870)
at javax.mail.Session.loadResource(Session.java:1084)
at javax.mail.Session.loadProviders(Session.java:889)
at javax.mail.Session.<init>(Session.java:210)
at javax.mail.Session.getInstance(Session.java:249)
.
.
.
.
.

2) At Dec 9 16:07,JVM quits and created pid file as below. So, there is ~2 minutes gap between OutOFMemory error log statement and for jvm to quit and create pid file.

$ls

-rw-rw-r-- 1 ins ins 156076 Dec 9 16:07 hs_err_pid29157.log
-rw------- 1 ins ins 3055525888 Dec 9 16:07 core.29157


#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 746 bytes for jbyte in /BUILD_AREA/jdk6_21/hotspot/src/share/vm/prims/jni.cpp. Out of swap space?
#
# Internal Error (allocation.inline.hpp:39), pid=21129, tid=667401120
# Error: jbyte in /BUILD_AREA/jdk6_21/hotspot/src/share/vm/prims/jni.cpp
#
# JRE version: 6.0_21-b06
# Java VM: Java HotSpot(TM) Server VM (17.0-b16 mixed mode linux-x86 )
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x2f929000): JavaThread "DRIVER:adapter:12:7" [_thread_in_vm, id=21834, stack(0x27c5b000,0x27c7c000)]

Stack: [0x27c5b000,0x27c7c000], sp=0x27c7a874, free space=7e27c7c000k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x6a9262]
V [libjvm.so+0x2b277f]
V [libjvm.so+0x3c0b0f]
C [libocijdbc10.so+0x108ff]
C [libocijdbc10.so+0x11788] Java_oracle_jdbc_driver_T2CConnection_lobGetLength+0x30
J oracle.jdbc.driver.T2CConnection.lobGetLength(J[BI)J

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J oracle.jdbc.driver.T2CConnection.lobGetLength(J[BI)J
J oracle.jdbc.driver.T2CConnection.length(Loracle/sql/BLOB;)J
J com.fh.common.database.ReadConnectionImpl.getByteArray(Ljava/lang/String;Ljava/sql/ResultSet;I)[B
J com.fh.common.database.ReadConnectionWrapper.getByteArray(Ljava/lang/String;Ljava/sql/ResultSet;I)[B
J com.fh.mr.storage.MultiJDBCStorage.fillInitiator(Lcom/fh/common/database/ReadConnection;Ljava/lang/String;Ljava/sql/ResultSet;Lcom/fh/mr/router/TransactionResponder;)Lcom/fh/mr/router/TransactionInitiator;
J com.fh.mr.storage.MultiJDBCStorage.addInitiators(Lcom/fh/mr/router/TransactionResponder;Ljava/util/Collection;)V
J com.fh.mr.storage.CacheStorage.getResponderByTransactionID(Ljava/lang/String;)Lcom/fh/mr/router/TransactionResponder;
j com.fh.mr.router.RoutingEngine.getResponderByTransactionID(Ljava/lang/String;)Lcom/fh/mr/router/TransactionResponder;+5
j com.fh.mr.router.RoutingEngine.scheduleStatusResend(Ljava/lang/String;)Z+53
j com.fh.mr.router.RoutingEngine.scheduleResend(Lcom/fh/mr/message/Message;I)Z+68
j com.fh.mr.drivers.Driver.getMessage(Ljava/lang/Object;)Lcom/fh/mr/message/Message;+260
J com.fh.mr.drivers.Driver$QueueWorker.run()V
j com.fh.common.threadpool.ThreadControl.run()V+47
j com.fh.common.threadpool.ThreadPool$PoolThread.run()V+467
v ~StubRoutines::call_stub


Thanks,
Vijay
 
Akhilesh Trivedi
Ranch Hand
Posts: 1608
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
vijayashankar dmoorthy wrote: I am pasting the excerpt from log to get a better understanding.

1) There are two out of memory errors. One at 2011.12.09 16.04.09:446 and another at 2011.12.09 16.04.40:818



Are the log entries right with time of execution? There are times we keep looking at old logs.
 
vijayashankar dmoorthy
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Akhilesh,

Yes. The logs are for the same day from the same machine.

Thanks,
Vijay
 
Rishi Shehrawat
Ranch Hand
Posts: 218
Hibernate Java Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
This type of behaviour is normal with OOM. When JVM encounters OOM it tries to clear memory by running full GC's. After some time if it is not able to clear up memory the JVM crashes. In case you enable garbage collection logging, you will see that full GC's are running & the JVM is not able to clear memory.

From the thread dumps it seems like you are doing something with blob, try looking into this code. In case this stack is there in all error files there is a good possibility that this is the cause. Another option is that you can ask JVM to produce heap dump when OOM is thrown. You can then look at the type of objects in the heap & how these objects are being created.
 
Jesus Angeles
Ranch Hand
Posts: 2068
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Just because an exception is thrown and dumped doesnt mean something will crash does it? A simple 'try catch' could be there to recover what is possibly still a good running server instance - no need to crash.

This happened to me also.

Tomcat occupied 90% of my memory, and Oracle DB and other services left with 10% to use. It could be my hibernate caches. I am trying to improve my hibernate entity xmls now, as simple stuff like 'lazy loading' could end up loading entire DB into memory.
 
Jeff Verdegan
Bartender
Posts: 6109
6
Android IntelliJ IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Rishi Shehrawat wrote:This type of behaviour is normal with OOM. When JVM encounters OOM it tries to clear memory by running full GC's. After some time if it is not able to clear up memory the JVM crashes.


No, that's backwards. First the JVM does all the GC it can, and then if it still doesn't have enough free heap memory to meet the request it throws OOME.
 
Chris Hurst
Ranch Hand
Posts: 443
3
C++ Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
As of Java 5, if GC is determined to take too long it throws i.e. its not strictly if the memory was available. Heap exhaustion and GC taking too long will look the same.
 
Jeff Verdegan
Bartender
Posts: 6109
6
Android IntelliJ IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Chris Hurst wrote:As of Java 5, if GC is determined to take too long it throws i.e. its not strictly if the memory was available. Heap exhaustion and GC taking too long will look the same.


True. I think the default is something like if the JVM is spending more than 98% of its time on GC and GC results in less than X% free heap, or something along those lines, then OOME is thrown.

Regardless of the underlying cause for a particular OOME, however it is still true that OOME can be caught and the app can continue (in some cases), and if the heap is insufficient for a given request, GC will be attempted before OOME is thrown.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic