Our
java application runs on weblogic 8.1 SP6 & uses Oracle DB. We also host webservices that are called by other applications. The app. has been very stable and hasn't caused any issues in the last several months. Last Sunday there was some network change (I am not sure on the details, but we were told its a very insignificant change - our database server was recycled to clear the dns cache or something, and we were told it doesnt even require the recycling of our application servers.
However, since then my application fails every now and then on one server after the other. I have had to recycle each of the 8 servers twice this week! Please find below the two type of errors I see in my weblogic logs ever since.
Error # 1 - This is where our webservice is called. (Looking at the threaddump it looks like it fails when trying to get a database connection, although I am not very sure) We use Hibernate/Spring to make a DB connection. We do not use the connectionpool.
+<Error> <WebLogicServer> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '0' for queue: 'weblogic.kernel.Default' has been busy for "885" seconds working on the request "Http Request: /myapp/MyAppAdminService", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>+
Error # 2 - I am not sure if I need to take this error seriously as it does not seem to be there in the threaddump.
+<Error> <WebLogicServer> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '17' for queue: 'weblogic.kernel.Default' has been busy for "921" seconds working on the request "Http Request: /my-portal/view/templates/generic_template.jsp", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>+
We recycled our application servers, weblogic admin server and managed servers on both the clusters today. However that doesnt seem to fix the problem!!
Any help would be appreciated!!
Could this be Database related? (Our DB Server was recycled to clear some DNS cache but I am not sure what else I can do other than recycling my app servers if this is the issue. Please note that the webservice fails sporadically and does not fail all the time) If this is DB related, what do I check with our DBA?
Could this be Weblogic related?
Could this be App Server / Memory / Load Balancer related?
Would redeploying my app help?
Can I do anything else to debug this further?
Any help would be greatly appreciated!! Please find the threaddump below -
===== FULL
THREAD DUMP ===============
Wed Feb 3 14:44:41 2010
BEA JRockit(R) R27.2.0-131-78843-1.4.2_13-20070320-1511-linux-ia32
"ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" id=9 idx=0x38 tid=28257 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: org/apache/commons/pool/impl/GenericObjectPool@0x1c202e20[fat lock]
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at java/lang/Object.wait(Object.java:429)
at org/apache/commons/pool/impl/GenericObjectPool.borrowObject(GenericObjectPool.java:810)
^-- Lock released while waiting: org/apache/commons/pool/impl/GenericObjectPool@0x1c202e20[fat lock]
at org/apache/commons/dbcp/PoolingDataSource.getConnection(PoolingDataSource.java:95)
at org/apache/commons/dbcp/BasicDataSource.getConnection(BasicDataSource.java:540)
at org/springframework/orm/hibernate3/LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at org/hibernate/jdbc/ConnectionManager.openConnection(ConnectionManager.java:417)
at org/hibernate/jdbc/ConnectionManager.getConnection(ConnectionManager.java:144)
at org/hibernate/jdbc/BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:40)
at $Proxy9.setReadOnly(Z)V(Unknown Source)
at org/springframework/jdbc/datasource/DataSourceUtils.prepareConnectionForTransaction(DataSourceUtils.java:156)
at org/springframework/orm/hibernate3/HibernateTransactionManager.doBegin(HibernateTransactionManager.java:444)
at org/springframework/transaction/support/AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:322)
at org/springframework/transaction/interceptor/TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:255)
at org/springframework/transaction/interceptor/TransactionInterceptor.invoke(TransactionInterceptor.java:102)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at org/springframework/aop/framework/Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:615)
at com/abc/amt/My/adapter/rrgt/generated/ItemDataHome$$EnhancerByCGLIB$$8e2fb3ac.findByPersonId(Ljava/lang/String;)Ljava/util/List;(<generated>)
at com/abc/amt/My/model/rrgt/RequestFactory.getDetails1(RequestFactory.java:339)
at com/abc/amt/My/adapter/rrgt/webservice/internal/generated/My55h4RequestAdminServiceImpl.getDetails1(My55h4RequestAdminServiceImpl.java:570)
at sun/reflect/GeneratedMethodAccessor271.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:324)[optimized]
at weblogic/webservice/component/javaclass/JavaClassInvocationHandler.invoke(JavaClassInvocationHandler.java:133)
at weblogic/webservice/core/handler/InvokeHandler.handleRequest(InvokeHandler.java:104)
at weblogic/webservice/core/HandlerChainImpl.handleRequest(HandlerChainImpl.java:143)
at weblogic/webservice/core/DefaultOperation.process(DefaultOperation.java:549)
at weblogic/webservice/server/Dispatcher.process(Dispatcher.java:204)
at weblogic/webservice/server/Dispatcher.doDispatch(Dispatcher.java:176)
at weblogic/webservice/server/Dispatcher.dispatch(Dispatcher.java:96)
at weblogic/webservice/server/WebServiceManager.dispatch(WebServiceManager.java:100)
at weblogic/webservice/server/servlet/WebServiceServlet.serverSideInvoke(WebServiceServlet.java:297)
at weblogic/webservice/server/servlet/ServletBase.doPost(ServletBase.java:498)
at weblogic/webservice/server/servlet/WebServiceServlet.doPost(WebServiceServlet.java:267)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:760)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:853)
at weblogic/servlet/internal/ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)[inlined]
at weblogic/servlet/internal/ServletStubImpl.invokeServlet(ServletStubImpl.java:465)[inlined]
at weblogic/servlet/internal/ServletStubImpl.invokeServlet(ServletStubImpl.java:348)[optimized]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7047)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[inlined]
at weblogic/servlet/internal/WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)[optimized]
at weblogic/servlet/internal/ServletRequestImpl.execute(ServletRequestImpl.java:2773)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:224)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace