• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

BEA-000337 StuckThreadMaxTime

 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Our java application runs on weblogic 8.1 SP6 & uses Oracle DB. We also host webservices that are called by other applications. The app. has been very stable and hasn't caused any issues in the last several months. Last Sunday there was some network change (I am not sure on the details, but we were told its a very insignificant change - our database server was recycled to clear the dns cache or something, and we were told it doesnt even require the recycling of our application servers.

However, since then my application fails every now and then on one server after the other. I have had to recycle each of the 8 servers twice this week! Please find below the two type of errors I see in my weblogic logs ever since.

Error # 1 - This is where our webservice is called. (Looking at the threaddump it looks like it fails when trying to get a database connection, although I am not very sure) We use Hibernate/Spring to make a DB connection. We do not use the connectionpool.

+<Error> <WebLogicServer> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '0' for queue: 'weblogic.kernel.Default' has been busy for "885" seconds working on the request "Http Request: /myapp/MyAppAdminService", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>+

Error # 2 - I am not sure if I need to take this error seriously as it does not seem to be there in the threaddump.

+<Error> <WebLogicServer> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '17' for queue: 'weblogic.kernel.Default' has been busy for "921" seconds working on the request "Http Request: /my-portal/view/templates/generic_template.jsp", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>+

We recycled our application servers, weblogic admin server and managed servers on both the clusters today. However that doesnt seem to fix the problem!!

Any help would be appreciated!!

Could this be Database related? (Our DB Server was recycled to clear some DNS cache but I am not sure what else I can do other than recycling my app servers if this is the issue. Please note that the webservice fails sporadically and does not fail all the time) If this is DB related, what do I check with our DBA?

Could this be Weblogic related?

Could this be App Server / Memory / Load Balancer related?

Would redeploying my app help?

Can I do anything else to debug this further?

Any help would be greatly appreciated!! Please find the threaddump below -

===== FULL THREAD DUMP ===============
Wed Feb 3 14:44:41 2010
BEA JRockit(R) R27.2.0-131-78843-1.4.2_13-20070320-1511-linux-ia32


"ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" id=9 idx=0x38 tid=28257 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: org/apache/commons/pool/impl/GenericObjectPool@0x1c202e20[fat lock]
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at java/lang/Object.wait(Object.java:429)
at org/apache/commons/pool/impl/GenericObjectPool.borrowObject(GenericObjectPool.java:810)
^-- Lock released while waiting: org/apache/commons/pool/impl/GenericObjectPool@0x1c202e20[fat lock]
at org/apache/commons/dbcp/PoolingDataSource.getConnection(PoolingDataSource.java:95)
at org/apache/commons/dbcp/BasicDataSource.getConnection(BasicDataSource.java:540)
at org/springframework/orm/hibernate3/LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at org/hibernate/jdbc/ConnectionManager.openConnection(ConnectionManager.java:417)
at org/hibernate/jdbc/ConnectionManager.getConnection(ConnectionManager.java:144)
at org/hibernate/jdbc/BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:40)
at $Proxy9.setReadOnly(Z)V(Unknown Source)
at org/springframework/jdbc/datasource/DataSourceUtils.prepareConnectionForTransaction(DataSourceUtils.java:156)
at org/springframework/orm/hibernate3/HibernateTransactionManager.doBegin(HibernateTransactionManager.java:444)
at org/springframework/transaction/support/AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:322)
at org/springframework/transaction/interceptor/TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:255)
at org/springframework/transaction/interceptor/TransactionInterceptor.invoke(TransactionInterceptor.java:102)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at org/springframework/aop/framework/Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:615)
at com/abc/amt/My/adapter/rrgt/generated/ItemDataHome$$EnhancerByCGLIB$$8e2fb3ac.findByPersonId(Ljava/lang/String;)Ljava/util/List;(<generated>)
at com/abc/amt/My/model/rrgt/RequestFactory.getDetails1(RequestFactory.java:339)
at com/abc/amt/My/adapter/rrgt/webservice/internal/generated/My55h4RequestAdminServiceImpl.getDetails1(My55h4RequestAdminServiceImpl.java:570)
at sun/reflect/GeneratedMethodAccessor271.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:324)[optimized]
at weblogic/webservice/component/javaclass/JavaClassInvocationHandler.invoke(JavaClassInvocationHandler.java:133)
at weblogic/webservice/core/handler/InvokeHandler.handleRequest(InvokeHandler.java:104)
at weblogic/webservice/core/HandlerChainImpl.handleRequest(HandlerChainImpl.java:143)
at weblogic/webservice/core/DefaultOperation.process(DefaultOperation.java:549)
at weblogic/webservice/server/Dispatcher.process(Dispatcher.java:204)
at weblogic/webservice/server/Dispatcher.doDispatch(Dispatcher.java:176)
at weblogic/webservice/server/Dispatcher.dispatch(Dispatcher.java:96)
at weblogic/webservice/server/WebServiceManager.dispatch(WebServiceManager.java:100)
at weblogic/webservice/server/servlet/WebServiceServlet.serverSideInvoke(WebServiceServlet.java:297)
at weblogic/webservice/server/servlet/ServletBase.doPost(ServletBase.java:498)
at weblogic/webservice/server/servlet/WebServiceServlet.doPost(WebServiceServlet.java:267)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:760)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:853)
at weblogic/servlet/internal/ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)[inlined]
at weblogic/servlet/internal/ServletStubImpl.invokeServlet(ServletStubImpl.java:465)[inlined]
at weblogic/servlet/internal/ServletStubImpl.invokeServlet(ServletStubImpl.java:348)[optimized]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7047)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[inlined]
at weblogic/servlet/internal/WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)[optimized]
at weblogic/servlet/internal/ServletRequestImpl.execute(ServletRequestImpl.java:2773)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:224)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace
 
Bartender
Posts: 6663
5
MyEclipse IDE Firefox Browser Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well there is no error here as such. The message simply means that the pages are taking more than 10 minutes to load, which is the maximum time that a thread can be kept in the running state before it is marked as STUCK by weblogic.

What is your definition of recycled ? Ask the network guys more details about the change.

Restarting things without knowing the root cause is not advisable.

You have a data source do you not ? What does the monitoring tab on that data source say ? What is the wait time before connections are served and what is the max concurrent connections that were served ? What is the max connections that the DB can serve and is it in line with the number of pooled connections on your data source ?
 
Let nothing stop you! Not even this tiny ad:
a bit of art, as a gift, that will fit in a stocking
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic