• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Another Stuck Thread Question - SocketNativeIO.readBytesPinned - PreparedStatement

 
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
We're getting several STUCK threads like the one below. These never clear up, the managed servers must be bounced to remove them. The STUCK threads in question are processing a prepared statement and hang indefinately on the socket read (readBytesPinned). We are tracking down the exact query involved. Enabling traces is a "process" in the environment where this happens.

How can I verify that the DB is actively processing this request?
Would the Oracle DB Alert Logs show that there is a/are long running requests?
Getting a DB Alert Log is a challenge, it has to be requested; should I ask for any particular options that would help identify long running connections or connections that have timed out/dropped?

Would a thread remain in the STUCK state even if the connection dropped? I would like to know what the thread is doing at method readBytesPinned. I'm Stuck

Has anyone tracked down these kind of thread issues? If so, will you please share any lessons learned?

This thread stack taken from a thread dump. We have taken several dumps and the thread never appears to have any progress.

Advisories: Stuck Thread DB Execute Socket Read Web Application Request
"[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x68 tid=13619 prio=1 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at oracle/net/nt/MetricsEnabledInputStream.read(TcpNTAdapter.java:564)[optimized]
at oracle/net/ns/Packet.receive(Packet.java:295)[inlined]
at oracle/net/ns/DataPacket.receive(DataPacket.java:94)[optimized]
at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:176)[optimized]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:121)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:96)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:81)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)[inlined]
at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1087)[optimized]
at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)[inlined]
at oracle/jdbc/driver/T4C8Oall.receive(T4C8Oall.java:625)[optimized]
at oracle/jdbc/driver/T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)[inlined]
at oracle/jdbc/driver/T4CPreparedStatement.fetch(T4CPreparedStatement.java:1121)[optimized]
at oracle/jdbc/driver/OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:571)[inlined]
at oracle/jdbc/driver/OracleResultSetImpl.next(OracleResultSetImpl.java:445)[optimized]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x14e5531a0[recursive]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x14e5531a0[thin lock]
at weblogic/jdbc/wrapper/ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next()Z(Unknown Source)[optimized]
at org/hibernate/loader/Loader.doQuery(Loader.java:697)[inlined]
at org/hibernate/loader/Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)[inlined]
at org/hibernate/loader/Loader.doList(Loader.java:2213)[inlined]
at org/hibernate/loader/Loader.listIgnoreQueryCache(Loader.java:2104)[optimized]
at org/hibernate/loader/Loader.list(Loader.java:2099)
at org/hibernate/loader/hql/QueryLoader.list(QueryLoader.java:378)
at org/hibernate/hql/ast/QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
at org/hibernate/engine/query/HQLQueryPlan.performList(HQLQueryPlan.java:172)
at org/hibernate/impl/SessionImpl.list(SessionImpl.java:1121)
at org/hibernate/impl/QueryImpl.list(QueryImpl.java:79)
at org/hibernate/ejb/QueryImpl.getResultList(QueryImpl.java:65)
at org/springframework/orm/jpa/JpaTemplate$11.doInJpa(JpaTemplate.java:351)
at org/springframework/orm/jpa/JpaTemplate.execute(JpaTemplate.java:184)
at org/springframework/orm/jpa/JpaTemplate.executeFind(JpaTemplate.java:151)
at org/springframework/orm/jpa/JpaTemplate.findByNamedQuery(JpaTemplate.java:343)
at aiat/csp/esb2/persistence/PersistenceManagerImpl.findByNamedQuery(PersistenceManagerImpl.java:224)
at com/aiat/odlp/commonutilities/referencedata/dao/ReferenceDataManagerDAO.getAirlineList(ReferenceDataManagerDAO.java:725)[optimized]
at com/aiat/odlp/commonutilities/referencedata/impl/ReferenceDataManagerImpl.listAirlineCodes(ReferenceDataManagerImpl.java:177)[optimized]
at sun/reflect/GeneratedMethodAccessor7000.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at org/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org/springframework/aop/framework/ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org/springframework/transaction/interceptor/TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org/springframework/aop/interceptor/ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org/springframework/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy247.listAirlineCodes(Lcom/aiat/odlp/commonutilities/referencedata/interfaces/common/ListAirlinesQueryDTO;)Lcom/aiat/odlp/commonutilities/referencedata/interfaces/common/AirlineListDTO;(Unknown Source)
at com/aiat/odlp/commonutilities/referencedata/endpoint/ReferenceDataRepositoryListEndPoint.listAirlines(ReferenceDataRepositoryListEndPoint.java:78)
at sun/reflect/GeneratedMethodAccessor6999.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at org/springframework/ws/server/endpoint/MethodEndpoint.invoke(MethodEndpoint.java:115)
at org/springframework/ws/server/endpoint/adapter/MarshallingMethodEndpointAdapter.invokeInternal(MarshallingMethodEndpointAdapter.java:135)
at org/springframework/ws/server/endpoint/adapter/AbstractMethodEndpointAdapter.invoke(AbstractMethodEndpointAdapter.java:58)
at org/springframework/ws/server/MessageDispatcher.dispatch(MessageDispatcher.java:236)
at org/springframework/ws/server/MessageDispatcher.receive(MessageDispatcher.java:177)
at com/aiat/odlp/server/endpoint/jaxws/JaxWsRequestHandler.handleRequest(JaxWsRequestHandler.java:96)
at com/aiat/odlp/server/endpoint/jaxws/VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:75)
at com/aiat/odlp/server/endpoint/jaxws/VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:38)
at sun/reflect/GeneratedMethodAccessor958.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at weblogic/wsee/jaxws/WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:92)
at weblogic/wsee/jaxws/WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:74)
at com/sun/xml/ws/api/server/Invoker.invokeProvider(Invoker.java:108)
at com/sun/xml/ws/server/InvokerTube$2.invokeProvider(InvokerTube.java:161)
at com/sun/xml/ws/server/provider/SyncProviderInvokerTube.processRequest(SyncProviderInvokerTube.java:78)
at com/sun/xml/ws/api/pipe/Fiber.__doRun(Fiber.java:604)[optimized]
at com/sun/xml/ws/api/pipe/Fiber._doRun(Fiber.java:563)
at com/sun/xml/ws/api/pipe/Fiber.doRun(Fiber.java:548)
at com/sun/xml/ws/api/pipe/Fiber.runSync(Fiber.java:445)
^-- Holding lock: com/sun/xml/ws/api/pipe/Fiber@0x14e5349d0[thin lock]
at com/sun/xml/ws/server/WSEndpointImpl$2.process(WSEndpointImpl.java:373)
at com/sun/xml/ws/transport/http/HttpAdapter$HttpToolkit.handle(HttpAdapter.java:524)
at com/sun/xml/ws/transport/http/HttpAdapter.handle(HttpAdapter.java:255)
at com/sun/xml/ws/transport/http/servlet/ServletAdapter.handle(ServletAdapter.java:140)
at weblogic/wsee/jaxws/WLSServletAdapter.handle(WLSServletAdapter.java:208)
at weblogic/wsee/jaxws/HttpServletAdapter$AuthorizedInvoke.run(HttpServletAdapter.java:310)
at weblogic/wsee/jaxws/HttpServletAdapter.post(HttpServletAdapter.java:223)
at weblogic/wsee/jaxws/JAXWSServlet.doPost(JAXWSServlet.java:124)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:727)
at weblogic/wsee/jaxws/JAXWSServlet.service(JAXWSServlet.java:79)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)
at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/dms/wls/DMSServletFilter.doFilter(DMSServletFilter.java:330)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.doIt(Lweblogic/servlet/internal/ServletStubImpl;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)Ljava/lang/Object;(Unknown Source)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;(Unknown Source)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[inlined]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[optimized]
at weblogic/servlet/internal/WebAppServletContext.securedExecute(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V(Unknown Source)
at weblogic/servlet/internal/WebAppServletContext.execute(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V(Unknown Source)
at weblogic/servlet/internal/ServletRequestImpl.run()V(Unknown Source)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)[optimized]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)


The managed server logs are showing "similar" issues but to make things interesting the line numbers shown in the stack traces are different. I am not sure I am comparing apples to apples. Here is an example of a similar thread excpetion from the Weblogic logs.

<Mar 21, 2014 12:59:35 PM GMT+00:00> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "605" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@2b80c9fa[
POST /rd-1.1.2/ReferenceData-1.1?wsdl HTTP/1.1
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://aiat/odlp/services/commonutilities/ReferenceDataRepository/v.0.0.0-28/action/ListAirlines/"
Transfer-Encoding: chunked
User-Agent: Java1.6.0_33
Accept: text/html, image/gif, image/jpeg, */*; q=.2
Connection: Keep-Alive

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-15 "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
oracle.net.ns.Packet.receive(Packet.java:249)
oracle.net.ns.DataPacket.receive(DataPacket.java:94)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1091)
oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:480)
oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:347)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@2d660b1e[thin lock]
^-- Holding lock: oracle.jdbc.driver.T4CConnection@2d660b1e[thin lock]
weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
org.hibernate.loader.Loader.doQuery(Loader.java:665)
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:232)
org.hibernate.loader.Loader.doList(Loader.java:2207)
org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
org.hibernate.loader.Loader.list(Loader.java:2092)
org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:377)
org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:321)
org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:149)
org.hibernate.impl.SessionImpl.list(SessionImpl.java:1110)
org.hibernate.impl.QueryImpl.list(QueryImpl.java:75)
org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:65)
org.springframework.orm.jpa.JpaTemplate$11.doInJpa(JpaTemplate.java:345)
org.springframework.orm.jpa.JpaTemplate.execute(JpaTemplate.java:169)
org.springframework.orm.jpa.JpaTemplate.executeFind(JpaTemplate.java:151)
org.springframework.orm.jpa.JpaTemplate.findByNamedQuery(JpaTemplate.java:343)
aiat.csp.esb2.persistence.PersistenceManagerImpl.findByNamedQuery(PersistenceManagerImpl.java:224)
com.aiat.odlp.commonutilities.referencedata.dao.ReferenceDataManagerDAO.getAirlineList(ReferenceDataManagerDAO.java:674)
com.aiat.odlp.commonutilities.referencedata.impl.ReferenceDataManagerImpl.listAirlineCodes(ReferenceDataManagerImpl.java:169)
sun.reflect.GeneratedMethodAccessor7000.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:306)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:92)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy247.listAirlineCodes(Unknown Source)
com.aiat.odlp.commonutilities.referencedata.endpoint.ReferenceDataRepositoryListEndPoint.listAirlines(ReferenceDataRepositoryListEndPoint.java:75)
sun.reflect.GeneratedMethodAccessor6999.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.springframework.ws.server.endpoint.MethodEndpoint.invoke(MethodEndpoint.java:109)
org.springframework.ws.server.endpoint.adapter.MarshallingMethodEndpointAdapter.invokeInternal(MarshallingMethodEndpointAdapter.java:133)
org.springframework.ws.server.endpoint.adapter.AbstractMethodEndpointAdapter.invoke(AbstractMethodEndpointAdapter.java:58)
org.springframework.ws.server.MessageDispatcher.dispatch(MessageDispatcher.java:211)
org.springframework.ws.server.MessageDispatcher.receive(MessageDispatcher.java:167)
com.aiat.odlp.server.endpoint.jaxws.JaxWsRequestHandler.handleRequest(JaxWsRequestHandler.java:90)
com.aiat.odlp.server.endpoint.jaxws.VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:66)
com.aiat.odlp.server.endpoint.jaxws.VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:38)
sun.reflect.GeneratedMethodAccessor958.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:92)
weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:73)
com.sun.xml.ws.api.server.Invoker.invokeProvider(Invoker.java:108)
com.sun.xml.ws.server.InvokerTube$2.invokeProvider(InvokerTube.java:159)
com.sun.xml.ws.server.provider.SyncProviderInvokerTube.processRequest(SyncProviderInvokerTube.java:70)
com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:580)
com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:553)
com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:535)
com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:433)
^-- Holding lock: com.sun.xml.ws.api.pipe.Fiber@2d35f1bf[thin lock]
com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:352)
com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:508)
com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:230)
com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:140)
weblogic.wsee.jaxws.WLSServletAdapter.handle(WLSServletAdapter.java:193)
weblogic.wsee.jaxws.HttpServletAdapter$AuthorizedInvoke.run(HttpServletAdapter.java:310)
weblogic.wsee.jaxws.HttpServletAdapter.post(HttpServletAdapter.java:187)
weblogic.wsee.jaxws.JAXWSServlet.doPost(JAXWSServlet.java:124)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
weblogic.wsee.jaxws.JAXWSServlet.service(JAXWSServlet.java:76)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(Unknown Source)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(Unknown Source)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
weblogic.servlet.internal.WebAppServletContext.securedExecute(Unknown Source)
weblogic.servlet.internal.WebAppServletContext.execute(Unknown Source)
weblogic.servlet.internal.ServletRequestImpl.run(Unknown Source)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}

>

Any help/guidance would be really appreciated.


Start of Managed Server log with envt details:
starting weblogic with Java version:
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
Oracle JRockit(R) (build R28.2.4-14-151097-1.6.0_33-20120618-1634-linux-x86_64, compiled mode)
Starting WLS with line:
/apps/sw/oracle/mw/jrockit-x64/bin/java -jrockit -Xms4096m -Xmx4096m -Dweblogic.Name=VWS-App-4
-Djava.security.policy=/apps/sw/oracle/mw/wls1033/wlserver_10.3/server/lib/weblogic.policy
-Dweblogic.system.BootIdentityFile=/apps/services/mw/VGR-WLS-STG/servers/VWS-App-4/data/nodemanager/boot.properties
-Dweblogic.nodemanager.ServiceEnabled=true
-Dweblogic.security.SSL.ignoreHostnameVerification=true
-Dweblogic.ReverseDNSAllowed=false
-Dweblogic.Stdout=/logs/servers/VWS-App-4/VWS-App-4.out
-Xverify:none -da -Dplatform.home=/apps/sw/oracle/mw/wls1033/wlserver_10.3
-Dwls.home=/apps/sw/oracle/mw/wls1033/wlserver_10.3/server
-Dweblogic.home=/apps/sw/oracle/mw/wls1033/wlserver_10.3/server
-Ddomain.home=/apps/services/mw/VGR-WLS-STG
-Dcommon.components.home=/apps/sw/oracle/mw/wls1033/oracle_common
-Djrf.version=11.1.1 -Djrockit.optfile=/apps/sw/oracle/mw/wls1033/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt
-Doracle.domain.config.dir=/apps/services/mw/VGR-WLS-STG/config/fmwconfig
-Doracle.server.config.dir=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/servers/VWS-App-4
-Doracle.security.jps.config=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/jps-config.xml
-Djava.protocol.handler.pkgs=oracle.mds.net.protocol
-Digf.arisidbeans.carmlloc=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/carml
-Digf.arisidstack.home=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/arisidprovider
-Dweblogic.alternateTypesDirectory=/apps/sw/oracle/mw/wls1033/oracle_common/modules/oracle.ossoiap_11.1.1,/apps/sw/oracle/mw/wls1033/oracle_common/modules/oracle.oamprovider_11.1.1
-Dweblogic.jdbc.remoteEnabled=false -Dem.oracle.home=/apps/sw/oracle/mw/wls1033/oracle_common -Djava.awt.headless=true -Dweblogic.management.discover=false
-Dweblogic.management.server=http://LAVDRXS011R:7001 -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole=
-Dweblogic.ext.dirs=/apps/sw/oracle/mw/wls1033/patch_wls1033/profiles/default/sysext_manifest_classpath:/apps/sw/oracle/mw/wls1033/patch_ocp353/profiles/default/sysext_manifest_classpath
-Xverbose:memory -Xverboselog:/logs/servers/VWS-App-4/verbose-gc.log -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder
-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.SSL.protocolVersion=SSL3 -Dlog4j.debug -Dtangosol.coherence.mode=prod
-Djavax.xml.soap.MessageFactory=weblogic.xml.saaj.MessageFactoryImpl -DnotificationBroker.instanceMode=PASSIVE -Xverify:none -Dwles.scm.port=
-Dwles.arme.port=9101 -Dwles.config.signer=lavdrxs011r.atlis.aiat.com
-Dlog4j.configuration=file:/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance/config/log4j.properties
-Dlog4j.ignoreTCL -Dwles.ssl.passwordFile=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/password.xml
-Dwles.ssl.passwordKeyFile=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/password.key
-Dwles.ssl.identityKeyStore=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/identity.jceks
-Dwles.ssl.identityKeyAlias=wles-ssm -Dwles.ssl.identityKeyPasswordAlias=wles-ssm
-Dwles.ssl.trustedCAKeyStore=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/trust.jks
-Dwles.ssl.trustedPeerKeyStore=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/peer.jks
-Djava.io.tmpdir=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance/work/jar_temp
-Dwles.ssmws.instance.home=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance
-Darme.configuration=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance/config/WLESarme.properties
-Dales.blm.home=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance -Dweblogic.security.enableNegotiate=true
-Dweblogic.alternateTypesDirectory=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/lib/providers/wls/v9
-Dwles.scm.useSSL=true
-XX:PermSize=128m
-XX:MaxPermSize=256m weblogic.Server
[WARN ] Use of -Djrockit.optfile is deprecated and discouraged.
 
Greenhorn
Posts: 16
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The issue is obviously with the database and not WebLogic. The thread managing the oracle connection is waiting to receive data from a query. After 600 seconds it is marked as STUCK. Try checking the AWR report around the hour when threads start getting stuck. That should give a clue as to which query is causing the problem.
 
M Paul Baker
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thank you for your suggestion. We have "requested" an AWR report, I'm hopeful that it will provide some clues to what is happening.

We are working on creating several scenarios that would/could duplicate this situation.

What are some conditions that would cause a thread to be marked "STUCK" at SocketNativeIO.readBytesPinned while reading a DB response.
  • DB Bounce? Y/N
  • DB Timeout? Y/N
  • DB Exceptions? Y/N
  • Socket Connection Issues btwn WLS and DB? Y/N
  • A long running query (i.e. no index)? Y/N


  •  
    With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
    reply
      Bookmark Topic Watch Topic
    • New Topic