We're getting several STUCK threads like the one below. These never clear up, the managed servers must be bounced to remove them. The STUCK threads in question are processing a prepared statement and hang indefinately on the socket read (readBytesPinned). We are tracking down the exact query involved. Enabling traces is a "process" in the environment where this happens.
How can I verify that the DB is actively processing this request?
Would the Oracle DB Alert Logs show that there is a/are long running requests?
Getting a DB Alert Log is a challenge, it has to be requested; should I ask for any particular options that would help identify long running connections or connections that have timed out/dropped?
Would a
thread remain in the STUCK state even if the connection dropped? I would like to know what the thread is doing at method readBytesPinned. I'm Stuck
Has anyone tracked down these kind of thread issues? If so, will you please share any lessons learned?
This thread stack taken from a thread dump. We have taken several dumps and the thread never appears to have any progress.
Advisories: Stuck Thread DB Execute Socket Read Web Application Request
"[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x68 tid=13619 prio=1 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at oracle/net/nt/MetricsEnabledInputStream.read(TcpNTAdapter.java:564)[optimized]
at oracle/net/ns/Packet.receive(Packet.java:295)[inlined]
at oracle/net/ns/DataPacket.receive(DataPacket.java:94)[optimized]
at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:176)[optimized]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:121)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:96)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:81)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)[inlined]
at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1087)[optimized]
at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)[inlined]
at oracle/jdbc/driver/T4C8Oall.receive(T4C8Oall.java:625)[optimized]
at oracle/jdbc/driver/T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)[inlined]
at oracle/jdbc/driver/T4CPreparedStatement.fetch(T4CPreparedStatement.java:1121)[optimized]
at oracle/jdbc/driver/OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:571)[inlined]
at oracle/jdbc/driver/OracleResultSetImpl.next(OracleResultSetImpl.java:445)[optimized]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x14e5531a0[recursive]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x14e5531a0[thin lock]
at weblogic/jdbc/wrapper/ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next()Z(Unknown Source)[optimized]
at org/hibernate/loader/Loader.doQuery(Loader.java:697)[inlined]
at org/hibernate/loader/Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)[inlined]
at org/hibernate/loader/Loader.doList(Loader.java:2213)[inlined]
at org/hibernate/loader/Loader.listIgnoreQueryCache(Loader.java:2104)[optimized]
at org/hibernate/loader/Loader.list(Loader.java:2099)
at org/hibernate/loader/hql/QueryLoader.list(QueryLoader.java:378)
at org/hibernate/hql/ast/QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
at org/hibernate/engine/query/HQLQueryPlan.performList(HQLQueryPlan.java:172)
at org/hibernate/impl/SessionImpl.list(SessionImpl.java:1121)
at org/hibernate/impl/QueryImpl.list(QueryImpl.java:79)
at org/hibernate/ejb/QueryImpl.getResultList(QueryImpl.java:65)
at org/springframework/orm/jpa/JpaTemplate$11.doInJpa(JpaTemplate.java:351)
at org/springframework/orm/jpa/JpaTemplate.execute(JpaTemplate.java:184)
at org/springframework/orm/jpa/JpaTemplate.executeFind(JpaTemplate.java:151)
at org/springframework/orm/jpa/JpaTemplate.findByNamedQuery(JpaTemplate.java:343)
at aiat/csp/esb2/persistence/PersistenceManagerImpl.findByNamedQuery(PersistenceManagerImpl.java:224)
at com/aiat/odlp/commonutilities/referencedata/dao/ReferenceDataManagerDAO.getAirlineList(ReferenceDataManagerDAO.java:725)[optimized]
at com/aiat/odlp/commonutilities/referencedata/impl/ReferenceDataManagerImpl.listAirlineCodes(ReferenceDataManagerImpl.java:177)[optimized]
at sun/reflect/GeneratedMethodAccessor7000.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at org/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org/springframework/aop/framework/ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org/springframework/transaction/interceptor/TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org/springframework/aop/interceptor/ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org/springframework/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy247.listAirlineCodes(Lcom/aiat/odlp/commonutilities/referencedata/interfaces/common/ListAirlinesQueryDTO;)Lcom/aiat/odlp/commonutilities/referencedata/interfaces/common/AirlineListDTO;(Unknown Source)
at com/aiat/odlp/commonutilities/referencedata/endpoint/ReferenceDataRepositoryListEndPoint.listAirlines(ReferenceDataRepositoryListEndPoint.java:78)
at sun/reflect/GeneratedMethodAccessor6999.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at org/springframework/ws/server/endpoint/MethodEndpoint.invoke(MethodEndpoint.java:115)
at org/springframework/ws/server/endpoint/adapter/MarshallingMethodEndpointAdapter.invokeInternal(MarshallingMethodEndpointAdapter.java:135)
at org/springframework/ws/server/endpoint/adapter/AbstractMethodEndpointAdapter.invoke(AbstractMethodEndpointAdapter.java:58)
at org/springframework/ws/server/MessageDispatcher.dispatch(MessageDispatcher.java:236)
at org/springframework/ws/server/MessageDispatcher.receive(MessageDispatcher.java:177)
at com/aiat/odlp/server/endpoint/jaxws/JaxWsRequestHandler.handleRequest(JaxWsRequestHandler.java:96)
at com/aiat/odlp/server/endpoint/jaxws/VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:75)
at com/aiat/odlp/server/endpoint/jaxws/VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:38)
at sun/reflect/GeneratedMethodAccessor958.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at weblogic/wsee/jaxws/WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:92)
at weblogic/wsee/jaxws/WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:74)
at com/sun/xml/ws/api/server/Invoker.invokeProvider(Invoker.java:108)
at com/sun/xml/ws/server/InvokerTube$2.invokeProvider(InvokerTube.java:161)
at com/sun/xml/ws/server/provider/SyncProviderInvokerTube.processRequest(SyncProviderInvokerTube.java:78)
at com/sun/xml/ws/api/pipe/Fiber.__doRun(Fiber.java:604)[optimized]
at com/sun/xml/ws/api/pipe/Fiber._doRun(Fiber.java:563)
at com/sun/xml/ws/api/pipe/Fiber.doRun(Fiber.java:548)
at com/sun/xml/ws/api/pipe/Fiber.runSync(Fiber.java:445)
^-- Holding lock: com/sun/xml/ws/api/pipe/Fiber@0x14e5349d0[thin lock]
at com/sun/xml/ws/server/WSEndpointImpl$2.process(WSEndpointImpl.java:373)
at com/sun/xml/ws/transport/http/HttpAdapter$HttpToolkit.handle(HttpAdapter.java:524)
at com/sun/xml/ws/transport/http/HttpAdapter.handle(HttpAdapter.java:255)
at com/sun/xml/ws/transport/http/servlet/ServletAdapter.handle(ServletAdapter.java:140)
at weblogic/wsee/jaxws/WLSServletAdapter.handle(WLSServletAdapter.java:208)
at weblogic/wsee/jaxws/HttpServletAdapter$AuthorizedInvoke.run(HttpServletAdapter.java:310)
at weblogic/wsee/jaxws/HttpServletAdapter.post(HttpServletAdapter.java:223)
at weblogic/wsee/jaxws/JAXWSServlet.doPost(JAXWSServlet.java:124)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:727)
at weblogic/wsee/jaxws/JAXWSServlet.service(JAXWSServlet.java:79)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)
at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/dms/wls/DMSServletFilter.doFilter(DMSServletFilter.java:330)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.doIt(Lweblogic/servlet/internal/ServletStubImpl;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)Ljava/lang/Object;(Unknown Source)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;(Unknown Source)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[inlined]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[optimized]
at weblogic/servlet/internal/WebAppServletContext.securedExecute(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V(Unknown Source)
at weblogic/servlet/internal/WebAppServletContext.execute(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V(Unknown Source)
at weblogic/servlet/internal/ServletRequestImpl.run()V(Unknown Source)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)[optimized]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
The managed server logs are showing "similar" issues but to make things interesting the line numbers shown in the stack traces are different. I am not sure I am comparing apples to apples. Here is an example of a similar thread excpetion from the Weblogic logs.
<Mar 21, 2014 12:59:35 PM GMT+00:00> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "605" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@2b80c9fa[
POST /rd-1.1.2/ReferenceData-1.1?wsdl HTTP/1.1
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://aiat/odlp/services/commonutilities/ReferenceDataRepository/v.0.0.0-28/action/ListAirlines/"
Transfer-Encoding: chunked
User-Agent: Java1.6.0_33
Accept: text/html, image/gif, image/jpeg, */*; q=.2
Connection: Keep-Alive
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-15 "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
oracle.net.ns.Packet.receive(Packet.java:249)
oracle.net.ns.DataPacket.receive(DataPacket.java:94)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1091)
oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:480)
oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:347)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@2d660b1e[thin lock]
^-- Holding lock: oracle.jdbc.driver.T4CConnection@2d660b1e[thin lock]
weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
org.hibernate.loader.Loader.doQuery(Loader.java:665)
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:232)
org.hibernate.loader.Loader.doList(Loader.java:2207)
org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
org.hibernate.loader.Loader.list(Loader.java:2092)
org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:377)
org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:321)
org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:149)
org.hibernate.impl.SessionImpl.list(SessionImpl.java:1110)
org.hibernate.impl.QueryImpl.list(QueryImpl.java:75)
org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:65)
org.springframework.orm.jpa.JpaTemplate$11.doInJpa(JpaTemplate.java:345)
org.springframework.orm.jpa.JpaTemplate.execute(JpaTemplate.java:169)
org.springframework.orm.jpa.JpaTemplate.executeFind(JpaTemplate.java:151)
org.springframework.orm.jpa.JpaTemplate.findByNamedQuery(JpaTemplate.java:343)
aiat.csp.esb2.persistence.PersistenceManagerImpl.findByNamedQuery(PersistenceManagerImpl.java:224)
com.aiat.odlp.commonutilities.referencedata.dao.ReferenceDataManagerDAO.getAirlineList(ReferenceDataManagerDAO.java:674)
com.aiat.odlp.commonutilities.referencedata.impl.ReferenceDataManagerImpl.listAirlineCodes(ReferenceDataManagerImpl.java:169)
sun.reflect.GeneratedMethodAccessor7000.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:306)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:92)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy247.listAirlineCodes(Unknown Source)
com.aiat.odlp.commonutilities.referencedata.endpoint.ReferenceDataRepositoryListEndPoint.listAirlines(ReferenceDataRepositoryListEndPoint.java:75)
sun.reflect.GeneratedMethodAccessor6999.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.springframework.ws.server.endpoint.MethodEndpoint.invoke(MethodEndpoint.java:109)
org.springframework.ws.server.endpoint.adapter.MarshallingMethodEndpointAdapter.invokeInternal(MarshallingMethodEndpointAdapter.java:133)
org.springframework.ws.server.endpoint.adapter.AbstractMethodEndpointAdapter.invoke(AbstractMethodEndpointAdapter.java:58)
org.springframework.ws.server.MessageDispatcher.dispatch(MessageDispatcher.java:211)
org.springframework.ws.server.MessageDispatcher.receive(MessageDispatcher.java:167)
com.aiat.odlp.server.endpoint.jaxws.JaxWsRequestHandler.handleRequest(JaxWsRequestHandler.java:90)
com.aiat.odlp.server.endpoint.jaxws.VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:66)
com.aiat.odlp.server.endpoint.jaxws.VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:38)
sun.reflect.GeneratedMethodAccessor958.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:92)
weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:73)
com.sun.xml.ws.api.server.Invoker.invokeProvider(Invoker.java:108)
com.sun.xml.ws.server.InvokerTube$2.invokeProvider(InvokerTube.java:159)
com.sun.xml.ws.server.provider.SyncProviderInvokerTube.processRequest(SyncProviderInvokerTube.java:70)
com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:580)
com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:553)
com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:535)
com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:433)
^-- Holding lock: com.sun.xml.ws.api.pipe.Fiber@2d35f1bf[thin lock]
com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:352)
com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:508)
com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:230)
com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:140)
weblogic.wsee.jaxws.WLSServletAdapter.handle(WLSServletAdapter.java:193)
weblogic.wsee.jaxws.HttpServletAdapter$AuthorizedInvoke.run(HttpServletAdapter.java:310)
weblogic.wsee.jaxws.HttpServletAdapter.post(HttpServletAdapter.java:187)
weblogic.wsee.jaxws.JAXWSServlet.doPost(JAXWSServlet.java:124)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
weblogic.wsee.jaxws.JAXWSServlet.service(JAXWSServlet.java:76)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(Unknown Source)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(Unknown Source)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
weblogic.servlet.internal.WebAppServletContext.securedExecute(Unknown Source)
weblogic.servlet.internal.WebAppServletContext.execute(Unknown Source)
weblogic.servlet.internal.ServletRequestImpl.run(Unknown Source)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}
>
Any help/guidance would be really appreciated.
Start of Managed Server log with envt details:
starting weblogic with
Java version:
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
Oracle JRockit(R) (build R28.2.4-14-151097-1.6.0_33-20120618-1634-linux-x86_64, compiled mode)
Starting WLS with line:
/apps/sw/oracle/mw/jrockit-x64/bin/java -jrockit -Xms4096m -Xmx4096m -Dweblogic.Name=VWS-App-4
-Djava.security.policy=/apps/sw/oracle/mw/wls1033/wlserver_10.3/server/lib/weblogic.policy
-Dweblogic.system.BootIdentityFile=/apps/services/mw/VGR-WLS-STG/servers/VWS-App-4/data/nodemanager/boot.properties
-Dweblogic.nodemanager.ServiceEnabled=true
-Dweblogic.security.SSL.ignoreHostnameVerification=true
-Dweblogic.ReverseDNSAllowed=false
-Dweblogic.Stdout=/logs/servers/VWS-App-4/VWS-App-4.out
-Xverify:none -da -Dplatform.home=/apps/sw/oracle/mw/wls1033/wlserver_10.3
-Dwls.home=/apps/sw/oracle/mw/wls1033/wlserver_10.3/server
-Dweblogic.home=/apps/sw/oracle/mw/wls1033/wlserver_10.3/server
-Ddomain.home=/apps/services/mw/VGR-WLS-STG
-Dcommon.components.home=/apps/sw/oracle/mw/wls1033/oracle_common
-Djrf.version=11.1.1 -Djrockit.optfile=/apps/sw/oracle/mw/wls1033/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt
-Doracle.domain.config.dir=/apps/services/mw/VGR-WLS-STG/config/fmwconfig
-Doracle.server.config.dir=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/servers/VWS-App-4
-Doracle.security.jps.config=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/jps-config.xml
-Djava.protocol.handler.pkgs=oracle.mds.net.protocol
-Digf.arisidbeans.carmlloc=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/carml
-Digf.arisidstack.home=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/arisidprovider
-Dweblogic.alternateTypesDirectory=/apps/sw/oracle/mw/wls1033/oracle_common/modules/oracle.ossoiap_11.1.1,/apps/sw/oracle/mw/wls1033/oracle_common/modules/oracle.oamprovider_11.1.1
-Dweblogic.jdbc.remoteEnabled=false -Dem.oracle.home=/apps/sw/oracle/mw/wls1033/oracle_common -Djava.awt.headless=true -Dweblogic.management.discover=false
-Dweblogic.management.server=http://LAVDRXS011R:7001 -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole=
-Dweblogic.ext.dirs=/apps/sw/oracle/mw/wls1033/patch_wls1033/profiles/default/sysext_manifest_classpath:/apps/sw/oracle/mw/wls1033/patch_ocp353/profiles/default/sysext_manifest_classpath
-Xverbose:memory -Xverboselog:/logs/servers/VWS-App-4/verbose-gc.log -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder
-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.SSL.protocolVersion=SSL3 -Dlog4j.debug -Dtangosol.coherence.mode=prod
-Djavax.xml.soap.MessageFactory=weblogic.xml.saaj.MessageFactoryImpl -DnotificationBroker.instanceMode=PASSIVE -Xverify:none -Dwles.scm.port=
-Dwles.arme.port=9101 -Dwles.config.signer=lavdrxs011r.atlis.aiat.com
-Dlog4j.configuration=file:/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance/config/log4j.properties
-Dlog4j.ignoreTCL -Dwles.ssl.passwordFile=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/password.xml
-Dwles.ssl.passwordKeyFile=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/password.key
-Dwles.ssl.identityKeyStore=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/identity.jceks
-Dwles.ssl.identityKeyAlias=wles-ssm -Dwles.ssl.identityKeyPasswordAlias=wles-ssm
-Dwles.ssl.trustedCAKeyStore=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/trust.jks
-Dwles.ssl.trustedPeerKeyStore=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/peer.jks
-Djava.io.tmpdir=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance/work/jar_temp
-Dwles.ssmws.instance.home=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance
-Darme.configuration=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance/config/WLESarme.properties
-Dales.blm.home=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance -Dweblogic.security.enableNegotiate=true
-Dweblogic.alternateTypesDirectory=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/lib/providers/wls/v9
-Dwles.scm.useSSL=true
-XX:PermSize=128m
-XX:MaxPermSize=256m weblogic.Server
[WARN ] Use of -Djrockit.optfile is deprecated and discouraged.