Application deployed in Weblogic server is too slow

[shincy sunny]

My application deployed in the Weblogic instance is dead slow sometimes.
I'm using ADF 12c for application development
Weblogic Version: 12.2.1 in windows server
Database : Oracle 11g
Jdk version : 1.8-65
I don't know how to resolve this issue and couldn't yet understood the root cause for this.
So far, I managed to gather the following details.
• At this specific time, Managed server log file shows the following error related to Stuck Threads:
   <Error> <WebLogicServer> <ManagedServer_1> <[STANDBY] ExecuteThread: '124' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c76372a-dbe4-4308-a06c-b9944a2d5d3a-0001b9dc> <1472636531716> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000337> <[STUCK] ExecuteThread: '117' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "630" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@714bd487[GET /abcV3/adwwls/wls.ads]
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
   oracle.adfinternal.view.faces.activedata.AdsProcessor.doProcess(AdsProcessor.java:184)
   oracle.adfinternal.view.faces.activedata.AdsServlet.doGet(AdsServlet.java:112)
   javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
   javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
   weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
   weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
   weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
   weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
   weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
   oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:166)
   weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
   weblogic.servlet.utils.FastSwapFilter.doFilter(FastSwapFilter.java:64)
   weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
   oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:141)
   java.security.AccessController.doPrivileged(Native Method)
   oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
   oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:649)
   oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:124)
   oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:232)
   oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:94)
   weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
   oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:224)
   weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
   weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:32)
   weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
   weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3654)
   weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3620)
   weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:326)
   weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:196)
   weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
   weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
   weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2423)
   weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2280)
   weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2258)
   weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1626)
   weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1586)
   weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:270)
   weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)
   weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)
   weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)
   weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
   weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:617)
   weblogic.work.ExecuteThread.execute(ExecuteThread.java:397)
   weblogic.work.ExecuteThread.run(ExecuteThread.java:346)
   >

Initially, Max Stuck Thread Time was 600 seconds. But, increasing the stuck threads time did not resolve my issue.

• The CPU usage of the server where Database is running :19% (Processes-19%, Services - 15%)
• The CPU usage of the server where application is running :97%
• PS_TXN table shows the count as : 7608
• Heap usage : 2228.21
• Active session : 29

Can anyone please help me in this regard? Any suggestions are most welcome.
Thanks in advance.

[Dave Tolls]

What is it doing in this GET?
GET /abcV3/adwwls/wls.ads

Can you see what is happening in there for that Thread?

[shincy sunny]

Hi,
As per my knowledge .ads is file associated with weblogic server. It doesn't have any connection with our application
But, we have a notification feature in our application, its working in real time. We suspect that this feature is the culprit for this issue, but we are not sure about this.
Is there any way out to find the actual reason without disabling this notification feature?

Thanks in advance.