• Post Reply Bookmark Topic Watch Topic
  • New Topic

Need help with stuck threads on weblogic  RSS feed

 
Sheldon Dsouza
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

We are seeing stuck threads on Weblogic in Production environment.
It cannot be replicated in Development environment.

We have 1 Web Service (API) that encounters stuck threads more often.
Need help in replicating the issue in Development and a fix for it.

It looks like there is some issue with CFX / Spring framework we are using.
CFX version: 3.1.4
Spring version: 4.2.2.RELEASE
Sprig Security: 4.0.3.RELEASE

Below is a snippet from thread dump. Please let me know of any inputs on how to fix the issue with lock on the Phase Interceptor

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
at com.minervanetworks.xtv.stb.service.impl.ClientContentManagerImpl$$EnhancerBySpringCGLIB$$115ca042.edgeBrowse(<generated>)
at sun.reflect.GeneratedMethodAccessor352.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:200)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
- locked <0x000000013132eca0> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:251)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)

Thanks,
Sheldon
 
Rob Spoor
Sheriff
Posts: 21050
85
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Welcome to the Ranch!

WebLogic only regards a thread as stuck after 10 minutes. Do you have such long running tasks? If so, how do you start them? Because I know that WebLogic also has a notion of long running tasks, you just need to do some extra work.
 
Sheldon Dsouza
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for the reply Rob.

So, we have noticed threads being stuck for more than 10 minutes (sometimes even for days) on Weblogic.

We use Web Services, and the threads get stuck when one of the API in the Web Services is called.
The API calls use the Java library: org.apache.http.impl.client.CloseableHttpClient for calling another Web Service (other Web Service is on a different server).
It uses Oracle as the database.
Our project uses CFX & Spring framework.
Please Note:
Only some threads of the API get stuck (sometimes as few as 2 or 3 or sometimes 100). And the API can be called multiple times in a minute.
We also have ActiveMQ message listeners which are supposed to listen to messages continually (not part of the API). So, those threads when stuck might not really be stuck.

It is my suspicion that there is an issue with any of the following: Weblogic, CloseableHttpClient, Oracle, CFX, Spring.
We are trying to replicate the issue below (lock on Phase Interceptor) in our Development environment.

Based on the information provided, can you provide any feedback / inputs, especially if there are some know issues?
If you need more information, let me know so that i can provide the same.
 
Rob Spoor
Sheriff
Posts: 21050
85
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It's very difficult to find out what exactly is causing the issue. It could be the CloseableHttpClient, if you have a pooled connection manager that runs out of connections. That can occur if you don't return the connection to the pool after you're done with it. That's actually a quite common error, I've seen it happen several times. Especially if there are errors while handling the response, or if you don't close the response, you can run into this issue.

What we often do to prevent this is the following:

That last line ensures that the entity content is fully consumed and the content stream is closed, so the connection can be reused. If the latter doesn't occur then the connection will never be released.
Don't close the response itself, because that will close the connection instead of simply returning it to the pool.
 
Sheldon Dsouza
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for the quick response Rob.

I did a load test on HttpClient and found that it does infact gets stuck / takes long to load.
We were using HttpBuilder without setting a Timeout / Pooling.
After changing the code to use the 1st snippet below, we dont see HttpClient getting stuck. And Pool rejects requests after a timeout.
We will also try the suggested change for using EntityUtils.

However, let me know if you have any other suggestions as well after reviewing the code snippet below.

I have 3 queries
1> Do we need to explicitly set the socket timeout for PoolingHttpClientConnectionManager as well apart from RequestConfig?
2> Is there a situation where the Pool gets automatically closed when there is a high load and if we need to re-create the Pool multiple times.
3> Do we need to created a separate monitoring thread to monitor the Pool and explicitly close stale or expired connections.

Below are Code Snippets
1> Code Snippet for Initializing HttpBuilder (this change of Pooling was not there originally)

2> Code Snippet for calling API (Original code was using: client = HttpClientBuilder.create().build() )
 
Rob Spoor
Sheriff
Posts: 21050
85
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Most of your code looks just fine. I have just a few comments on it:
1) Don't throw Exception, but use some sub type instead. Using Exception and RuntimeException directly is considered bad style.
2) If the status code is not 200 you still need to close the entity like I suggested, otherwise you run into the issue I described previously.

Now to answer your questions:
1) No, you don't need to set the socket timeout on the manager. Those of a RequestConfig will always take precedence.
2) The connection manager should never be closed automatically. Its connections might, but the connection manager should take care of that.
3) You should call closeExpiredConnections and closeIdleConnections on a regular interval (e.g. using a timer). If you don't, some requests may fail because the connection has been closed from the other side. The connection manager will recover for future connections, but that one request will fail. There's no need for further monitoring.


On a side note, perhaps you could use code tags in the future. I've added them this time, to make your code more readable.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!