Threading/Queue Issue With Stuck Thread

[Jeremy Munnerlyn]

Hi,

First post. I am going to try and provide as much detail as possible.

Scenario: Please know that I did not design this and that I am only supporting/debugging. I have a group of File Watchers. Each of these is an EAR file application deployed on a Weblogic 11g Server instance. Essentially each file watcher polls a different directory every minute on a Notification Listener design. When a file is recognized, a message is sent to a Weblogic JMS Distributed Queue where a common (clustered on two nodes) EAR file application consisting of MDBs are listening to the queue. Below are the ejb configurations for the MDB pool:

weblogic-ejb-jar.xml:
<weblogic-ejb-jar> <description><![CDATA[Generated by XDoclet]]></description> <weblogic-enterprise-bean> <ejb-name>TimeQueueWatch</ejb-name> <message-driven-descriptor> <pool> <max-beans-in-free-pool>100</max-beans-in-free-pool> <initial-beans-in-free-pool>100</initial-beans-in-free-pool> </pool> <destination-jndi-name>DACServices40.TimerQueue</destination-jndi-name> </message-driven-descriptor> <reference-descriptor> </reference-descriptor> <dispatch-policy>DACServicesClusterWorkManager</dispatch-policy> </weblogic-enterprise-bean>

ejb-jar.xml:
<description></description> <display-name>TimeQueueWatch</display-name> <enterprise-beans> <message-driven > <description><![CDATA[<!-- begin-xdoclet-definition -->]]></description> <ejb-name>TimeQueueWatch</ejb-name> <ejb-class>com.jpmorgan.gcrm.dacservices.timequeuewatcher.TimeQueueWatcherMdb</ejb-class> <transaction-type>Container</transaction-type> <acknowledge-mode>Auto-acknowledge</acknowledge-mode> <message-driven-destination> <destination-type>javax.jms.Queue</destination-type> </message-driven-destination> </message-driven>

The workmanager DACServicesClusterWorkManager implements a MaxThreadsConstraint of 200.

Normally everything processes as expected and MDB threads are freed as the process completes. However, sometimes our Operate team can manually lock a thread (in the MDBs) if they do not want the process to proceed. They perform the locking by doing an update to a database table. If the thread needs to lock itself based off of the database entry, it enters a thread.sleep() state (I understand that this is a bad implementation) and checks every minute to see if the lock has been freed in the database.

Problem: This has only happened once to date, but it cannot happen as these are critical processes that need to be performed without delay. Here is what happened:
1. File Watcher A Thread for File A recognizes a file – Thread 1
2. File Watcher A sends message to Weblogic JMS Queue – Thread 1
3. MDB Thread for File A Kicks Off – Thread 2
4. MDB Thread for File A Encounters Lock and Sleeps – Thread 2
....Some time later
5. File Watcher B Thread for File B recognizes a file – Thread 3
6. File Watcher B sends message to Weblogic JMS Queue – Thread 3
....Some time later
7. Everyone is in an uproar because File B hasn't been processed yet
....Some time later
8. Lock is released for MDB Thread for File A – Thread 2
9. MDB Thread for File A completes processing – Thread 2
10. Instantly MDB Thread for File B kicks off and completes processing – Thread 2

Note there are 100 beans available and there was never a situation where maximum bean usage was even over 5. Also, several other files processed during this time, so other beans were being used.

So my question is why didn't the message for File B trigger a thread that was not in use instead of waiting on the thread that was being locked by File A? Why didn't it kick off a thread that was available like the other files that processed during this time? The thread numbers are listed in the log and appear just like I have outlined above.

I am unable to provide an explanation for this, so any help would be greatly appreciated.

Thanks in Advance,
Jeremy