I am using JBoss 5.1 JMS Queues. After processing certain no. of messages, say after a 50,000 messages, the messages in the queue are not consumed. Then the delivering count stays the same and the messaging count goes on increasing. I get around 30 to 40 messages per second. I searched it on google and did the following
1) close the queueSession and queueConnection.
2) changed to mysql db from hsqldb.
3) increased the transaction timeout.
Even after doing all these, my messages are still getting stuck in the queue.
The application is like this.
A c++ application sends the messages over socket. The java converter application receives the messages and puts them in a blocking queue. And some 20 threads access the blocking queue (receivedQ) and puts them in RHInputQueue (Queue). The BL application uses an MDB (ProcessRequest) to consume those messages from RHInputQueue, does some manipulations on the message and puts that message in the RHOutputQueue (Queue). The converter application then uses another MDB to consume those messages in the RHOutputQueue and puts them in another blocking queue. And one thread accesses that blocking queue (processedQ) and sends them over the socket to the c++ application.
Here [Attached] is the thread dump from the jboss jmx console after it stopped processing the messages from the RHInputQueue
Adding a 4000 line thread dump to the text of your post is a sure way to have people ignore your post entirely. I have, instead, turned it into an attachment.
Having said that, I am not willing to dig through those 4000 lines, even attached. You need to pare the thread dump down to the few threads that are not simply waiting for user input. Threads waiting for user input usually have a short stack trace (less than a dozen lines), and usually have a wait of some kind as the first line. That leaves the thread dumps that are lengthy (many lines in the stack trace). You need to filter out the ones that are actually doing work, and the best way to do that is to take 2 or 3 thread dump about 10 to 15 seconds apart. Look for threads that seem to be doing the same thing in each dump.
Having said that, unless you find that the message readers are hung (possibly in infinite loops), then I doubt that the thread dump will reveal the cause of the hanging messages.
Sorry for posting such a huge message and thanks for the reply.
I have checked the thread dump and removed the threads that are in RUNNABLE, WAITING and TIMED_WAITING states.
I found that the thread that calls the onMessage is blocked and it is locked on an address. What does it imply and how should I proceed further?
But did you take multiple thread dumps spaced 10-15 seconds apart to see if those threads changed? Some of the threads appear to be caught in the middle of doing work, others are waiting on a lock. Are they still doing the same things in 10 seconds?
Also, it looks like your onMessage method also sends a message. Hopefully it is not sending the message back to the same queue...
The thread dump that I have checked, is for the last time (Oct 2nd) that the messages were not consumed from the queue (RHInputQueue). It is actually a production system. Till now, all the messages are consumed from the queue (RHInputQueue) properly. When it stops consuming, I will then take some 3 to 4 thread dumps with a gap of 10 to 15 seconds and analyze that.
And yes, the onMessage method processes the incoming message from RHInputQueue and sends it to another queue (RHOutputQueue).