Message lost or are not pulled by MDB

[Raymond Ong]

Hi,

I have a pool of MDBs listening to a queue. I bombard the queue with 1000 messages concurrently (I made an app that creates 1000 threads that each sends a JMS message to the queue all at the same time approximately). The messages in the queue are persisted in a MySQL database.

As the onMessage() method of the MDB is triggered, I log that event to a file. I'm wondering why sometimes, I dont get 1000 logs which I expect and instead get logs less than 1000.

There are times when a number of messages stay stuck on the queue even if there are active MDBs listening. I can say the MDBs are listening because when I send a JMS message, it gets processed, yet the previously stuck messages remain stucked.

I suspect It has something to do with the memory. I tried to cut down the number of threads into half, so that's 500 concurrent messages sent to the queue and out of the 12 times I tested, only once did I not receive all 500. I lost about 48 messages, which neither was logged (pulled by MDB) nor exists in the queue.

Another reason I suspect the memory is because I stripped down my onMessage() method. I simply logged into a file. I haven't lost any message even If there were 1000 threads that sent the jms messages concurrently.

Here's my ejb-jar.xml

 <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE ejb-jar PUBLIC "-//Sun Microsystems, Inc.//DTD Enterprise JavaBeans 2.0//EN" "http://java.sun.com/dtd/ejb-jar_2_0.dtd">
<ejb-jar>
<description>RequestProcessor</description>
<display-name>RequestProcessor</display-name>
<enterprise-beans>
<message-driven>
<ejb-name>RequestProcessor</ejb-name>
<ejb-class>com.mobilegroupinc.gospel.RequestProcessor</ejb-class>
<transaction-type>Container</transaction-type>
<message-selector></message-selector>
<message-driven-destination>
<destination-type>javax.jms.Queue</destination-type>
</message-driven-destination>
</message-driven>
 </enterprise-beans>    
 <assembly-descriptor>
<container-transaction>
<method>
<ejb-name>RequestProcessor</ejb-name>
<method-name>onMessage</method-name>
<method-params>
<method-param>javax.jms.Message</method-param>
</method-params>
</method>
<trans-attribute>Required</trans-attribute>
</container-transaction>
</assembly-descriptor>  
</ejb-jar> 

Here's my jboss.xml

  
<?xml version="1.0"?>
<jboss>
 
<invoker-proxy-bindings>
<invoker-proxy-binding>
<name>mgi-message-driven-bean</name>
<invoker-mbean>default</invoker-mbean>
<proxy-factory>org.jboss.ejb.plugins.jms.JMSContainerInvoker</proxy-factory>
<proxy-factory-config>
<JMSProviderAdapterJNDI>DefaultJMSProvider</JMSProviderAdapterJNDI>
<ServerSessionPoolFactoryJNDI>StdJMSPool</ServerSessionPoolFactoryJNDI>
<MinimumSize>1</MinimumSize>
<MaximumSize>20</MaximumSize>
<KeepAliveMillis>30000</KeepAliveMillis>
<MaxMessages>1</MaxMessages>
<MDBConfig>
<ReconnectIntervalSec>10</ReconnectIntervalSec>
<DLQConfig>
<DestinationQueue>queue/DLQ</DestinationQueue>
<MaxTimesRedelivered>10</MaxTimesRedelivered>
<TimeToLive>0</TimeToLive>
</DLQConfig>
</MDBConfig>
</proxy-factory-config>
</invoker-proxy-binding>
</invoker-proxy-bindings>
 
<container-configuration>
<container-name>MGI Message Driven Bean</container-name>
<call-logging>false</call-logging>
<container-interceptors>
<interceptor>org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.LogInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.RunAsSecurityInterceptor</interceptor>
<!-- CMT -->
<interceptor transaction="Container">org.jboss.ejb.plugins.TxInterceptorCMT</interceptor>
<interceptor transaction="Container">org.jboss.ejb.plugins.CallValidationInterceptor</interceptor>
<interceptor transaction="Container">org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor</interceptor>
<!-- BMT -->
<interceptor transaction="Bean">org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor</interceptor>
<interceptor transaction="Bean">org.jboss.ejb.plugins.MessageDrivenTxInterceptorBMT</interceptor>
<interceptor transaction="Bean">org.jboss.ejb.plugins.CallValidationInterceptor</interceptor>
<interceptor>org.jboss.resource.connectionmanager.CachedConnectionInterceptor</interceptor>
</container-interceptors>
<instance-pool>org.jboss.ejb.plugins.MessageDrivenInstancePool</instance-pool>
<instance-cache></instance-cache>
<persistence-manager></persistence-manager>
<container-pool-conf>
<MinimumSize>1</MinimumSize>        
<MaximumSize>20</MaximumSize>
<MaxMessages>1</MaxMessages>
</container-pool-conf>
</container-configuration>
 
<enterprise-beans>
<message-driven>
<ejb-name>RequestProcessor</ejb-name>
<configuration-bean>MGI Message Driven Bean</configuration-bean>
<destination-jndi-name>queue/gospelQueue</destination-jndi-name>
<invoker-bindings>
<invoker>      
<invoker-proxy-binding-name>mgi-message-driven-bean</invoker-proxy-binding-name>
</invoker>
</invoker-bindings>
<mdb-user>mdb</mdb-user>
<mdb-passwd>mgimdb</mdb-passwd>
<!--<resource-ref>
<res-ref-name>java:/JVMILXAConnectionFactory</res-ref-name>
<res-type>javax.jms.QueueConnectionFactory</res-type>
<jndi-name>java:/JVMILXAConnectionFactory</jndi-name>
</resource-ref>-->
</message-driven>
</enterprise-beans>       
</jboss>

My threads are set to send the message with the following settings:
priority=1
timeout=30000ms

My queue settings:
redelivery=3
redeliverywait=3000ms

I'm on a development pc:
P4 3.06GHz
512RAM
WinXP Home
using JBoss 4.0.2, MySQL 4.1

Is this a tweaking issue or a programming issue?
How do I tweak JBoss so JMS will behave as expected?
Any help is appreciated.

Thank you very much.

[Stan James]

Curious problem. What is your JMS provider? You mentioned the messages are backed by a database ... is that a "persistant queue" setting?

Are your queues and the database fully transactional? A committed put should be on queue and persisted for assured delivery. You mentioned in the 500 thread test that some messages were not picked up or left in queue which makes me wonder if the put worked. Make sure you commit somehow & check results.

Items left in queue sound buggy for sure. JMS should trigger onMessage for all messages unless something isn't really thread safe in there somwhere.

[Raymond Ong]

Originally posted by Stan James:
Curious problem. What is your JMS provider? You mentioned the messages are backed by a database ... is that a "persistant queue" setting?

Are your queues and the database fully transactional? A committed put should be on queue and persisted for assured delivery. You mentioned in the 500 thread test that some messages were not picked up or left in queue which makes me wonder if the put worked. Make sure you commit somehow & check results.

Items left in queue sound buggy for sure. JMS should trigger onMessage for all messages unless something isn't really thread safe in there somwhere.

I have set JBoss JMS to persist the messages to MySQL. The JMS message tables are InnoDB so it supports transactions. In the 12 times that I've created 500 threads to simultaneously sent JMS messages, only 1 failed. For the failed test around 50 messages of the total 500 were lost. It's pretty weird since the destination specified in the JMS message table is my original queue name, which means I messages are not being pulled by the MDBs which I am certain are alive. If the messages had been processed previously and the number of resending attempts has lapsed then the destination would have been my dead letter queue.

I will try to integrate JBoss with ActiveMQ and see how it goes.

Thanks