jboss ejb jms redelivery

[subhash uppalapati]

Hi,
I am using EJB 3 with JBOSS. I am using the default JMS provider JBOSSMQ.
When the session bean delivers a message(entity) to the JMS destination, the MDB receives the object in its onMessage() method. If I have a problem with the insert query in the onMessage method, because of the redelivery feature the same message is getting delivered multiple times and in turn creating duplicate records in the DB for other entities.
Is making the number of redelivery attempts in standardjboss.xml to 1 a solution? Is there any other way to avoid the redelivery?

Note: I have logic to catch the exception, if any occurs, in the onMessage method.

Thanks,
subhash

[Reza Rahman]

Subhash,

That seems a little strange. If it's an unexpected exception, JTA should be rolling back all changes and the message should be re-delivered. Typically, this creates an infinite loop, not erroneous records? At any rate, take a look at dead letter queues (DLQs) for JBoss messaging and max retries.

Regards,
Reza

[subhash uppalapati]

Thanks Reza for your reply.
I checked my code again and the problem seems to be I am not properly catching the right exception.

[RaviNada Kiran]

What is the argument you set in the message.setRedeliveryLimit() parameter??

[subhash uppalapati]

I am not at all using this method to set redeliverlimit.
How can I use this? I tried to find out if there is any method like this for Objectmessage but I could not find it.

[RaviNada Kiran]

It may be of any Type of message this property will be there .

So i your case as you did not set ,it is taking the properties from the server which will be default to -1.

so override this with

message.setRedeliveryLimit()// NO parameter

Try setting an empty parameter and check

[subhash uppalapati]

Let me explain my problem again..
1) I am sending message which is an entity

producer.send(message);

2) The MDB receives the message and tries to persist using

em.persist(entity);

3) The persistance fails because the table structure in the DB is not the same as the entity. Also happens when there is bad data in the entity which cannot be inserted in the DB.
4) JBOSSMQ tries to deliver the message again and again causing the server to kneel down.

Things tried:
1) Setting the redelivered to false

 message.setJMSRedelivered(false);

2) Catching the exception in the MDB onMessage method
3) Chaging the redelivery limit by adding following line to jbossmq-service.xml mbean DLQ.

<attribute name="RedeliveryLimit">1</attribute> 

4) Also changed the value in the standardjboss.xml file

<MaxTimesRedelivered>1</MaxTimesRedelivered>

Please let me know if there is anything else that I can try. Please let me know if you have any ideas/suggestions.

[Reza Rahman]

Subhash,

What is the issue at this point? Is the "bad message" not in the DLQ once the MDB rolls back? If not, some other folks can probably help you with the JBoss MQ configuration. I think the MDB code is fairly sound.

Regards,
Reza

[subhash uppalapati]

The issue that I am trying to solve is:
1) I am getting the following exception

 org.hibernate.exception.SQLGrammarException: could not insert: 

2) After that following warning appears

WARN  [AbstractDLQHandler] Message redelivered=2 max=1 sending it to the dlq org.jboss.mq.SpyObjectMessage {
Header { 
   jmsDestination  : QUEUE.DLQ
   jmsDeliveryMode : 2
   jmsExpiration   : 0
   jmsPriority     : 4
   jmsMessageID    : ID:3-12367966871291051
   jmsTimeStamp    : 1236796687129
   jmsCorrelationID: null
   jmsReplyTo      : null
   jmsType         : null
   jmsRedelivered  : true
   jmsProperties   : {JBOSS_ORIG_MESSAGEID=ID:3-12367966871231050, JBOSS_ORIG_DESTINATION=QUEUE.DLQ, tp=test, JMS_JBOSS_REDELIVERY_LIMIT=1, JMS_J
BOSS_REDELIVERY_COUNT=2}
   jmsPropReadWrite: false
   msgReadOnly     : true
   producerClientId: ID:3
}

3) This goes into infinite loop inturn increasing the size of the log file and the server runs out of memory.

[Reza Rahman]

Subhash,

This appears to be working correctly. Is the message not in the DLQ? Is the MDB really being invoked multiple times for the same message? Are you sure there isn't an issue somewhere else that's causing the server to crash (such as on the client end?)

Regards,
Reza

[subhash uppalapati]

Hi,
The message are going to the DB for sure and getting stored in JMS_MESSAGES table. The MDB is getting called again and again for the same message. I could not find any issue with the client.
The problem is whent he MDB tries to persist the entity, it cannot due to SQL exception and the same MDB is getting called and trying the same persist again and again.
Please let me know .

Thanks,
Subhash

[RaviNada Kiran]

There is no possibility to block the onMessage() method of your MDB to be called. What I can think of as a solution for your problem is the following: you should make the check and see if the conditions are fulfilled in the onMessage() method. Only then process the message if the conditions passed. If they didn't pass, you must make sure the message is not lost and not redelivered.

[RaviNada Kiran]

i have thought an idea for your problem :

Does messages stored in a Queue have a correlation id ??

If it has know extract the id and take over the control .

This way you can have your pool of MDB's also.

[Jaikiran Pai]

The problem is whent he MDB tries to persist the entity, it cannot due to SQL exception and the same MDB is getting called and trying the same persist again and again.
Please let me know .

Redelivering the message in case of a failure is the expected behaviour. Why not set a redelivery limit if you want the message not to redelivered after X number of attempts?

[RaviNada Kiran]

Jaikiran Pai wrote:

Redelivering the message in case of a failure is the expected behaviour. Why not set a redelivery limit if you want the message not to redelivered after X number of attempts?

I think you did not read his all posts of this question.

[Anurag Blore]

Is your subscriber is Durable If yes then the message must be acknowledged to tell topic/queue the msg is received.

To resolve your issue, you must set the unique Id for each message may be in msg header. Now you should use a Cache either a HashMap or Jboss cache or something else to keep track of each failed message. You should get the unique msg key in cache to identify if a redelivered msg is failed or not. If you are getting a failed message you should not process this msg again and just store in a log table and commit the message. If you will commit the msg queue/topic will not redeliver the same msg as an infinite flow. Later you can check the log table to analyse the msg further.

Regarding DLQ queue, its a configuration which depends on jms server used its not available with Tibco ems I believe.

[Reza Rahman]

Anurag,

Tibco EMS has a DLQ as well. All modern messaging servers do.

The issue here is that the JBoss configuration is not correct. I would post that in detail.

Regards,
Reza

[Jaikiran Pai]

RaviNada Kiran wrote:

Jaikiran Pai wrote:
Redelivering the message in case of a failure is the expected behaviour. Why not set a redelivery limit if you want the message not to redelivered after X number of attempts?

I think you did not read his all posts of this question.

I must admit, i am a bit confused after reading all the posts again I am finding it hard to understand whether the issue is:

1) Even after the persisting of some entity fails in the MDB, the data is present in the DB?
2) OR the persisting of the entity fails, transaction is correctly rolled back and the data in the DB is cleared. The message is then redelivered and the same persistence exception occurs. This goes on indefinitely (i.e. the redelivery happens indefinitely)

What is suggested was to handle #2. Which would limit the number of times the redelivery happens. However i am still not clear why the OutOfMemoryError happens since, even if the redelivery limit is not set explicitly, i think the default is 10 which will then direct the message to the DLQ.

Subhash,

The best thing at this point would be to post the relevant logs with the OutOfMemoryError.

[subhash uppalapati]

Please find the error log messages that I am getting below.
OutOfMemory occurs not in Java but because in less time, the transaction manager or somebody tries to persist this again and again increasing the logs. The log file size grows beyond the allowed file system size. It causes the server to lack resources to write new logs and in turn bringing down the server which is trying to persist the entity which it cannot because of SQL exception.

Following logs repeats.....

ERROR [org.hibernate.util.JDBCExceptionReporter] Unknown column 'anonymousId' in 'field list'
2009-03-30 10:43:34,353 ERROR [packageName] org.hibernate.exception.SQLGrammarException: could not insert: [EntityName]
2009-03-30 10:43:34,353 DEBUG [MDBName] Done Persisting message
2009-03-30 10:43:34,355 DEBUG [org.jboss.ejb3.entity.ManagedEntityManagerFactory] ************** closing entity managersession **************
2009-03-30 10:43:34,356 WARN  [org.jboss.resource.adapter.jms.inflow.dlq.AbstractDLQHandler] Message redelivered=2 max=1 sending it to the dlq org.jboss.mq.SpyObjectMessage {
Header {
   jmsDestination  : QUEUE.DLQ
   jmsDeliveryMode : 2
   jmsExpiration   : 0
   jmsPriority     : 4
   jmsMessageID    : ID:1-1238435014345363
   jmsTimeStamp    : 1238435014345
   jmsCorrelationID: null
   jmsReplyTo      : null
   jmsType         : null
   jmsRedelivered  : true
   jmsProperties   : {JBOSS_ORIG_MESSAGEID=ID:1-1238435014334362, JBOSS_ORIG_DESTINATION=QUEUE.DLQ, tp=temp, JMS_JBOSS_REDELIVERY_LIMIT=1, JMS_JBOSS_REDELIVERY_COUNT=2}
   jmsPropReadWrite: false
   msgReadOnly     : true
   producerClientId: ID:1
}
}
2009-03-30 10:43:34,357 DEBUG [MDBName] Persisting message
2009-03-30 10:43:34,358 INFO  [STDOUT] Hibernate: insert into "query"
2009-03-30 10:43:34,359 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 1054, SQLState: 42S22

[Anurag Blore]

Subhash,


you can check the post here to handle the logs here
logging
This will avoid the resource crunch and also set the log level to error.

In your case are you getting error for redelivered message again and again. Can you please re-implement something which I mentioned in this link to commit the message if it has failed in first time processing and failed due to application error.

[mahor mahor]

Try

@TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED)

in your MDB class.

[subhash uppalapati]

I am getting the following exception if I add the Transaction not supported attribute.

EntityManager must be access within a transaction

[mahor mahor]

Do you have an EntityManager in your MDB ?

Use this annotation in your MDB
@TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED)

and this where you want to create a transaction
@TransactionAttribute(TransactionAttributeType.REQUIRE_NEW)

[subhash uppalapati]

Hi Guys,
Thank you for all your replies. Today I came to know that I am using DLQ as the default queue, which is a blunder. That's why the message is getting delivered again and again causing infinite loop.
Now I moved to using my own queue and all problems are solved. Thank you to all of you.