• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

MDB Listeners down

 
Greenhorn
Posts: 2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello All,

We have two MDB listeners on our APP server. These are used by two applications to contact our applciation and get data.
These listeners are getting down frequently in recent times. This is causing severe business loss as ours is a mission critical application.
These listeners when get message to our application, we are running couple of DB queries to get data.
Some large transaction is holding the database tables as we see some unusual lock escalation happening on DB tables at the time listeners were down.

Please see below logs and please help me in finding root cause for the issue. Is it a DB related issue? or JDBC connections related issue??? Please help me out!!

Prod APP server logs:

[4/11/13 15:17:00:114 EDT] 00000042 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/ABCD_EFGH.
[4/11/13 15:17:00:118 EDT] 00000042 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 0, SQLState: null
[4/11/13 15:17:00:119 EDT] 00000042 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions Connection not available, Timed out waiting for 10005
[4/11/13 15:17:00:126 EDT] 00000042 SibMessage W [:] CWSJY0003W: JMSCC0109: A message driven bean threw a runtime exception '
Message : java.lang.StringIndexOutOfBoundsException: String index out of range: -814
Class : class java.lang.StringIndexOutOfBoundsException
Stack : java.lang.String.substring(String.java:1937)

[4/11/13 15:17:00:132 EDT] 00000042 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /global/site/vendor/WAS/WebSphere7/AppServer/profiles/base/logs/ffdc/ABCD_EFGH.txt com.ibm.ejs.jms.listener.ServerSession.connectionConsumerOnMessage 450
[4/11/13 15:17:01:094 EDT] 00000043 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/ABCD_EFGH.
[4/11/13 15:17:01:103 EDT] 00000042 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /global/site/vendor/WAS/WebSphere7/AppServer/profiles/base/logs/ffdc/ABCD_EFGH1.txt com.ibm.ejs.container.MDBWrapper.onMessage 83
[4/11/13 15:17:01:106 EDT] 00000043 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 0, SQLState: null
[4/11/13 15:17:01:107 EDT] 00000043 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions Connection not available, Timed out waiting for 10001
[4/11/13 15:17:01:111 EDT] 00000043 SibMessage W [:] CWSJY0003W: JMSCC0109: A message driven bean threw a runtime exception '
Message : java.lang.StringIndexOutOfBoundsException: String index out of range: -814
Class : class java.lang.StringIndexOutOfBoundsException
Stack : java.lang.String.substring(String.java:1937)

[4/11/13 15:17:01:111 EDT] 00000042 LocalExceptio E CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method "onMessage" on bean "BeanId(ABCDEFGH#ABCDMDB.jar#ABCDEFHBean, null)". Exception data: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor2357.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:44)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:100)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:136)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:567)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -814
at java.lang.String.substring(String.java:1937)

[4/11/13 15:17:01:674 EDT] 00000042 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /global/site/vendor/WAS/WebSphere7/AppServer/profiles/base/logs/ffdc/ABCD.txt com.ibm.ejs.container.LocalExceptionMappingStrategy.setUncheckedException 178
[4/11/13 15:17:01:689 EDT] 00000042 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[4/11/13 15:17:01:689 EDT] 00000043 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[4/11/13 15:17:02:256 EDT] 00000042 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /global/site/vendor/WAS/WebSphere7/AppServer/profiles/base/logs/ffdc/ABCD1.txt com.ibm.ejs.csi.TranStrategy.rollback 375
[4/11/13 15:17:02:295 EDT] 00000042 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /global/site/vendor/WAS/WebSphere7/AppServer/profiles/base/logs/ffdc/ABCD2.txt com.ibm.ejs.jms.listener.ServerSession.run 283
[4/11/13 15:17:02:434 EDT] 00000042 ServerSession E com.ibm.ejs.jms.listener.ServerSession run WMSG0036E: Maximum message delivery retry count of 0 reached for MDB ABCDBean, JMSDestination jms/ABCDReq, MDBListener stopped
[4/11/13 15:17:02:434 EDT] 00000043 ServerSession E com.ibm.ejs.jms.listener.ServerSession run WMSG0036E: Maximum message delivery retry count of 0 reached for MDB DEFHBean, JMSDestination jms/DEFHReq, MDBListener stopped


[4/11/13 15:17:07:485 EDT] 00000079 MDBListenerIm I WMSG0043I: MDB Listener stopped for JMSDestination jms/ABCDReq
[4/11/13 15:17:07:519 EDT] 0000007b MDBListenerIm I WMSG0043I: MDB Listener stopped for JMSDestination jms/DEFHReq
[4/11/13 15:17:12:464 EDT] 00000073 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/ABCD_EFGH.
[4/11/13 15:17:12:467 EDT] 00000073 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 0, SQLState: null
[4/11/13 15:17:12:468 EDT] 00000073 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions Connection not available, Timed out waiting for 10001
[4/11/13 15:17:12:471 EDT] 00000073 servlet E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: Uncaught exception created in one of the service methods of the servlet ABCDValidationService in application ABCDEFGH. Exception created : org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
at sun.reflect.GeneratedMethodAccessor300.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.hibernate.context.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:301)
at $Proxy32.beginTransaction(Unknown Source)
at com.util.InsertABCDTables.createRequestId(InsertABCDTables.java:113)
at com.handler.MainValidatonHandler.validateContract(MainValidatonHandler.java:85)
at com.service.ABCDValidationService.doPost(ABCDValidationService.java:69)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:557)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:607)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:984)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1069)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Caused by: com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 10001
at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1524)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:685)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:635)
at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:69)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
... 31 more
Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 10001
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1666)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2496)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1063)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:700)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:668)
... 34 more

[4/11/13 15:17:28:349 EDT] 0000000c ThreadMonitor W WSVR0605W: Thread "WebContainer : 5" (00000047) has been active for 635968 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at com.wily.introscope.agent.probe.net.ManagedSocketInputStream.read(ManagedSocketInputStream.java:214)
at com.ibm.db2.jcc.t4.ab.b(ab.java:195)
at com.ibm.db2.jcc.t4.ab.c(ab.java:243)
at com.ibm.db2.jcc.t4.ab.c(ab.java:354)
at com.ibm.db2.jcc.t4.ab.v(ab.java:1139)
at com.ibm.db2.jcc.t4.db.a(db.java:37)
at com.ibm.db2.jcc.t4.t.a(t.java:32)
at com.ibm.db2.jcc.t4.sb.h(sb.java:141)
at com.ibm.db2.jcc.b.fm.bb(fm.java:1923)
at com.ibm.db2.jcc.b.gm.dc(gm.java:2551)
at com.ibm.db2.jcc.b.gm.e(gm.java:3252)
at com.ibm.db2.jcc.b.gm.Rb(gm.java:612)
at com.ibm.db2.jcc.b.gm.executeUpdate(gm.java:595)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecuteUpdate(WSJdbcPreparedStatement.java:1175)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.executeUpdate(WSJdbcPreparedStatement.java:816)
at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:23)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2247)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2660)
at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:56)
at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:250)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:234)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:141)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
at com.util.InsertABCDTables.createRequestId(InsertABCDTables.java:118)
at com.handler.MainValidatonHandler.validateContract(MainValidatonHandler.java:85)
at com.validation.service.ABCDValidationService.doPost(ABCDValidationService.java:69)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:557)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:607)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:984)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1069)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
 
Greenhorn
Posts: 19
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi, Indu
There are 2 issue in your description.

First of all check datasource connection pool, it is not enough connection:

Connection not available while invoking method createOrWaitForConnection for resource jdbc/ABCD_EFGH.
[4/11/13 15:17:12:467 EDT] 00000073 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 0, SQLState: null
[4/11/13 15:17:12:468 EDT] 00000073 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions Connection not available, Timed out waiting for 10001



All your threads waited free connection form pool. Also i woukd recommend to you to read Hibernate + WebSphere

Second problem is application design. As far as i understand you get data to MDB and try to process it immideatly. This is not good for your application.
You can you use delayed processing with claim check pattern to overcome your issues.
Just process data in EJB with new transaction, do not join MDB.



 
Priya Parthiban
Greenhorn
Posts: 2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Indu Priyanka wrote:Hello All,

We have two MDB listeners on our APP server. These are used by two applications to contact our applciation and get data.
These listeners are getting down frequently in recent times. This is causing severe business loss as ours is a mission critical application.
These listeners when get message to our application, we are running couple of DB queries to get data.
Some large transaction is holding the database tables as we see some unusual lock escalation happening on DB tables at the time listeners were down.

 
Why fit in when you were born to stand out? - Seuss. Tiny ad:
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic