Hi,
We are facing issue with Weblogic in Production Environment.
We are using weblogic 10.3.1 and have created two domain servers listening on ports 8001 and 9001.
Both domains are not related to each other and are admin servers.
Issue:
One of these servers i.e. listening on port 8001 got stuck and data was not processed.
When I restarted the server I found that server is not able to connect to JMS modules mounted on the same server due to outOfMemoryError.
Log :
####<Sep 20, 2010 12:30:13 PM IST> <Info> <JMS> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966013501> <BEA-040321> <JMSServer "WseeJmsServer" is resuming.>
####<Sep 20, 2010 12:30:13 PM IST> <Info> <Messaging> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966013501> <BEA-282003> <The messaging kernel WseeJmsServer will use up to 357,913,941 bytes of memory before paging begins>
####<Sep 20, 2010 12:30:13 PM IST> <Info> <Store> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966013517> <BEA-280050> <Persistent store "WseeJmsServer" opened: directory="E:\osb_home1031\user_projects\domains\JCA_Domain\servers\AdminServer\tmp" writePolicy="Non-Durable" blockSize=512 directIO=false driver="wlfileio2">
####<Sep 20, 2010 12:30:13 PM IST> <Info> <Messaging> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966013517> <BEA-282001> <The messaging kernel WseeJmsServer is beginning recovery of its persistent state>
####<Sep 20, 2010 12:30:13 PM IST> <Info> <Messaging> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966013533> <BEA-282002> <The messaging kernel WseeJmsServer has recovered 0 persistent messages>
####<Sep 20, 2010 12:30:13 PM IST> <Info> <JMS> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966013548> <BEA-040321> <JMSServer "wlsbJMSServer" is resuming.>
####<Sep 20, 2010 12:30:13 PM IST> <Info> <Messaging> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966013548> <BEA-282003> <The messaging kernel wlsbJMSServer will use up to 357,913,941 bytes of memory before paging begins>
####<Sep 20, 2010 12:30:13 PM IST> <Info> <Store> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966013548> <BEA-280050> <Persistent store "wlsbJMSServer" opened: directory="E:\osb_home1031\user_projects\domains\JCA_Domain\servers\AdminServer\tmp" writePolicy="Non-Durable" blockSize=512 directIO=false driver="wlfileio2">
####<Sep 20, 2010 12:30:13 PM IST> <Info> <Messaging> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966013548> <BEA-282001> <The messaging kernel wlsbJMSServer is beginning recovery of its persistent state>
####<Sep 20, 2010 12:32:45 PM IST> <Critical> <WorkManager> <ismart4> <AdminServer> <weblogic.timers.TimerThread> <<WLS Kernel>> <> <> <1284966165671> <BEA-002911> <WorkManager weblogic.kernel.Default failed to schedule a request due to java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
>
####<Sep 20, 2010 12:32:45 PM IST> <Error> <JMS> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966165671> <BEA-040123> <Failed to start JMS Server "wlsbJMSServer" due to java.lang.OutOfMemoryError.
java.lang.OutOfMemoryError
>
####<Sep 20, 2010 12:32:45 PM IST> <Warning> <Management> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966165671> <BEA-141197> <The deployment of wlsbJMSServer failed.
weblogic.management.DeploymentException: Internalrror activating the JMS Server wlsbJMSServer: java.lang.OutOfMemoryError
at weblogic.jms.deployer.BEAdminHandler.activate(BEAdminHandler.java:206)
at weblogic.management.utils.GenericManagedService.activateDeployment(GenericManagedService.java:239)
at weblogic.management.utils.GenericServiceManager.activateDeployment(GenericServiceManager.java:131)
at weblogic.management.internal.DeploymentHandlerHome.invokeHandlers(DeploymentHandlerHome.java:591)
at weblogic.management.internal.DeploymentHandlerHome.activateInitialDeployments(DeploymentHandlerHome.java:452)
at weblogic.management.internal.DeploymentHandlerHome.activateInitialDeployments(DeploymentHandlerHome.java:432)
at weblogic.management.deploy.internal.DeploymentAdapter$2.doPrepare(DeploymentAdapter.java:97)
at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:187)
at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:21)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:233)
at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:165)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:173)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:89)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
java.lang.OutOfMemoryError
>
####<Sep 20, 2010 12:32:45 PM IST> <Error> <Kernel> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966165687> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: nativeGetNewTLA.
java.lang.OutOfMemoryError: nativeGetNewTLA
at java.util.HashMap.newValueIterator(HashMap.java:971)
at java.util.HashMap$Values.iterator(HashMap.java:1039)
at weblogic.transaction.internal.TransactionManagerImpl.wakeUp(TransactionManagerImpl.java:1870)
at weblogic.transaction.internal.ServerTransactionManagerImpl.wakeUp(ServerTransactionManagerImpl.java:1517)
at weblogic.transaction.internal.WLSTimer.timerExpired(WLSTimer.java:36)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
>
####<Sep 20, 2010 12:32:45 PM IST> <Info> <WorkManager> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966165687> <BEA-002903> <Creating WorkManager from "weblogic.wsee.mdb.DispatchPolicy" WorkManagerMBean for application "WseeJmsModule">
####<Sep 20, 2010 12:32:45 PM IST> <Info> <WorkManager> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966165827> <BEA-002903> <Creating WorkManager from "weblogic.wsee.mdb.DispatchPolicy" WorkManagerMBean for application "jmsResources">
####<Sep 20, 2010 12:32:45 PM IST> <Error> <Deployer> <ismart4> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1284966165937> <BEA-149231> <Unable to set the activation state to true for the application 'jmsResources'.
weblogic.application.ModuleException: ERROR: Could not activate jmsResources!wlsb.internal.transport.task.queue.ftp
at weblogic.jms.backend.BEDestinationRuntimeDelegate.activate(BEDestinationRuntimeDelegate.java:238)
at weblogic.jms.module.JMSModule$EntityState.setState(JMSModule.java:1717)
at weblogic.jms.module.JMSModule$EntityState.setState(JMSModule.java:1667)
at weblogic.jms.module.JMSModule$EntityState.access$100(JMSModule.java:1608)
at weblogic.jms.module.JMSModule.activate(JMSModule.java:332)
at weblogic.jms.module.ModuleCoordinator.activate(ModuleCoordinator.java:194)
at weblogic.application.internal.flow.ModuleListenerInvoker.activate(ModuleListenerInvoker.java:107)
at weblogic.application.internal.flow.DeploymentCallbackFlow$2.next(DeploymentCallbackFlow.java:411)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:37)
at weblogic.application.internal.flow.DeploymentCallbackFlow.activate(DeploymentCallbackFlow.java:74)
at weblogic.application.internal.flow.DeploymentCallbackFlow.activate(DeploymentCallbackFlow.java:66)
at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:635)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:37)
at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:212)
at weblogic.application.internal.SingleModuleDeployment.activate(SingleModuleDeployment.java:16)
at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:162)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79)
at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicDeployment.java:184)
at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromServerLifecycle(BasicDeployment.java:361)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:42)
at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:187)
at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:21)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:233)
at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:165)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:173)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:89)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
weblogic.messaging.kernel.KernelException: The Messaging Kernel wlsbJMSServer has not yet been opened
at weblogic.messaging.kernel.internal.KernelImpl.checkOpened(KernelImpl.java:191)
at weblogic.messaging.kernel.internal.DestinationImpl.activate(DestinationImpl.java:280)
at weblogic.messaging.kernel.internal.QueueImpl.activate(QueueImpl.java:1687)
at weblogic.messaging.kernel.internal.DestinationImpl.resume(DestinationImpl.java:264)
at weblogic.messaging.kernel.internal.QueueImpl.resume(QueueImpl.java:1863)
at weblogic.jms.backend.BEDestinationImpl.start(BEDestinationImpl.java:609)
at weblogic.jms.backend.BEDestinationRuntimeDelegate.activate(BEDestinationRuntimeDelegate.java:235)
at weblogic.jms.module.JMSModule$EntityState.setState(JMSModule.java:1717)
at weblogic.jms.module.JMSModule$EntityState.setState(JMSModule.java:1667)
at weblogic.jms.module.JMSModule$EntityState.access$100(JMSModule.java:1608)
at weblogic.jms.module.JMSModule.activate(JMSModule.java:332)
at weblogic.jms.module.ModuleCoordinator.activate(ModuleCoordinator.java:194)
at weblogic.application.internal.flow.ModuleListenerInvoker.activate(ModuleListenerInvoker.java:107)
at weblogic.application.internal.flow.DeploymentCallbackFlow$2.next(DeploymentCallbackFlow.java:411)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:37)
at weblogic.application.internal.flow.DeploymentCallbackFlow.activate(DeploymentCallbackFlow.java:74)
at weblogic.application.internal.flow.DeploymentCallbackFlow.activate(DeploymentCallbackFlow.java:66)
at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:635)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:37)
at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:212)
at weblogic.application.internal.SingleModuleDeployment.activate(SingleModuleDeployment.java:16)
at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:162)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79)
at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicDeployment.java:184)
at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromServerLifecycle(BasicDeployment.java:361)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:42)
at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:187)
at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:21)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:233)
at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:165)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:173)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:89)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
And then It is showing message that it is unable to connect to JMS modules.
Please help.
Thanks & Regards
Mitesh
