java.rmi.UnmarshalException: error unmarshalling arguments

[Shailendher Ramanujam]

Hi,

We are facing an issue in our production env. We have searched the net high and low and we were not able to come up with any answers.
This error occurs when an ejb lookup is made from managed server 1 to manager server 2. Virtual ip is used for the lookup. It occurs intermittently and at random intervals. We are not able to identify any pattern and If the ejb call is attempted two or three times, it gets through successfully.

Env details :
server : weblogic 10.0 MP1 running on java 1.5
os : solaris

Please revert if any other details are required.

Source used for lookup :

private TreControlRemote getController() throws Exception {

Context context = null;
Properties p = new Properties();
TreControlHome treHome = null;
TreControlRemote remote = null;
ConfigurationLoader lAppLoader = null;
try {
mLog.debug("Entering");
lAppLoader = PropertiesFileLoader.getInstance("context.properties");
p.put(Context.INITIAL_CONTEXT_FACTORY, lAppLoader.getValue("INITIAL_CONTEXT_FACTORY"));
p.put(Context.PROVIDER_URL, lAppLoader.getValue("PROVIDER_URL"));
context = new InitialContext(p);
mLog.debug("context : " + context.getEnvironment());
remote = null;
treHome = (TreControlHome) context.lookup("CONTROL");
mLog.debug("Object --->>>>" + treHome);
remote = (TreControlRemote) treHome.create();
mLog.debug("Leaving");
} catch (Exception ex) {
mLog.fatal("Exception while getting remote", ex);
ex.printStackTrace();
throw ex;
} finally {
lAppLoader = null;
}
return remote;
}

The url is a virtual ip pointing to managed server 2 and it contains a ejb with jndi "CONTROL". The problem is that it successful on certain occassions and fails randomly.

stack trace of the error :

javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.io.StreamCorruptedException]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:74)
at weblogic.jndi.internal.WLContextImpl.translateException(WLContextImpl.java:426)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:382)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:367)
at javax.naming.InitialContext.lookup(InitialContext.java:351)
```````````````````````````````````````````````````````````````````
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.io.StreamCorruptedException
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:221)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:338)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:252)
at weblogic.jndi.internal.ServerNamingNode_1001_WLStub.lookup(Unknown Source)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:379)
... 33 more
Caused by: java.io.StreamCorruptedException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1332)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
at weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:195)
at weblogic.rjvm.MsgAbbrevInputStream.readObject(MsgAbbrevInputStream.java:565)
at weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:191)
at weblogic.jndi.internal.RootNamingNode_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:224)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:479)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:475)
at weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:59)
at weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:1016)
... 2 more

Obtained the below mentioned stacktrace from the weblogic log. Could this error be related to our problem mentioned above?

####<Aug 25, 2009 2:11:04 AM BST> <Info> <RJVM> <pkssv049> <M1AP4> <ACTIVE ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <1251162664181> <BEA-000513> <Failure in heartbeat trigger for RJVM: 5433424963141690658S:169.93.73.0:10040,10040,-1,-1,-1,-1,-1:pkssv049.***.net:10240,pkssv049.***.net:10241,pkssv050.***.net:10240,pkssv050.***.net:10241:LIQP1_LMSDomain:M1AP3
java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl@189ed0e - id: '5433424963141690658S:169.93.73.0:10040,10040,-1,-1,-1,-1,-1:pkssv049.***.net:10240,pkssv049.***.net:10241,pkssv050.***.net:10240,pkssv050.***.net:10241:LIQP1_LMSDomain:M1AP3' connect time: 'Mon Aug 24 20:24:02 BST 2009'' has already been shut down.
java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl@189ed0e - id: '5433424963141690658S:169.93.73.0:10040,10040,-1,-1,-1,-1,-1:pkssv049.***.net:10240,pkssv049.***.net:10241,pkssv050.***.net:10240,pkssv050.***.net:10241:LIQP1_LMSDomain:M1AP3' connect time: 'Mon Aug 24 20:24:02 BST 2009'' has already been shut down
at weblogic.rjvm.ConnectionManager.getOutputStream(ConnectionManager.java:1686)
at weblogic.rjvm.ConnectionManager.createHeartbeatMsg(ConnectionManager.java:1629)
at weblogic.rjvm.ConnectionManager.sendHeartbeatMsg(ConnectionManager.java:607)
at weblogic.rjvm.RJVMImpl$HeartbeatChecker.timerExpired(RJVMImpl.java:1540)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:464)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)

Any help would be greatly appreciated.>

[Shailendher Ramanujam]

Just wanted to add the following info.

Is the problem intermittent, or does reproduce every single time? If the problem is intermittent, do you know what conditions it occurs under?
It occurs intermittently and we are not able to observe any pattern.

Are there any other errors/warnings logged either on the local server or on the remote server?
We see a lot of connection refused errors in the weblogic log

Are both the managed servers in the same domain?
Yes

[Deepak Bala]

<Failure in heartbeat trigger for RJVM:

Looks like a network problem. Heartbeat fails after some time and the server becomes invisible. What might have caused the Unmarshalling problem was that the server severed the connection midway, which lead to a Corrupted stream. The corrupted stream is unmarshalled and this leads to an exception.

Does this occur for this EJB alone or all other EJBs ?>

[Shailendher Ramanujam]

Hi,

We applied a release(but no code change in this ejb call) on Aug 8th, after which we started getting the streamcorruptedexception error. I check the managed server logs and observed that the heartbeat failure started occurring from aug 10th (around 26times from aug 10th to till date - sometimes twice a day and on other days none)

Any idea why the 'Failure in heartbeat trigger for RJVM' error occurs? We have been told that there was no change in the environment from july. Meanwhile i'll search the net for more infomation.

I want to replicate the same in my offshore environment? is it possible?

Many thanks,
Shailendher

[Deepak Bala]

It is difficult to know why the heart beat is actually failing. You should be able to replicate the problem by trying to connect to an EJB deployed in your colleague's machine and then disconnecting the LAN cable midway.

[Nehal Shahs]

Hi Shailendher

Did you find a solution to this connection closed exception?
I have a similar issue with weblogic 10 in our production environment. The process is doing a lookup to a remote server for the db connection and on random basis, the connection has already been closed error occurs.

Please let me know if you found a solution to this?

[Deepak Bala]

Nehal Shahs wrote:Hi Shailendher

Did you find a solution to this connection closed exception?
I have a similar issue with weblogic 10 in our production environment. The process is doing a lookup to a remote server for the db connection and on random basis, the connection has already been closed error occurs.

Please let me know if you found a solution to this?

Your problem seems different. Please start a new thread and post a stack trace