Application disconnects from session in production server

[vijay jamadade]

Hi,

I have an application developed in Jsf, hibernate eh-cache. i deployed it on jboss server and users are directly accessing jboss url on port 8080 in internal network. Now problem is, number of users for this application has been increased. This application have one page containing 20 records which gets fetched from cache on jsf page. I have used a4j:poller tag which refreshes this page via ajax call and updates it. Earlier this was working fine but nowadays number of users have been increased. Issue is i get nullpointer exception after some time once i logged in to application and user gets redirected to login page. If user re-logins he gets directed to same page number before logout. There is no reason of throwing null pointer exception and this application works fine in qualification environment. We checked everything,

1) session is still there on server. thing is user gets disconnected from it.
2) Network bandwidth seems to be okay.
3) we have used hibernate eh cache and serialized java classes.

I am still clueless now how to troubleshoot and find root cause for this. Could you guys please throw some light on it for checking it? Some people say that it is due to ajax calls from multiple users and ajax response it getting lost in between.

[Jayesh A Lalwani]

You'll have to see where the NullPointerException is being thrown from. You can start by looking at the stacktrace

Most of the time, NullPointerExceptions are not because of higher load. It could be because you have lots of differrent kinds of users, they might be providing some data that you never tested on. It could be bad data that causes NullPointerException

There is a slight chance, that NullPointerException is because of a concurrency issue. If your code is not implemented with concurrency in mind, you might be sharing objects between requests, and once request might be overwriting other requests references. If this is the case, you are screwed, and that's putting it mildly.

So, what you need to do is
a) Get the stack trace of the exception
b) Figure out how to reproduce the exception. You might already have logs, or maybe you need to add logging. You will have to find out what exactly does the user do to cause the problem
c) Try to reproduce the error in your QA/Dev environment

If you can reproduce the error, you have hopes of fixing it. You can debug through the code to find the problem. If you cannot reproduce the error, you are kind of screwed. Yes, screwed is the technical term here. If it is impossible to reproduce the problem 100% of the time, it is most likely a concurrency problem. You will need to start praying to the software gods

[vijay jamadade]

Thanks Jayesh.

I tried to add more logging to application but nullpointer exceptions have no reason for being thrown. Value is there and suddenly as its getting disconnected from session null pointer exception occurs. One more thing most of classes are implement Serializable interface. I am not sure whether it will impact or cause any issues. User gets disconnected from any screen any click. No fix sequence to reproduce this which is major stopper for fixing this.

[William Brogden]

I tried to add more logging to application but nullpointer exceptions have no reason for being thrown.

Of course there is a reason, you just have not found it yet.

Why are you using the expression:

Value is there and suddenly as its getting disconnected from session

You say

One more thing most of classes are implement Serializable interface.

Note that "most" is not what you need, ALL objects with references in a session MUST be serializable or you get the mysterious vanishing object problem.

Bill

[Jayesh A Lalwani]

Just throwing this out there. Did you add servers to your farm when the number of users increased? If you did, then you have to do one out of these 2 things
a) Configure sticky sessions in the firewall. This will make sure that all requests from the same HTTP session go to the same web server
b) Enable session replication between web servers

If you don;t do one of these, then requests from a particular user might be distributed across differrent servers. Since, the web servers are not sharing the sessions, one of them might have nothing in it's session. From the logs it will look like that the session "dissappeared", but really it didn't.

[vijay jamadade]

ote that "most" is not what you need, ALL objects with references in a session MUST be serializable or you get the mysterious vanishing object problem.

Hi William, I will check this.

Hi Jayesh, there is no web server used. Users are directly accessing application server using ip address and port in internal network. We are using only one server no managed instances so i think sticky session implementation is not required. I have a strong feeling that this is due to some change in environment but finding it has become very difficult.

[E Armitage]

The advice to investigate this from the exception you are getting is very good. What code is throwing that nullpointer exception? I