• Post Reply Bookmark Topic Watch Topic
  • New Topic

tomcat connections are not cleared on RHEL server after failover  RSS feed

 
stavros tseriotis
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hello,
I have 2 RHEL 5.11 servers that run tomcat application. Both of them are running the application but only one of them is acting as the "live" server.
There is a firewall responsible to redirect the user connections depending on which server is the active one. When the firewall decides that something is wrong with the active server (ex. timeout in network communication) then redirects the packages to the other server which is now acting as the "live" one.
The problem is that when this failover occurs even though in the firewall all connections are cleared from the first server, i can see that when i login to the linux server connections are still "established" on the application port. These connections are only cleared when i stop and start tomcat.
When i stop tomcat i can see these messages in catalina.out:
Dec 15, 2015 2:14:46 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapp] created a ThreadLocal with key of type [org.apache.xmlbeans.XmlBeans$1] (value [org.apache.xmlbeans.XmlBeans$1@1584b8d]) and a value of type [java.lang.ref.SoftReference] (value [java.lang.ref.SoftReference@1f8183b]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Dec 15, 2015 2:14:46 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapp] created a ThreadLocal with key of type [com.arcot.core.util.PerfCountersPerThread$2] (value [com.arcot.core.util.PerfCountersPerThread$2@2d2d8c]) and a value of type [java.util.HashMap] (value [{authentication.=com.arcot.core.util.PerfCounters@fee424, issuance.=com.arcot.core.util.PerfCounters@342b49}]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.


Is there a way to clear these connections without restarting tomcat application?

Thank you.
 
Campbell Ritchie
Marshal
Posts: 56518
172
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Don't know, but welcome to the Ranch and I shall copy this question in our Tomcat forum.
 
stavros tseriotis
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you Ritchie!
 
Tim Holloway
Saloon Keeper
Posts: 18789
74
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
That takes us into clustering and load-balancing.

But first, I should make something very clear. The HTTP protocol is not about "establishing a connection" to a webapp server. HTTP is not a time-sharing protocol. Instead, it's a series of grab-and-go request/response cycles. A client makes a request to the server, the server responds to the request in a strict one-to-one correspondence. Each request-response cycle is done as a separate client connection to the server and when the entirety of the response has been received from the server, the connection is severed. A new request requires a new connection.

Theoretically, anyway. In reality the overhead of establishing connections is so high that it's common for webapp servers to maintain a "keep-alive" mechanism and re-use the connection from a previous request. However, this mechanism is required to be transparent in operation, so it doesn't really "exist" for most discussions, including this one.

Also firewalls aren't usually smart enough to determine dead servers and how to route around them. That function is usually done by some sort of reverse proxy, which may be a hardware device (an F5 for example) or software (such as a front-end Apache or Nginx server).

If your webapp is based on the ReST protocol and you have a symmetrical collection of backend Tomcat servers, the proxy is at liberty to throw incoming requests at any one of those servers. However, if you are maintaining an HTTPSession, then it gets a little stickier, since HTTPSession objects by default are in-memory constructs on whatever Tomcat instance constructed them. So if that server goes down, then the HttpSession information is lost. Unless.

The first line of defense against session data loss is to have Tomcat configured such that it serializes session data to a persistent storage. For example, as a file in the Tomcat "work" directory. That makes it more likely that when Tomcat comes back up, the session will be intact. However, if you're looking for a High-Availability cluster, you need further capabilities, because otherwise the user session is unavailable until (or if) the failing server comes back online. This is where you configure Tomcat to do session sharing. That's an option that allows Tomcat servers to pass serialized HTTPSessions between each other. That way if the original server goes offline, one of the other servers can take over for the user.

For more details, look at the Tomcat documentation's clustering information.
 
stavros tseriotis
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you Tim for your reply and your explanation.
The firewall is indeed an F5 firewall. Although from F5 the administrator of the firewall sees that the packages are dropped from the server and that only one of them has connections i can see that in the "failed" tomcat the connections exists and are not cleared.
Also when the failover occurs in reality the server is not coming down, just F5 cannot ping tomcat server for few seconds and it thinks that the server is offline thats why it drops the connections from that server and send it to the other.
The ideal scenario for me is to be able to have one active tomcat server and when F5 decides that the server is "offline" to send the packages to the other server without remaining connections on the "failed" server.
But as i understand from your post you are saying that in order to have a high availability cluster then tomcat should be configured so that both servers are active and share connections from users. I dont see any problem on that i will try to look how this can be done.
Thank you again for your reply!
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!