I could really use some expertise from someone who knows exactly how mod_jk +
tomcat work together when you have multiple web servers w/ mod_jk pointed at multiple jboss/tomcat app servers. My team has read pages & pages of forums & documentation, and we can't seem to find anybody who can definitely tell us how to get mod_jk to behave the way we need it to (or if its even possible). Although we've been working with this set up for over a year, its possible we're missing something completely stupid =).
We've got a production environment that serves tens of thousands of concurrent requests when under load. We've got multiple web servers configured with Apache 2 and mod_jk. Each web server has a worker configured for *each* app server (so, if we had 5 app servers, each web server's mod_jk config has 5 workers configured). Each app server (running a single instance of
JBoss 4.0.5) is configured to handle 200 requests in the tomcat server.xml. We do not have session federation, so we are using mod_jk's sticky sessions.
The problem is when we get under high load and the app servers get saturated with requests. Since mod_jk can't get an open
thread on the desired app server, it starts throwing 500 errors and then "fails over" to the next app server. Since that server is also busy, it throws more errors & keeps failing over. Since all 5 of our app servers may be totally busy at any given instant, mod_jk is just throwing errors galore, and the whole site becomes useless (even though its still technically "up").
So, basically, we need to be able to control the flow of traffic from mod_jk into our app servers, so that if mod_jk finds the desired app server (based on session cookie) is too busy, it will QUEUE requests until that server opens up, instead of failing over to the next, while still using a load-balanced approach for requests that don't yet have a session. Finally, if that queue gets too deep, we want to be able to reject new requests altogether without impacting our ability to handle the already-in-process requests. And ideally, we'd need this setup to work with multiple web servers & multiple app servers, but we're totally willing to rejigger things a bit if we can get something close to the behavior we need without totally killing performance.
Is this even possible with mod_jk (we're also looking into mod_proxy_ajp to see if it will work)? Or is there another approach we can take without having to completely change our architecture? We're running out of time to solve this problem (at least in a temporary way for the time being), so any/all help is appreciated. If there's someone out there with experience configuring multiple web servers & app servers with mod_jk in a single environment, we'd love to talk to you directly. We may even have the option of paying for support if we can get fast enough help.
Thanks to everyone who took the time to read this!
- Josh W