help w/ multiple web servers using mod_jk in Production env

[Josh White]

I could really use some expertise from someone who knows exactly how mod_jk + tomcat work together when you have multiple web servers w/ mod_jk pointed at multiple jboss/tomcat app servers. My team has read pages & pages of forums & documentation, and we can't seem to find anybody who can definitely tell us how to get mod_jk to behave the way we need it to (or if its even possible). Although we've been working with this set up for over a year, its possible we're missing something completely stupid =).

We've got a production environment that serves tens of thousands of concurrent requests when under load. We've got multiple web servers configured with Apache 2 and mod_jk. Each web server has a worker configured for *each* app server (so, if we had 5 app servers, each web server's mod_jk config has 5 workers configured). Each app server (running a single instance of JBoss 4.0.5) is configured to handle 200 requests in the tomcat server.xml. We do not have session federation, so we are using mod_jk's sticky sessions.

The problem is when we get under high load and the app servers get saturated with requests. Since mod_jk can't get an open thread on the desired app server, it starts throwing 500 errors and then "fails over" to the next app server. Since that server is also busy, it throws more errors & keeps failing over. Since all 5 of our app servers may be totally busy at any given instant, mod_jk is just throwing errors galore, and the whole site becomes useless (even though its still technically "up").

So, basically, we need to be able to control the flow of traffic from mod_jk into our app servers, so that if mod_jk finds the desired app server (based on session cookie) is too busy, it will QUEUE requests until that server opens up, instead of failing over to the next, while still using a load-balanced approach for requests that don't yet have a session. Finally, if that queue gets too deep, we want to be able to reject new requests altogether without impacting our ability to handle the already-in-process requests. And ideally, we'd need this setup to work with multiple web servers & multiple app servers, but we're totally willing to rejigger things a bit if we can get something close to the behavior we need without totally killing performance.

Is this even possible with mod_jk (we're also looking into mod_proxy_ajp to see if it will work)? Or is there another approach we can take without having to completely change our architecture? We're running out of time to solve this problem (at least in a temporary way for the time being), so any/all help is appreciated. If there's someone out there with experience configuring multiple web servers & app servers with mod_jk in a single environment, we'd love to talk to you directly. We may even have the option of paying for support if we can get fast enough help.

Thanks to everyone who took the time to read this!

- Josh W

[Tim Holloway]

With that kind of workload that's asking for a major enterprise-level solution, and from the looks of it, is probably something that would require a vendor's proprietary enhancements. We're talking something like coughing up some significant cash to Red Hat, Sun, IBM, whoever.

This is the kind of challenge I enjoy solving, but I don't happen to have anything sitting in my back pocket ready to use. It would probably take me a couple of weeks of analysis, research and maybe even testing and the end result would quite possibly be a list of vendors who have solutions for sale.

I like quick and cheap, too, but sometimes it's not a viable option.

[Josh White]

Thanks for your reply, Tim!

I was under the impression that JBoss was capable of handling this kind of load just fine with proper scaling. I've used it with several other clients with high loads (not quite this high) and never had a problem. Its mod_jk I'm not so sure about. What I'd love is to find someone with significant expertise in mod_jk at least to talk about exactly how it handles load balancing (not just what the documentation says) and if there's any settings/tuning we might be able to make changes to. Ultimately, we're going to ditch mod_jk and use stateless requests, but don't have time to do that before our next big influx of traffic.

Anyone out there with this kind of expertise whom I may be able to get in touch with?

[Tim Holloway]

JBoss's primary strength is that it provides a full J2EE stack. It does this by building on other components, and in the case of servlet/JSP that means Tomcat or Jetty. Jetty is supposed to provide better performance, but I don't know how well it does clustering, and once you've maxed out a single server, you need clustering.

I haven't been into the details of mod_jk lately, but I don't recall it having any major load-balancing capacity. Its primary purpose is to funnel HTTP requests from Apache to Tomcat (or JBoss), and there are some who assert that actually a straight proxy is better for that purpose.

To get a properly balanced cluster workload, you need to be able to cycle requests among multiple servers. Ideally, by session, so that the servers don't have to pass session data back and forth. If the load is statistically even, a round-robin scheme will do, but in cases where there are irregularities, you might need something more advanced - either by routing heavy-duty requests between a set of servers better able to handle them or by having the backend servers pass load information up to the front-end so that it can adjust the scheduling algorithms. That would require software that's not (as far as I know) part of the standard kit - both to do the feedback and to do the enhanced load balancing.

Then there's yet another possibility. You might also place a hardware load balancing device in front of the apache servers. That works best when the backend loading is even.

[Josh White]

Tim, sent you a private msg to follow up. Thanks!!!