Any Insight on how JavaRanch scales

[m lathe]

Hi Guys,
I was just wondering if the JavaRanch team could provide some insight to how JForum is deployed to scale on this site.
You appear to be the largest implementation of JForum so some pointers would be appreciated.

The gist from what i understand on the JForum site is that it doesn't scale horizontally (ie you can't distribute one instance across several boxes)
http://jforum.net/posts/list/4898.page
http://jforum.net/posts/list/2553.page

Rafael basically says that it scales on one box and the best we could is run a cold backup connected to the common DB. Nor does it look like offloading reads to a replicated DB are allowed.

One solution that is posed is that in theory connecting several instances with a common cache (ie ehcache with multicast messages to advertise dirty caches) should work. However this doesn't appear to be supported.

Any information would be helpful.
Thanks
--Matthias

[paul wheaton]

Others will be able to provide more accurate data, but I think we are currently using about 5% of our CPU and we are serving about 6.5 million page views per month. And this is with caching turned off.

I've been thinking that if there was a need to run on more than one box, it would be wise to divide the forums: some forums would be on one box and the rest on a second box. But I don't see that happening.

[m lathe]

Thanks for the reply.

So the application runs on one box (presumably commodity hardware) and the DB on another box (presumably MySQL on commodity hardware)?

Do you run a cold/warm standby?

We have also considered running parallel instances, where some boards are hosted on one silo of boxes (ie app and DB) and others are hosted on another silo. This is in effect just a "poor man's" sharding. Did you consider what would happen for shared resources (notably post counts etc?) would you just sync this in the background somehow?

Thanks again
--Matthias

[paul wheaton]

Who are you making forums for that you would need something bigger than what we have here?

[m lathe]

It's for a customer website for a company. You're probably correct that the load will not be as large as what is found here. However I think it's more of a question of how do you build in redundancy so you can roll and test changes without bringing down the site and also how to deal with crashes without having to start up a cold backup.

I guess the lesson here is that even the big installs of JForum only deploy one instance. No one seems to attempt to cluster several instances with a cache.

Thanks

[Ernest Friedman-Hill]

We do have a completely separate install that we use as a test server, so after developers test locally, we deploy to the test server and bang on that. But I'll agree that JForum isn't really designed for five-nines operation.

[Jeanne Boyarsky]

m lathe wrote: Did you consider what would happen for shared resources (notably post counts etc?) would you just sync this in the background somehow?

I would imagine you would still have one database and just multiple app servers.

m lathe wrote:However I think it's more of a question of how do you build in redundancy so you can roll and test changes without bringing down the site

Who tests changes in production?

m lathe wrote: also how to deal with crashes without having to start up a cold backup.

The software doesn't crash. The only time we are down is for deployments. Which in a real company would be scheduled. If you mean the box crashing, you can have the software on another box and route the web server to it. Of course, starting up another instance of Tomcat doesn't take long.

[m lathe]

Jeanne, your comments are apt. A suitably fast DB (worst case Oracle RAC) would probably allow allow you to run multiple app servers. That being said there might be some issues with how JForum writes to the DB that count on only one app running. One example might be to queue writes in the app layer and periodically flush them to the DB (ie view counts). I'm not sure if it does this, but it might.

Jeanne Boyarsky wrote:
Who tests changes in production?

Everyone should test in production, but you are right it shouldn't be the first time you test the change. If you are trying to get 100% uptime you really don't want to release a product to your customer until it was verified in production. So while you are doing functional testing in the test environments, you don't want to just slap a war into production and walk away. Ideally you have two nodes, you take one away from the customer, roll your change, verify, add it back and do the same to the second node.

Thanks everyone.

Please holler if you have any pointers for how to cluster JForum.

[Jeanne Boyarsky]

You are correct that JForum caches things in a way that prevents clustering. I meant have an app server upp that nobody can access just in case something happens to the first one.

I don't see a forum as being so mission critical though. I work for a bank and we are still allowed X hours of downtime a year.

[Ulf Dittmer]

JForum supports several cache implementations, though, one of them being EhCache, which is billed as a "distributed cache". So you should be able to configure it to handle caches in multiple JVMs: http://ehcache.org/documentation/distributed_caching.html

That is probably easier to set up -and, IMO, architecturally cleaner- than having separate servers handle separate forums (which by itself wouldn't solve all caching issues).