clustering

[Jun Hong]

I have a question about horizontal clustering. If the I have a Solaris 2.8 machine and I am using WAS 4.0, how many clones can I have?
The RAM on that machines is 5000MB and 4 CPUs. Right now 50% CPUs are used. If you gave me a suggested number, say 3 application servers, please tell me the setting for Heap on each JVM.
Right now I am not doing clustering. My project is experiencing performance problem. I have C++ client -> CORBA -> Java client - > stateless EJB -> Oracle thin driver -> Oracle. If I sent a request for 2 years data which is about 500 rows, it takes 15 seconds. If I sent 10 requests at the same time, it takes 80 seconds for all of them to be done. Something is wrong, the application server is not handling them in parallel processes. It is almost sequential. I set the ORB thread pool size to 20. The database pool has a min 10 and max of 30. By the way, the CORBA part is not a bottle neck. I have 10 Corba servers running at the same time.
By the way, on the console where can I set the stateless session pool size. I see Cache size = 2047 somewhere. Is this stateless session bean pool?
Any suggestion is wellcome.
[ February 02, 2002: Message edited by: Jun Hong ]

[Kyle Brown]

Vertical clustering (adding more JVM's on a multi-CPU box) rarely helps much if the box has fewer than 6 CPU's. If you start a second clone you might find a slight improvement in response times. However, if you start many more you'll probably see that improvement tail off, and then you'll see response times start getting worse...
By the way, from your description, it sounds like the bottleneck might be in your code...
Kyle
[ February 03, 2002: Message edited by: Kyle Brown ]

[Jun Hong]

Kely,
Believe it or not. I put 7 application server clones on that machine and it works fine. The performance is considerable improved. The use of CPU is about 85% and I still have 1000~2000MB left. Each clone is using about 380MB. Now if I send 10 requests, they will finished in 20 seconds. If I query 250 stocks, 2 years data, and send 10 of them together it takes 10 mins for all of them to be done. Yesterday, it took 3 hours. I am safe for next week. More improvement is expected later.
What you said is probably right. I am quite suspicious that our project has a bottle neck which cannot be handled by application server. Since I add more servers the bottle neck is gone. I already fix one bottle neck. I found all clients log to the same file using log4j. I turn logging off. I have to figure out how to handle this later.
I read from 3.5 documents that 4 CPUs is the limit. Somebody did a test and find cluster only helps when you have more than four CPUs. So for 4.0, it is 6 CPUs?
Anyway, we ordered two more CPUs and 5000MB RAM. This is an expensive project, we cannot fail this one. That 5000MB RAM cost us $40,000 (looks like it is not needed anymore). How come Sun machine's RAM is so expensive? Also, I will let Unix people adjust the ethernet settings.
Thanks for your imformation

[Kyle Brown]

Jun,
I'm glad that you found that vertical scaling helped your project. I'm a bit surprised that you saw improvements all the way up to 7 JVM's -- I would have expected the tailoff in performance to have happened a bit earlier -- it sounds like you really DO have a bottleneck in your code...
The thing is that the single App Server process is supposed to be able to take up all of the CPU time on all of the processors, even with a single JVM (e.g. each thread goes on a different processor). Since you're seeing improvements by just adding JVM's, it means the bottleneck is in the code -- all of the threads in the process are waiting for SOMETHING -- adding more processes just means that that means fewer requests in total are waiting for that same something...
Anyway, the numbers on improvements really start looking good at 6 to 8 CPUs, but we've run tests on all the way up to 24 processors. However, since every application is different, your mileage may vary...
Kyle

[Kyle Brown]

By the way -- did you look for the obvious bottlenecks in Harvey Gunther's wonderful paper on performance tuning? For instannce, if this code isn't using Connection pooling, that could be a problem... Likewise if they are all synchronizing on an instance variable in a servlet that could be another problem that could cause these results...
Kyle

[Jun Hong]

Hi Kyle,
Thanks for your reply. I read that article you mentioned. It is a good one an I like it.
Again,
Thanks