I have one question regarding the performance tuning of the weblogic portal application (or may be any J2EE application for that matter) :
I understand that there could be lot of performance bottleneck areas e.g (a) network issues (not enough bandwidth), (b) bottleneck because of the DBs, (c) application design/coding (d) JVM might not have been tuned as per the business requirement of the application (e) lack of clustering (f) hardware problems (CPUs, RAMs etc)
Inorder to do the capacity planning for such an enterprise applications, what would be the best starting point?
Is it a good idea to start and profile the application using some profiling tools like OptimizeIt or JProbe? and then analyze the profiling the data and conclusion can be drawn interms of what and where the major bottleneck is into the application and accordingly it can be worked upon.
Or if there is any better and elegant way doing capacity planning?
I think Performance Tuning and Capacity planning are different things. Capacity planning - decide the hardware and software configuration required for specific application load (specific TPS) Performance Tuning - Tune the application/software/hardware to optimize the TPS for a specific hardware/software configuration. Typically we will be doing the performance tuning as part of application development. Capacity planning before deploying the application for any customer. However we may need to do further performance tuning after capacity planning. Performance tuning: First pass: 1) Do a load testing for 15 minutes. 2) Monitor the different parameters a) CPU b) Time spent in application vs time spend in Database (or any other external system) c) Garbage collection (time spend in GC vs total time) d) Take multiple thread dump and look for threads waiting for monitor entry (you may use samurai thread dump analysis) e) network i/o rate f) if total commit charge exceeds the physical memory Analysis: if more time spend on DB tune DB if lot of threads in waiting for monitor entry, tune code to remove synchronization blocs if lot of time spend on GC, try increase/decrease heap, multiple JVMs if total commit charge exceeds physical memory decrease heap size if network i/o is close to i/o capacity, use compression to reduce i/o Second pass: Run the application with CPU profiler Find the hotspots. Analyze the code for optimization. Capacity planning: Run load testing for 15 minutes with application on standard machine and database other dependencies in higher machines measure the TPS Find TPS per CPU network i/o per transaction (typically this may not be a big issue) Calculate the number of CPUs required Decide number of machines (m number of n CPU machines) Load testing: 1) Find the key transaction (end user input) 2) Find the ratio of key transactions (end user input) 3) Write load testing stubs (using some load testing tool) 4) Increase the number of threads until you reach CPU usage of application to 80+% DB tuning (oracle 9.2) take snaps before and after load testing take statspack report look at top 5 timed events. Tune DB to reduce item which is taking more time. Typically increase db buffer size, increase pga aggregate etc Find the Sqls taking more time. Find the explain plan for these SQLs Tune these SQLs to have better plan. Typically proper indexing will do.
WOW! Rejeev Divakaran said it better than I think I have ever read it before, in such a great summarized form.
I will offer a few things.. 1. Find an Expert If this is mission critical, it may be worth it to find someone who has had experience in performance tuning with your environment/servers. Relationships with local consulting companies are great resources. They want to get in and be able to help you, and usually the more one does this type of work, the better they are at it.
2. The Top 5 Like a letterman show.. usually the Top 5 performance bottlenecks may be responsible for 90% of your performance issues. We don't have JProbe, or Optimize it, but there are some good opens source/free-ish tools, with Netbeans, and Eclipse, which is a good start if you have nothing.
3. Allow Time All too often optimization, performance tunning, capacity planning are done at the end of project life cycle. Try to integrate them into the development cycle. Ant/Cruise control and a number of open source projects has the means to assist you in this effort. Surprises at the end of a project are never kind. Especially the ones that require capital and resources. Those don't usually come quickly as well.
4. Be cautious of publicized capacities Remember, application servers have a goal of outdoing each other. Yes there are some independent benchmarks but you must test your application before you have a true graps on the capacity of your application/infrastructures. I have had to come up is scales of formulas many times, because I have been forced to come up with numbers/resources, even before the project is completed and maybe not the same hardware, but this is always a key point.
5. Underestimate / (or Over estimate - however you think of it) Everyone loves being surprised to handle more requests, faster than expected. No one is happy with a new system does less. That's a bad feeling having to either further optimize or explain why.
I like Rejeev Divakaran, post better than mine.
Tony Sun Certified Web Business Component Developer Sun Certified Web Components Developer Sun Certified Programmer for the Java 2 Platform
Tony McClay<br />Architect / Developer, SOA and Jave Enterprise Edition 1-5<br />---------------------------------------------------------------------- <br />Sun Certified Enterprise Architect, Enterprise Edition 5 (Step 1 of 3)<br />Sun Certified Web Component Developer, Enterprise Edition 4<br />Sun Certified Business Component Developer Enterprise Edition 5<br />Sun Certified Programmer , Standard Edition 5.0
Thanks Tony for the compliment. I have recently tried YourKit profiler and I found it very useful. It is much less expensive than JProbe and OptimizeIt (By the way OptimizeIt is retired). It doesn't support method tracing with JDK 1.4.x Anybody knows any tool (referably free)supports method tracing with JDK 1.4?