• Post Reply Bookmark Topic Watch Topic
  • New Topic

Does JMeter really work?

 
Bruno Dery
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all!

I have two load testing applications I'm trying on my web server, JMeter and another one which is called Web Applications Testing (WAPT) from http://www.loadtestingtool.com/. Now the thing is I have noticed several weird results under JMeter which I have a hard time explaining, and it seems the other tool does not have the same results.

First off, I do a simple test, loading my home page (index.jsp) with 10 concurrent users and have 10 seconds of delay between each iteration. The first weird thing is that I noticed (this is very obvious viewing the graph results) that the responses are slow at first and keep improving over time. Like the first 100 samples are about 2 to 3 times slower than after 1000 samples for example. So at first the page takes around 10 seconds and eventually (around the 1000th sample) takes 3 seconds roughly.

Now I thought maybe this is a behavior of the server somehow, but using the other tool (a windows app) I get very consistent results, around 2.5 seconds for loading my page.

So my question, why does JMeter starts slow like that and improve over time?? Obviously this is not due to my application as proved by the other tool... I run JMeter with the xms to 256 and xmx384, but anyways for only 10 users I would not expect it to use too much resources. I have no other resource eating result either... Any clues on what's happening here??
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Bruno Dery:
First off, I do a simple test, loading my home page (index.jsp) with 10 concurrent users and have 10 seconds of delay between each iteration. The first weird thing is that I noticed (this is very obvious viewing the graph results) that the responses are slow at first and keep improving over time. Like the first 100 samples are about 2 to 3 times slower than after 1000 samples for example. So at first the page takes around 10 seconds and eventually (around the 1000th sample) takes 3 seconds roughly.


That's not surprising at all. At first access, the JSP needs to be compiled into a servlet. Then that servlet gets executed in interpreted mode, until the hotspot engine decides how to optimize the code and compiles it to machine code.

Generally, it's quite normal that Java code gets orders of magnitudes faster the more often it is executed (inside the same VM). Especially if you use the -server option.

Now I thought maybe this is a behavior of the server somehow, but using the other tool (a windows app) I get very consistent results, around 2.5 seconds for loading my page.


Did you restart the server before you tried this? If not, the JSP will already be compiled and optimized, so won't see the initial performance drop down again. (To see the full performance hit, you might even have to change the JSP, to force a recompilation.)

Does that help?
 
Bruno Dery
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
No all the jsp's are supposed to be compiled, the application has been sitting on the server for a while and things should be stable and optimized I believe. Still, I get the same exact behavior from JMeter every time I run the tests.

I did some further testing on this friday, and results are surprising. JMeter posts the same curve in results (kind of like a reversed logarythmic function) but using my other tool (a windows based app) results are stable. Furthermore, JMeter can't handle that much threads before starting to act weird (gui doesnt refresh, errors seem to happen in the background) while the other application has no problems simulating up to 300 hundred users on the same machine (although our office connection problably is a bottleneck in that case but I ran this just for comparison sake).

Anybody else obversed similar results? It's kind of difficult to sell the virtues of load testing to managment with a tool that doesnt seem to post accurate and stable results...
 
Bruno Dery
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Any other thoughts on this guys?

I'll post my own findings, which is basicly I gave up on JMeter mostly because of the incorrect results I was getting, and also because of the general 'slowness' of the application itself, not to mention some gui problems (refreshing gets hecktic after running some tests, I have to manually scroll the splitpane around to make it refresh), etc.

We have used WAPT (web applications testing from http://www.loadtestingtool.com/ ) instead and it seems to fill in all our needs, and runs fine with simulating up to 300 hundred users (that's the most I've tried on my system, but I doubt that our network connection here can handle that much traffic anyways so it was mostly to stress test the tool itself). It also supports some basic scripting so we were able to simulate different user logins, etc.

Anyhow, I'm still curious as to what people think of this...
 
Paul Sturrock
Bartender
Posts: 10336
Eclipse IDE Hibernate Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
JMeter is OK as far as generating multi-user stress, load or performance testing. The results tend to tally with other tools I've used so I have no reason to believe its not accurate. Anedotal evidence on the web and in books suggests this too.

However, the GUI it comes with is not good. There are a bunch of documented bugs against it listed on JMeter's site, and you need to be aware of the problemes this might cause when your tests become very complex or the reports you use get a bit too numerous.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!