• Post Reply Bookmark Topic Watch Topic
  • New Topic

Unexplicable performance deviations  RSS feed

 
Sven Richter
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

We have an OSGi product running in an OSGi container (obviously). For this we do some performance tests and one of them is to call one of the services (with some wrappers from other OSGI services inbetween) and measure its average response time.
The problem is that these response times will have a deviation of up to 400%, depending on the restart of the complete OSGi container.
So whenever we restart the container we will have different average response times, but, they will be stable as long as the container is not restarted.

What we did so far to hunt down this bug is:
- Restart only our OSGi product inside the container - made no difference
- Try to reproduce it with a different product in the OSGi container- no success
- track GC timings - hardly a difference between the runs (FGCT was around 16s during a run of 5 - 20 minutes)
- run the tests under full load versus only one cpu core - made no difference
- tried to track down different usage of libraries, nothing we found, but we are not sure if our method is sufficient (using OSGi diagnostics)
- Profiling CPU and Memory of a fast / slow run - nothing found, but again we are no profiling experts and may have missed something here
- there is no memory leak inside our application (at least not during the test runs)
- different test system setups (in one setup the OSGi container was the only "active" process running on the machine)
- and maybe some more that I don't remember right now

What we are looking for are some tips what other ways there might be to find this bug. We are open for every suggestion that one might think of.

Thanks,
Sven
 
Jayesh A Lalwani
Rancher
Posts: 2762
32
Eclipse IDE Spring Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Possibly your application is going through some sort of initialization at startup. Classes need to be loaded, objects initialized, database connected, etc.

I would try to narrow down which part of the application is behaving slow. If you already have good logging, look at your application logs. If you don't have good logging, add logs and then look at your application logs.
 
Sven Richter
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

Changing the min heap space did not improve anything.

However, I was able to track down a few lines that cause the deviation:



If I cache the return value of this method in a hashmap the deviation goes down to 10 percent from 400 percent. Now as this may seem as a dirty hack it might be valid as a solution, but still, I don't understand why this is happening.

To make things clear, the code above deviates up to 400 percent between restarts of the OSGi environment. So, does anyone have an idea or experience when reflection is slower or faster depending on some environmental circumstances?

Thanks,
Sven
 
Jayesh A Lalwani
Rancher
Posts: 2762
32
Eclipse IDE Spring Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Reflection is slow compared to direct calls. However, reflection shouldn't introduce significant delays unless you are doing it lots of times. For example, reflection is a lot faster than opening a database connection. Also, in J2SE1.4, there were significant improvements made in Class.newInstance and Method.invoke, and they should be relatively very quick. So, if you are introspecting a class every time you want to call a method, then you might run into performance problems. However, if you aready have a Method object, calling invoke on it many many times shouldn't make your application too slow.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!