load test

[Migrated From Jforum.net]

hi rafael;
i was playing with jmeter and put 2 threads in a loop requesting the main forum page over and over again.
after a short while i kept getting the exception below.
that test was run on 2.0.2 so this might have already been fixed.
do you know anything about this issue? looks like a collection is being modified while iterating over.
thanks.

the stack trace:
INFO: Server startup in 3406 ms
set from - User
17:16:45,290 ERROR [ForumException ] java.lang.reflect.InvocationTargetExce
ption
at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at net.jforum.Command.process(Command.java:88)
at net.jforum.JForum.service(JForum.java:262)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:856)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ssConnection(Http11Protocol.java:744)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpo
int.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFol
lowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadP
ool.java:684)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.ja
va:358)
at java.util.LinkedHashMap$ValueIterator.next(LinkedHashMap.java:373)
at java.util.AbstractCollection.toArray(AbstractCollection.java:174)
at java.util.ArrayList.<init>(ArrayList.java:136)
at net.jforum.SessionFacade.getAllSessions(SessionFacade.java:129)
at net.jforum.view.forum.ForumAction.list(ForumAction.java:202)
... 20 more

[originally posted on jforum.net by gkatz]

[Migrated From Jforum.net]

ConcurrentModificationException are expected to occur rarely. Can you please sent the jmeter test plan to my email? I've been trying to use it, but I don't know exactly how to interpret the result graphics and etc.. do you know it? If so, I'd appreciate some explanation, since the documentation of jmeter doesn't help

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Hello there,

I like much more TheGrinder as a stress test tool, as it is not so generalistic and I found the UI much more intuitive. But it's just a personal preference ;)

Regarding the tests, the basic initial idea would be to run a couple of simulatenous threads that stress the application to detect any "race conditions" that might be caused by lack of synchronization. A ConcurrentModificationException would be one of the symptoms that might tell you something needs to be synchronised when it's not.

Irregardless of the response time, these tests should come out clean with no errors.

After that one might decide to "check" how much workload can a web application sustain and see if it degrades gracefully or not when the load increases.

They come also quite handy when used along a "profiling tool" to detect memory leaks or excessive "memory trashing", as you can stress test the application while seeing the internal behaviour regarding memory consumption, object creation etc... Sometimes the problem is that the profiling tool interferes with the stress test, so you have take that into account.

This last part might not be fundamental for every web application, if real usage is not going to be too high, but the "synchronisation check" is very important as even low real usage would cause problems, sooner or later, if things are not properly synchronised.

Cheers!
[originally posted on jforum.net by GreenEyed]

[Migrated From Jforum.net]

As a side note and given the error...

The problem seems to arise when manipulating the "cache" static member inside the class SessionFacade, and having a look at the code one can see that:
.- The member "cache" is static
.- There are several static methods that modify the content of the "cache" object
.- There are no synchronisation mechanisms to prevent race conditions

So I would say the class is not thread safe, hence prone to problems when accessed simultaneously. I would recommend synchronising access to all shared members, as web applications are most likely going to be accessed simultaneously.

Cheers!



[originally posted on jforum.net by GreenEyed]

[Migrated From Jforum.net]

hi rafael;
well, i just started using it so i dont really know the trenches. but its by far the coolest thing around - i cant believe i havnt used it already.
i will send you the test plan, it works fine with 1 thread but with 2 thread the exceptions are thrown quickly.
again, i am using 2.0.2 so this might have already been fixed.
please respond when you have some thoughts about this.
Guy.
[originally posted on jforum.net by gkatz]

[Migrated From Jforum.net]

by the way;
i am using jmeter 2.0.3 with a table result view (not a graph) - i figured it will take less resources.
also, you should have jmeter run on machine A checking a jforum application on machine B.
thanks.
[originally posted on jforum.net by gkatz]

[Migrated From Jforum.net]

I intentionally made it without sync blocks. I was expecting to get something like 95% of success (aka, no concurrent modifications). I know some quite busy sites running jforum that run fine, with concurrent modifications ocurring "rarely".

What I'm afraid of is to loose a reasonable performance with "incorrect" made sync blocks. As almost all access are read, there is no need to sync getUserSession() for example. The problem arises when some thread is iterating over the collection and another thread removes / adds an entry.

As of java 1.4, there is no good classes to work with threading - I mean, the available ones are much worst than the concurrent package from java 5. The best approach, in my opinion, is to use doug lea's concurrent classes, which will solve almost concerns.

But I may be wrong. What do you guys think?

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Hey there,

On a side note, I would not recommend using ReadWriteLock for the "cache" issue or similar ones because, as the API of class mentions, it is suitable if:

...The methods are relatively time-consuming (as a rough rule of thumb, exceed more than a hundred instructions), so it pays to introduce a bit more overhead associated with ReadWrite locks compared to simple synchronized methods etc in order to allow concurrency among reader threads.

and this is not the case. In the "cache" example the reads are very short (just returning an object from a Map, mostly) so the contention between reading threads would be less than the overhead that ReadWriteLock introduces to check whether it needs to lock the access or not. Not saying that the library is not useful, but just that in this specific case I think we might loose more than what we win.

Regarding your concerns of having to synchronise everything multiple times, I agree with you, Rafael. The key point here is that you have to analise the code and find the "shared resources", which are the ones whose access needs to be synchronised when a race condition might ocurr. On the other hand, there are many instances of data that you don't need to worry about, as they won't be shared across requests, so you don't need to sync. access to them.

As a rule of a thumb, objects that are static, or are stored in the servlet context, or in the servlet session are usually shared, so if they are going to be read & modified, they need to be controlled. On the other hand, method members, local variables etc belong to one thread, so there's usually no need to synchronise access to those ones.

In the end, there are usually just a couple of objects that are shared (like the cache of logged in users) that are shared among requests and most of the others are thread-safe on their own. Data inside the database is another story, dealt differently through transactions etc...

But as I said, this might not be too important for an application like a forum, but I think it would be a good idea for you to introduce it so you can learn to deal with it, preparing yourself for the day your applications need to be 100% safe in this regard. ;)

Cheers!
[originally posted on jforum.net by GreenEyed]

[Migrated From Jforum.net]

Well,

A forum application is most of the times a non-critical application so a random error every now and then might be considered ok. However, in my case I alway prefer to err on the safe side, as I consider a 95%-off-the-times correct application is still not correct. :roll:

Regarding performance, it is true that synchronization comes with a price, everything has one, but if you do it correctly it should not be "too much". It might sound "too much" but if you have mostly reading operations and one writing operation, you still need to synchronise the access for the reading operations, to guarantee your program is thread safe. On the other hand, the milliseconds you might loose for a couple of synchronization blocks will be negligible when compared to the time spent retrieving the data from the database, sending the data across the network or rendering the HTML... so the net gain versus the drawbacks are IMHO, not worthy.

And I haven't found many issues using the synchronization mechanisms available in JDK1.4, no need to deal with threads as you just need to synchronise access, so what are the problems you have encountered?

For example, in this case, given that the problem is accesing the "cache" object, you could simply define a lockObject and use synchronize(lockObject){...} to wrap any code that accesses the object non-atomically (like iterating over the collection of values) or modifies the cache content(add, remove...) As the cache object is just an interface, I would also synchronise the simple reading operations, unless you make it mandatory that they have to be performed atomically.

The problem with not having it synchronised, composed with not having foreign keys, is that under stress, or with pure bad luck, you might end up with a corrupt database (no foreign keys to protect it and no thread-safe code to protect it either).

But as I say, it's a matter of "preference", where the application is non-critical. To me, safer applications mean less maintenance headaches so... ;)

My 2ec
[originally posted on jforum.net by GreenEyed]

[Migrated From Jforum.net]

hi guys;
i have ben working with doug's package for a while in other applications.
there are locks that confirm to a read mostly scenario (which from my understanding this is what you want rafael). see:
http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/ReadWriteLock.html
plus it will be easy to port to 5.0 when you want. you will just need to change the package name...
[originally posted on jforum.net by gkatz]

[Migrated From Jforum.net]

Hey gkatz,

that's what I thought too. Also, there is a backport for the java.util.concurrent made by Dawid Kurzyniec:

http://www.mathcs.emory.edu/dcl/util/backport-util-concurrent/

Using it may be the best way to go, since changing to j2se 5.0 later will be very easy.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

GreenEyed wrote:Well, [....] My 2ec

I agree with your points. The main "problem" I was / am afraid is to have to sync the access, everytime, with no exceptions, to data that is accessed many times in the same request, by different users, impacting performance hardly.

I did not do any tests anyway, was just a supposition. I mentioned doug's concurrent package because the locking mechanism is more efficient.

I don't know if I can put it until 2.1.5, but I'm with you that the current code is quite sloppy in this aspect.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

I'm doing some tests using JMeter and a test servlet. More precisely, testing ReadWriteLock (WriterPreferenceReadWriteLock), ConcurrentHashMap e regular synchronized() blocks..

Test tests consist in basicall having one action putting data into the map, and another action iterating over it, as in

public void addToConcurrent()
{
	concurrentMao.put("key" + System.currentTimeMillis(), new Integer(1));
}

public void listConcurrent()
{
	List l = new ArrayList();
	
	for (Iterator iter = concurrentMap.values().iterator(); iter.hasNext(); ) {
		l.add("ee" + iter.next());
	}
}

(this is for ConcurrentHashMap). The sync()'ed version is like

synchronized (syncMap) {
	syncMap.put("key" + System.currentTimeMillis(), new Integer(1));
}

and the jmeter test plan starts 5 threads, with 1000 iterations, calling the addToXxx and listXxx in a random fashion.

All I can say so far is that it's hard to say which is better. In general, the sync()'ed version was a bit worst than using the concurrent package, but sometimes it get much close. But I'm not that good with jmeter, so consider that it's possible that my interpretation of the results are wrong ;)

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

I would say that unless you get spectacular differences, the slight advantage you would get from one method or the other would be seldom seen and even in that case it would be "dilluted" among the bigger delays that are retrieven data and sending it through the network.
I mean, you might end up saving ocasionally 20-30 milliseconds, being very generous, in the context of a request that for your user might take between 1-2 secs... so... :roll:

I would opt for the most simple approach, but that's just me ;)
[originally posted on jforum.net by GreenEyed]

[Migrated From Jforum.net]

hi;
i must say first that i did not take a look at the jforum code.
however, you might want to consider ConcurrentReaderHashMap without synchronizing any code.

the specific problem of the concurrent modification will not exist as this class returns a fail safe iterator, so if you dont really care that the cache was modified while already iterating than its perfect.
also, as the class name implies its optimized for readers and the docs specifically state that its excelent in cases where you read to a cache on startup, and mostly read from it.

thanks.
[originally posted on jforum.net by gkatz]