Memory Issues

[Migrated From Jforum.net]

Under heavy load, jforum seems to really be consuming some memory most likely due to java Reflections and dynamic class loading. Rafael, with the normal jvm 1.4.2 (no startup params) and running tomcat 5, the FULL GC seems to really be busy at times halting the entire web server for up to 3 seconds. I would highly recommend that you do some major memory cleanup somehow to make it easier on the garbage collector. I also tried some of the new concurrent jvm parameters like -XX:+UseConcMarkSweepGC -Xmx1000m -Xms1000m to make the jvm run on seperate threads with up to 1 gig of max memory. This seemed to work fine until my entire site came to a screaming halt after it ran out of memory. I have major traffic pooring into jforum so the ability to test the memory.

I found an instance of System.gc() in jforum.java. Anytime you have to use an explicit call to the gc, you know there is potential for a memory leak.

Please, if possible, check this out as it would be nice to have these memory issues taken care of in the next release.
[originally posted on jforum.net by Anonymous]

[Migrated From Jforum.net]

System.gc is called from PooledConnection.java, not JForum.java. I already removed it from there, in the development version.

About reflection, I guess there isn't much to do about.

I agree with you that there may be a memory leak, but I never was able to realize the source of the problem.

Any clues about how to try to solve it are very welcome (how to generate high traffic and etc.)

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Tomcat 5 basically blocks all incoming requests as it tries to do cleanup on tenured memory. Depending on traffic, this can stop the web server for minutes at a time.

It does a Full GC producing the following

[Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccesso
6]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor121]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor81]
[Unloading class sun.reflect.GeneratedMethodAccessor285]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor99]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor115]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor108]
[Unloading class sun.reflect.GeneratedMethodAccessor346]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor123]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor120]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor105]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor114]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor127]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor82]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor125]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor110]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor111]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor109]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor119]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor124]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor91]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor106]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor90]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor117]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor118]
[originally posted on jforum.net by Anonymous]

[Migrated From Jforum.net]

I'm able to reproduce the OOM "quickly" with jmeter.

The first thing that I will check is how many memory the session objects are using. Basically speaking, I have an UserSession instance for each user, including anonymous, and, theorically, the entries are only removed when its session expires (when <session-timeout> runs).

So, if the product of the size of all UserSession instances and the number of active sessions grow too much, you will run out of memory.

But let's suppose that there is not leak, and consider that the OOM is being caused because there are too many active sessions and not enough memory to hold them all. What are us suposed to do then? Probably JForum can try to use less in-memory data (which will not be that easy).. but, besides that, what else?

Rafael
(ps: please register for an account, so we can know to who we're talking to )

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Ok, looks like the bug it's not the session size.

Let's go ahead.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Hi Rafael, sorry I didn't log in.

This is Brakker (Chad). I talked to you previously about a few things in the bugs forum.

I've been trying to trace down the issue by watching the behavior. Doesn't seem like sessions running out of memory....seems like a class unloading problem in a particular area of the code that only gets hit so at certain times. For instance, the server came down recently and I restarted it...within 5 minutes the gc brought it down again taking too long to clean up whatever was in memory. Sometimes it happens right away and other times it will take hours with the same traffic. I currently have the max heap size set to 1.5 gig so that should be plenty...there are no more than 80 current sessions logged into the forums at any given time. When i watch the heap, it will be at a constant 400meg and then, all the sudden it will start climbing rapidly.

What do you think the bug is at this point?

Thanks for looking into this so fast as it help me out. Maybe once we get this solved, I can give you a nice testimonial as I am running a pretty big site.

Chad


[originally posted on jforum.net by coolbreeze]

[Migrated From Jforum.net]

Although I haven't been able to prove it yet (logging is off), I believe this issue could be getting caused by major bot activity. I have a lot of search engine bots that like to spider my site producing 1-100 hits a second. This could be what is causing the quick rise in memory and tomcat/jvm deadlock.

Brakker
[originally posted on jforum.net by coolbreeze]

[Migrated From Jforum.net]

I'm doing some tests. At least I'm able to reproduce the OOM with jmeter. Also, I noted that, when using Sun's VM, the OOM is raised much faster than when using BEA's JRockit (both java 5).

I'm going to try their Memory leak detector (http://e-docs.bea.com/wljrockit/docs50/usingMMLeak/index.html) and see what I get.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

brakker wrote:Although I haven't been able to prove it yet (logging is off), I believe this issue could be getting caused by major bot activity. I have a lot of search engine bots that like to spider my site producing 1-100 hits a second. This could be what is causing the quick rise in memory and tomcat/jvm deadlock.

Brakker

Development version 2.1.5 has a servlet filter that logs bots. Althouth it will not prevent them for indexing, you will know when they're comming.

Also, using the bot discovery classes, we can extend the feature to completely "disable" bots (aka, we can send a 403 forbidden, or something like that).

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Ok, my first shot:

the PooledConnection class.

I ran JForum for a long time using SimpleConnection, and the memory usage was stable. However, when I move to PooledConnection, it increases really fast.

Anyway, it's just a guess. Maybe the bug is not there.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Interesting.

I actually have been using the SimpleConnection...trying to keep things as "default" as possible.

I think the only way to detect what is going on is to use a memory profiler like jprofiler or something.

JMeter seems to be worthless as a load tester. I tried using it on my app and couldn't detect anything...as soon as I put my app in production, bam, the problem would come back.

Brakker
[originally posted on jforum.net by coolbreeze]

[Migrated From Jforum.net]

hhhm... so you're using SimpleConnection..

I'm able to force an OOM using PooledConnection, but not with SimpleConnection.

well, I'll keep trying.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Ok Rafael, I am doing some investigation on the 1.4.x+ jvm.

Using the following startup jvm options with tomcat:

-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC

We can get the class unloading to happen on multiple threads which may speed things up quite a bit and not bring down the server do to garbage collection. It seems as though the issue has to do with class unloading. I can tell this from doing a -verbose:gc as another startup jvm option. You may want to test this yourself by setting the verbose and then running jmeter.

I am going to test this in production shortly...let's keep our fingers crossed.

Brakker
[originally posted on jforum.net by coolbreeze]

[Migrated From Jforum.net]

Sorry, I am see the all class unloading using jprofile, not the verbose:gc option.

Brakker
[originally posted on jforum.net by coolbreeze]

[Migrated From Jforum.net]

Nice.

Another point: I was testing about 7 urls, and found that the OOM is being thrown in the post listing page. I say that because I run one stress test per url, and only the post listing raised OOM.

I'm going deeper now to confirm that.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Is there a way to we know which classes are being unloaded, or something like that? There is a bit of reflection here and there, but not "a lot".
I'm still learning JProfiler, and verbose:gc does not help too much.

Any guidances are very welcome.

ps: do you think that using a library like cglib to do reflection will help?

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Well, I found the memory leak. I just don't know yet how to fix it .

Using the latest cvs head code, in lines 140 and 141 of net/jforum/view/forum/PostAction.java the code

Map usersMap = new HashMap();
List helperList = PostCommon.topicPosts(pm, um, usersMap, canEdit, us.getUserId(), topic.getId(), start, count);

here, the interesting part is the usersMap map. It is used to hold all unique users who have posted in the topic. JForum does that for caching purposes, to not have to go to the database for each user entry.

The method topicPost() then make a call to

PostCommon.addToTopicPosters(p.getUserId(), usersMap, um);

which then follows to

public static void addToTopicPosters(int userId, Map usersMap, UserDAO um) throws Exception
{
	Integer posterId = new Integer(userId);
	if (!usersMap.containsKey(posterId)) {
		usersMap.put(posterId, PostCommon.getUserForDisplay(userId));
	}
}

public static User getUserForDisplay(int userId) throws Exception
{
	User u = DataAccessDriver.getInstance().newUserDAO().selectById(userId);
	
	if (u.getSignature() != null) {
		u.setSignature(u.getSignature().replaceAll("\n", "<br>"));
		u.setSignature(PostCommon.processText(u.getSignature()));
		u.setSignature(PostCommon.processSmilies(u.getSignature(), SmiliesRepository.getSmilies()));
	}
	
	return u;
}

now the catch: if I comment the lines

if (!usersMap.containsKey(posterId)) {
	usersMap.put(posterId, PostCommon.getUserForDisplay(userId));
}

(from addToTopicPosters() ), the OOM will not happen.

I don't know yet why it happens, because it's not clear who is keeping the data alive.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Ok, bug fixed

Here's the diff:

https://jforum.dev.java.net/source/browse/jforum/src/net/jforum/dao/generic/GenericUserDAO.java?r1=1.5&r2=1.6

Yes: that simple thing was the root of all problems.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Excellent.

I will put this into production today and see what kind of difference it will make. I may not be able to tell until Moday as traffic slows down to about half for me on the weekends.

I will let you know.

Brakker
[originally posted on jforum.net by Anonymous]

[Migrated From Jforum.net]

Sorry, I forgot to log in on my last reply.

Anyway, I hope your fix does help as I am also still concerned about the unloading of classes. When I watched the app in jprofile, every click to a topic caused a lot of class unloading. If a bot or very heavy traffic were to hit the forums all at once, I believe it would be too much for the gb to handle. I am hoping that this new option of allowing class unloading on multiple threads will also help things.

Brakker
[originally posted on jforum.net by coolbreeze]

[Migrated From Jforum.net]

I am not sure if there is a way to see what classes are being unloaded. I will keep a lookout.

I am not sure if cglib would help at this point because I am not very familiar with it.

Rafael, let me ask you something...

I didn't even realize this until now, but the code that was failing for me was version 2.1.4. It seems that this DAO you fixed is not part of version 2.1.4. Is this code you fixed somewhere else in 2.1.4?

I will upgrade anyway, but I am curious about this.

Brakker
[originally posted on jforum.net by coolbreeze]

[Migrated From Jforum.net]

Ah, until version 2.1.4 it is net/jforum/drivers/generic/UserModel.java

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Rafael,

This has nothing to do with the memory issue, but I didn't want to create a new topic since this one is already recent...

I just upgraded to the new version and everything works fine except for the same bug that I had experienced in the previous version. The problem is when a normal user or moderator (Non-Admin) makes a new topic post (the very first post under a new topic) and then tries to edit the topic subject or message. I get an error at PostAction 512 and it has to do with there not being a hidden input of topic_type on post_form.htm page. If I manually put in anything for the topic_type on post_form.htm, all works fine. The problem with this is that I then receive an error when trying to create a new topic (meaning I can then edit the first post under any topic, but I cannot create a new topic because there shouldn't be a topic_type at that point).

Any thoughts?

Brakker
[originally posted on jforum.net by coolbreeze]

[Migrated From Jforum.net]

hm, are you using customized templates? If so, please check if your templates/default/post_form.htm has this line of code:

<#if setType?default(true) && forum?exists && canCreateStickyOrAnnouncementTopics?default(false)>
    html html htmk
<#elseif topic?exists>
    <input type="hidden" name="topic_type" value="${topic.type}" />
</#if>

this is near line ~360 / 380.

I tested my local installation and it's working. However, based on your report, I fixes a minor bug, that may allow users to change the topic type. The diff is here:

https://jforum.dev.java.net/source/browse/jforum/src/net/jforum/view/forum/PostAction.java?r1=1.82&r2=1.83

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

That seemed to work, thanks.

I will put this new version into production either tonight or tomorrow moring and let you know how it goes.

Brakker
[originally posted on jforum.net by coolbreeze]

[Migrated From Jforum.net]

Rafael,

On a side note, I added

<script type="text/javascript" src="${JForumContext.encodeURL("/js/list/utils")}"></script>

to my version of post_show.html as it was missing the javascript for bookmarks.

Brakker


[originally posted on jforum.net by coolbreeze]

[Migrated From Jforum.net]

Ok.

Just make sure you get the latest cvs version again, before putting it in production. I did a lot of changes today.

Do you know that JForum has support to scheduling of posts? eg, instead of indexing the posts for search in the time the message is sent, you can schedule it to occur once per day (for example.. it's all configurable).

I'll write a doc for it at http://www.jforum.net/confluence

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

I think that is a great idea to schedule the indexing of the posts once a day or so as long as it doesn't cause the search result performance to majorly degrade...I will check it out.

Brakker
[originally posted on jforum.net by coolbreeze]