My forum hangs sometimes after maxxing out DB connections

[Migrated From Jforum.net]

It seems to happen about once a day. The forum just freezes up. It gets 51 DB connections and then it stops working. Usually I can fix that by shutting down Tomcat, and then shutting down Postgres, and then restarting them, but now it isn't coming back up at all. The site is getting about 50,000 hits per day.

Any ideas on this?

It's great software, but it doesn't seem very stable.

Thanks

[originally posted on jforum.net by CaliforniaCCW]

[Migrated From Jforum.net]

CaliforniaCCW wrote:It seems to happen about once a day. The forum just freezes up. It gets 51 DB connections and then it stops working. Usually I can fix that by shutting down Tomcat, and then shutting down Postgres, and then restarting them, but now it isn't coming back up at all. The site is getting about 50,000 hits per day.

Any ideas on this?

It's great software, but it doesn't seem very stable.

Thanks

Which version of jForum you use?
Can you provide Tomcat's stdout_xxx.log file created just after jForum hangs?
[originally posted on jforum.net by Serge Maslyukov]

[Migrated From Jforum.net]

This is the latest JForum, 2.1.6. This is what the process table looks like when it hangs:

postgres 17615  0.0  1.4  30852 13732 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1040) UPDATE waiting
postgres 17616  0.0  1.4  30736 13720 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1041) UPDATE waiting
postgres 17617  0.0  1.4  30940 13768 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1042) UPDATE waiting
postgres 17618  0.0  1.4  30744 13760 ?        S    Aug26   0:03 postgres: jforum californiaccw 127.0.0.1(1043) UPDATE waiting
postgres 17619  0.0  1.4  30872 13872 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1044) idle
postgres 17620  0.0  1.4  31000 13972 ?        S    Aug26   0:04 postgres: jforum californiaccw 127.0.0.1(1045) UPDATE waiting
postgres 17621  0.0  1.4  30912 13724 ?        S    Aug26   0:08 postgres: jforum californiaccw 127.0.0.1(1046) idle
postgres 17622  0.0  1.4  30856 13836 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1047) idle
postgres 17623  0.0  1.4  30932 13648 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1048) idle

(etc)

Only particular threads "hang" when this happens. It's happening right now in fact. Somehow I can't read a particular thread until this UPDATE waiting thing clears on the thread, but it never does until I restart Tomcat. Any ideas?

[originally posted on jforum.net by CaliforniaCCW]

[Migrated From Jforum.net]

CaliforniaCCW wrote:This is the latest JForum, 2.1.6. This is what the process table looks like when it hangs:

postgres 17615  0.0  1.4  30852 13732 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1040) UPDATE waiting
postgres 17616  0.0  1.4  30736 13720 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1041) UPDATE waiting
postgres 17617  0.0  1.4  30940 13768 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1042) UPDATE waiting
postgres 17618  0.0  1.4  30744 13760 ?        S    Aug26   0:03 postgres: jforum californiaccw 127.0.0.1(1043) UPDATE waiting
postgres 17619  0.0  1.4  30872 13872 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1044) idle
postgres 17620  0.0  1.4  31000 13972 ?        S    Aug26   0:04 postgres: jforum californiaccw 127.0.0.1(1045) UPDATE waiting
postgres 17621  0.0  1.4  30912 13724 ?        S    Aug26   0:08 postgres: jforum californiaccw 127.0.0.1(1046) idle
postgres 17622  0.0  1.4  30856 13836 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1047) idle
postgres 17623  0.0  1.4  30932 13648 ?        S    Aug26   0:02 postgres: jforum californiaccw 127.0.0.1(1048) idle

(etc)

Only particular threads "hang" when this happens. It's happening right now in fact. Somehow I can't read a particular thread until this UPDATE waiting thing clears on the thread, but it never does until I restart Tomcat. Any ideas?

When you say "only particular threads hang" does that just mean it seems to be random? Or are the threads tied back to a particular forum or user?

[originally posted on jforum.net by GatorBait3]

[Migrated From Jforum.net]

It's seemingly random. Some threads become "locked", such that if you click on them, they never load. I look in the processes and there are these "update waiting" processes. I have to kill Tomcat to get it going again.

This is totally repeatable. It happens once a day.

Any ideas?

[originally posted on jforum.net by CaliforniaCCW]

[Migrated From Jforum.net]

CaliforniaCCW wrote:It's seemingly random. Some threads become "locked", such that if you click on them, they never load. I look in the processes and there are these "update waiting" processes. I have to kill Tomcat to get it going again.

This is totally repeatable. It happens once a day.

Any ideas?

Probably, all of these "update waiting" processes are waiting on a table with foreign keys that is locked -- possibly locked by one of your "idle" processes. If any of the "idle" processes are not critical (e.g., one is just some statistics collector), try killing it and restarting it. Basically, some sql code has locked the table and hasn't freed the lock. This could be due to a seriously ugly postgres bug (some residual hash table problem) or it could be due to a plain ugly sql script bug (e.g., in jforum code, itself).
[originally posted on jforum.net by rseaton]

[Migrated From Jforum.net]

Better answer -- wait for version 2.1.7 (or whatever the next release is). v2.1.6 is dated 3/1/06. Note the following comment and it's date.

"Jforum deadlock problem solved?
Posted June 19, 2006

When cleaning out the needless jforum logic that determined how to handle non-transactional DB connections (sakai should always be transactional), I inadvertantly removed the commit() call. This should have been a pretty easy thing to debug��? if the DB connection is never committed, nothing should ever be saved to the database.

Oh, how naive! In sakai, we have a database connection pool. When jforum finished it�۪s request/response thread, it was returning its connection to the pool like any well behaved application should. The problem was that other application services were grabbing that same dirty connection, which was never committed and never rolled back. When that other service called commit() or rollBack(), the sql issued from jforum would suffer the same fate."

In more detail, it seems that Jforum code to update the thread-viewed and user-posted counts may be leaving tables locked. (Bad Cody!)
[originally posted on jforum.net by rseaton]

[Migrated From Jforum.net]

It's 99% likely a bug in JForum's SQL in that case. I'm running the latest version of Postgres (8.1.4 I think). PG is a very stable and mature database.

Any ideas on catching this bug?

Edited: Ah, ok, I just saw that post. It is a bug in JForum. Any ideas on when 2.1.7 is coming?

[originally posted on jforum.net by CaliforniaCCW]

[Migrated From Jforum.net]

Well,

this is kinda confusing. If someone could help on this, it will be great.

CaliforniaCCW, JForum is very stable, but obviously not bug free. We'll do our best to fix all reported problems quickly, and we count with your help - and everybode else!

Let us know if you find anything new. I'll check the database code and error handling, to make sure nothing is being left behind.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Rafael Steil wrote:Well,

this is kinda confusing. If someone could help on this, it will be great.

CaliforniaCCW, JForum is very stable, but obviously not bug free. We'll do our best to fix all reported problems quickly, and we count with your help - and everybode else!

Let us know if you find anything new. I'll check the database code and error handling, to make sure nothing is being left behind.

Rafael

Thanks, Rafael. Apparently, when CaliforniaCCW is seeing a large number of visits and/or posts, one of the database processes looks like it's forgetting to unlock a row (or table) and this is causing other db processes to continue waiting on the lock. The user symptom is that attempting to visit a particular message (or section) never finishes loading, but other messages (or sections) can be loaded and/or refreshed. The admin symptom is that some db processes are sitting idle. When this happens, from the user's perspective there are typically more and more messages (or sections) which become autistic in this fashion until nothing on the forum site will load/refresh. My guess is that from the admin perspective, more and more idle db processes are waiting on locks (maybe not the same locks) until there are no more available db processes to reuse for new db requests. This appears to be a new behavior that you haven't seen before with Jforum. Thanks.
[originally posted on jforum.net by rseaton]

[Migrated From Jforum.net]

Thanks Rafael. It's 100% reproducible. We would probably get 100,000 hits a day if it didn't go down so often, so this is a high-volume site. Is there anything I can do to help you figure this out? I could set up any type of debugging you need, let you log in by SSH, etc. You name it. The site goes down at least once a day now.

[originally posted on jforum.net by CaliforniaCCW]

[Migrated From Jforum.net]

Hello
I am evaluating JForum to integrate it on a hight traffic website.
Looking at the code of the DAOs, I have noticed a disturbing thing : the jdbc operations are not wrapped in try/catch/finally blocks
This could leads to very bad things such as database cursor leaks.
Below a page explaining in detail how should be wrapped JDBC operations : http://www.javalobby.org/java/forums/t18930.html

Hope it helps

Alex


[originally posted on jforum.net by alexvictoor]

[Migrated From Jforum.net]

At this point I would not recommend JForum for a high-traffic site. I have a high-traffic site and I have to restart the server at least once a day due to leaks of various kinds.

Other problems with JForum: It uses a cool page template system, but would be easier to use if it used plain old JSP. There are many more tools for JSP these days, and now JSP has expression language, etc, so it's quite nice. In fact, JSF would be a better way to go also. Another problem is JForum does not yet ship with a build.xml file so I'm not even completely sure what's the right way to build it.

As for using JDBC, there is no right way to use JDBC in a modern Java app. JDBC should not be used except in a few cases. The right way to do things now is with EJB3 or perhaps Hibernate. If you use EJB3 and Seam, JBoss will handle all the database and SQL issues. You can use this to build a whole site that has no need for JDBC or SQL code.

[originally posted on jforum.net by CaliforniaCCW]

[Migrated From Jforum.net]

I do not think there is no right way to use JDBC, I think it is not a good thing to use JDBC directly in your code, without any wrapper framework.
Spring-jdbc, iBatis ... there are many good frameworks out there.

CaliforniaCCW, if you are considering using stuff such as EJB3/seam and so on, you might look at Jboss portal bb module.



[originally posted on jforum.net by alexvictoor]

[Migrated From Jforum.net]

We already have so many posts within JForum, I don't think it would be possible to convert them to any other format.

On that subject... it sure would be cool if there were a common forum dump XML standard, so I could, say, dump all my forum data (users, posts, etc) into this XML file from VBulletin and then import it into JForum, that kind of thing. Right now, when you get a forum going in one type of software it is very hard to switch.

When I have time I might make my own fork of JForum, by moving it over into JBoss and switching more to EJBs and JSF for presentation.

[originally posted on jforum.net by CaliforniaCCW]

[Migrated From Jforum.net]

alexvictoor wrote:Hello
I am evaluating JForum to integrate it on a hight traffic website.
Looking at the code of the DAOs, I have noticed a disturbing thing : the jdbc operations are not wrapped in try/catch/finally blocks
This could leads to very bad things such as database cursor leaks.
Below a page explaining in detail how should be wrapped JDBC operations : http://www.javalobby.org/java/forums/t18930.html

Hope it helps

Alex

CVS of jForum's DAO located here:
https://jforum.dev.java.net/source/browse/jforum/src/net/jforum/dao/

can you point to me where try/catch/finally are missed?


[originally posted on jforum.net by Serge Maslyukov]

[Migrated From Jforum.net]

IMHO moving JForum to lighter solution such as spring and spring-JDBC whould be more effective. You wont get any permormance /reliability gain with JSF. JSF needs to keep a "control tree" in the http session... Your application will need more ram.
FYI an alpha release of Jboss forum, using JSF and hibernate, will be available at the end of the month.
[originally posted on jforum.net by alexvictoor]

[Migrated From Jforum.net]

My deep apologies !!
I do not see any try/catch/finally missed in the CVS.
However, in the last stable release (2.1.6 if I am not wrong), the JDBC operations are not wrapped in try/catch/finally blocks.

[originally posted on jforum.net by alexvictoor]

[Migrated From Jforum.net]

alexvictoor, the code is new for 2.17.

CaliforniaCCW, I'm adding to the cvs a new key to SystemGlobals.properties named "c3p0.extra.params", where it's possible to pass additional parameters to the connection pool.

The default value is

checkTimeout=180; debugUnreturnedConnectionStackTraces=false; unreturnedConnectionTimeout=180

and it's possible to use any other configuration property of cC3P0.

We're trying to figure out how the database hangs. That's a relly anoying behaviour.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Ok, the changes are in the cvs now.

Setthing checkoutTimeout=120000 and debugUnreturnedConnectionStackTraces=true may help to trace the problem.

There are other interesting options in the C3P0 documentation

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Rafael, if I am not wrong CaliforniaCCW is using jforum2.16 . Perhaps the new DAO code from the CVS would solve his problem...

[originally posted on jforum.net by alexvictoor]

[Migrated From Jforum.net]

I see.

I guess it's time to prepare an official beta build of 2.1.7

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Ok, I put the beta download at http://www.jforum.net/beta/latest_dev.jsp

Who could test it, we appreciate.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

alexvictoor wrote:Rafael, if I am not wrong CaliforniaCCW is using jforum2.16 . Perhaps the new DAO code from the CVS would solve his problem...

Yes I'm using 2.16, the official release. I haven't tried any of the CVS code. When there is an official 2.17 release I'll definitely run it. I'm guessing that my forum starts hanging when people perhaps click "cancel" while a thread is loading, and that leaves some connection hung. With try-catch-finally that should fix it.

I'll be monitoring this forum eagerly for the next release. Everyone on my forum is eager for it too because the thing gets hung at least once a day. We have a lot of traffic, up to 80,000 hits a day lately. We would have more if the software didn't hang up.

[originally posted on jforum.net by CaliforniaCCW]

[Migrated From Jforum.net]

CaliforniaCCW, the current cvs code is the one which will go as production. There isn't any planed changes to the database code.

I know that it sounds ridiculous to ask you to put a development package in your very busy website, so, if you want, I can be online on a specific time of the day and give you any assistance needed, for example.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Rafael Steil wrote:CaliforniaCCW, the current cvs code is the one which will go as production. There isn't any planed changes to the database code.

I know that it sounds ridiculous to ask you to put a development package in your very busy website,

I'm ready to do it! It's a good idea. I believe in testing code on production sites, in cases where it's not critical to the business.

so, if you want, I can be online on a specific time of the day and give you any assistance needed, for example.

Thank you for the offer Rafael! Actually if you can point me to a tar or a zip file that I should grab and deploy, I'll do it and post my experience here. We're a fairly high-traffic forum, with about 100,000 hits a day, about 100 posts a day, so we're a good test site.

My question: with the new tar ball, can I just install the jar files, or do I need to reinstall the .htm files also? The reason I ask is because i have made a few customizations to the templates so it would be easiest if I could just replace jar files.

Anyway... please send me a link to a zip file, or a CVS command to use and I'll put it on our live server over the weekend.

A few recommendations btw: I hope it has an ant build.xml file in it. It's much easier to work with software that has that. And if you're using CVS I would look into SVN. CVS hurts my brain. SVN is good.

Thanks again for the excellent forum software. Yes it has bugs and instability, but once that is resolved this is going to be a lot better than everything else, especially that horrible PHP forum software. Did you know that when vbulletin stores strings in the DB, it stores them with HTML-entities? It is painful to think about.

[originally posted on jforum.net by CaliforniaCCW]

[Migrated From Jforum.net]

Hello Jforum,

Can anyone point me to a zip file of a somewhat stable code release? My forum is now going over 100k hits a day and we are having outages several times a day. If we didn't have outages we would be over 200k hits a day. I would be deeply thankful if there's a release I could install to stop these outages. As I said above I would be happy to provide any type of debugging info related to this, even including letting the JForum developers ssh into the machine to see what's going on during the frequent outages.

Thanks

[originally posted on jforum.net by CaliforniaCCW]

[Migrated From Jforum.net]

Sorry man. Here you go:

http://www.jforum.net/beta/latest_dev.jsp

Rafael
[originally posted on jforum.net by Rafael Steil]