Win a copy of Functional Reactive Programming this week in the Other Languages forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

100% CPU usage on JBoss with 8GB memory

 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all,

We've got an application that stores a large amount of data in-memory which is then used by a lot of users.

Subsequently we're seeing CPU usage go through the use, and every now and then hit 100% (on a quad core machine with 8GB of mem) which I believe is when the JVM tries to garbage collect. The web app can't be used until JBoss is killed and restarted.

We currently have these JVM opts:
JAVA_OPTS="$JAVA_OPTS -Xms5g -Xmx5g -XX:MaxNewSize=3g -XX:PermSize=512m -XX:MaxP
ermSize=1g -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterv
al=3600000 -Djava.awt.headless=true -d64 -verbose:gc"

Which may or may not be causing the problems - my knowledge of GC management is very limited.


We see something similar to this in the logs:

(lots more)


(lots more)

Any help would be much appreciated.

JBoss 4.2.1GA
SuSE Enterprise 10 SP1
Java 1.6.0_05
 
Michal Glowacki
Ranch Hand
Posts: 114
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you using EJB in your application? When there is lot of ejb transactions it can result in deadlock and in fact heap runs out of memory. In that case gc can't come to the rescue.
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, the system is predominately based around EJBs.

I'm not sure why a large use of EJBs would result in deadlock?
 
rajesh bala
Ranch Hand
Posts: 66
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Did you try taking a thread dump?. 100% CPU utilization problems can be easily solved by taking a threaddump and viewing what is going in the threads.

Use kill -QUIT or kill -3 for linux based and ctrl+pause-break in windows in the jboss console.

You could post the threaddump over here if you prefer.

~Rajesh.B
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Cheers rajesh - I'll do that next time it does it.
 
Michal Glowacki
Ranch Hand
Posts: 114
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Dean Pullen:
Yes, the system is predominately based around EJBs.

I'm not sure why a large use of EJBs would result in deadlock?


It is bad practice, actually even anti-pattern to use EJB as read-only objects - it puts table locks what increase overhead. Where you do not need transaction management you should use data access objects. I can recommend you a tool to generate DAO for your database - it's http://mdaog.sourceforge.net/. It's very ease to replece your entity bean calls with dao and transfer objects using library generated by this tool.
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I hadn't actually realised this.

They are in fact not read-only. They are generated and then added to the DB.

Would it be best to use standard EJBs to create them, and then retrieve them as DAO's at a latter point for caching?

Also...if we simply detach them from the hibernate session they effectively become POJOs, isn't this sufficient as opposed to creating a new DAO etc?
 
Michal Glowacki
Ranch Hand
Posts: 114
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Not sure about hibernate.

I mean not to use ejb as read-only objects in places where you only read from them. Feel free to use them to insert/update tables in db where you need transaction management.

So what I mean you can (and for some/most cases should) have both dao and entity beans for the table. You use dao when you read (except from some transactions, fx in session beans where you need to be sure the data meanwhile hasn't changed it's state) and ejb to insert/update.

Personally, I use BASICALLY entity beans to insert/update data in tables with balances, etc. But when I for example record login attemp, I use dao to insert record into table. For read - I use everywhere dao, except my session bean methods, which for example make calculations and update important table, again, like balances etc.

Hope that helps.
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I can imagine from what you're saying that you're using EJB2.1 and not EJB3?
 
Michal Glowacki
Ranch Hand
Posts: 114
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
yes, 2.1 on JBoss 4.0.5GA
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The DAO mentality is somewhat different with EJB3 - entities are basically Pojo's removing the need for seperate DAOs.

Trying to figure out how best to cache them is my next step.
 
Roger Chung-Wee
Ranch Hand
Posts: 1683
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, the system is predominately based around EJBs

Are you using entity beans? JBoss uses dynamic proxies to generate the container classes at run time, making extensive use of reflection which can really affect performance.
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by rajesh bala:
Did you try taking a thread dump?. 100% CPU utilization problems can be easily solved by taking a threaddump and viewing what is going in the threads.

Use kill -QUIT or kill -3 for linux based and ctrl+pause-break in windows in the jboss console.

You could post the threaddump over here if you prefer.

~Rajesh.B


I've done a kill -3 and received the following output:
Stack trace

I notice two threads linked to the class CsiAdminAccessChildren.

Is it possible this is in an infinite loop or similar, thus causing the 100% CPU usage?
 
rajesh bala
Ranch Hand
Posts: 66
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Most of the threads seems to be fine in the stacktrace. If you have taken this when the CPU was 100% its good.

1. I hope you have access to the sourcecode.
2. If so, please look at CsiAdminAccessLevelCache.java:169. This is being accessed from 3 threads and all the three are in runnable state.
3. Good part is that, all the threads are rooted from CsiAdminAccessLevelCache only. So worst case scenario, you could try turning it off and try out if possible (Try this option if you dont have source code and you can try it in staging box first)
4. Check what is the size of the TreeMap in the cache.

But otherwise, the memory footprint and others threads are very normal. If possible you could post the code snippet here (provided your company allows it. ..).

~Rajesh.B
 
Dean Pullen
Ranch Hand
Posts: 58
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
As I thought - and you too - there was a problem with that class.

Infinite loop. Thus the 100% CPU usage!

Deary me! The kill 3 was a master stroke, I'll be using that more often.

Thanks again.
 
rajesh bala
Ranch Hand
Posts: 66
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Good to hear that it solved the problem. You are welcome Dean.

~Rajesh.B
 
Bobby Sharma
Ranch Hand
Posts: 598
3
Google App Engine jQuery Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thanks Rajesh, I also learned that
 
M Fern�ndez
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all,
we're facing a similar problem in our production environment. First we try to replicate the problem in a test environment, a clone of the production environment, with little success. Fortunately we've discovered this forum thread and the kill -3 option.
We toke a threaddump of the production server, while 3 of the 4 cpu's shows a 100% occupation, and found two methods being accessed from several threads and all them are in runnable state. This two methods have loops, I've pasted two code snippets at the bottom of this post.
If you could have a look at the threaddump and confirm us we are in the right way we will appreciate your help a lot.
Thanks in advance


public void service(HttpServletRequest request, HttpServletResponse response){
try {
...
response.setContentType ("application/download");
response.setHeader ("Content-Disposition", "attachment; filename="+nombreFichero );

java.io.File file = new java.io.File(nombreFichero);
response.setContentLength((int)file.length());
java.io.InputStream in = new java.io.BufferedInputStream (new java.io.FileInputStream(file) );
int length;
byte[] buf = new byte[response.getBufferSize()];
java.io.OutputStream op = response.getOutputStream ();
while ((in != null) && ((length = in.read(buf)) != -1)){
op.write(buf,0,length);
}
if (in != null){
in.close();
}
if ( op != null){
op.close();
}

}catch (Exception e){
...
}
}



private ApplicationObjectInterface buscarFamiliaEnJerarquia(String id) throws Exception{
if(nivelesJerarquia != null){
ApplicationObjectInterface obj = null;

for( Iterator iter= this.nivelesJerarquia.values().iterator();iter.hasNext() ; ) {
obj = (ApplicationObjectInterface) iter.next();
if ( obj != null && obj.getObjetosHijos() != null){
HashMap gamasCat = obj.getObjetosHijos();

for (Iterator iter1=gamasCat.values().iterator(); iter1.hasNext() ; ) {
obj = (ApplicationObjectInterface) iter1.next();

if ( obj != null && obj.getObjetosHijos() != null){
HashMap familiasCat = obj.getObjetosHijos();
for (Iterator iter2=familiasCat.values().iterator(); iter2.hasNext() ; ) {
obj = (ApplicationObjectInterface) iter2.next();
if ( obj != null && obj.getId().equals(id)){
return (ApplicationObjectInterface) obj;
}//if
}//for
}//if
}//for
}//for
}//for
}//if
return null;
}

[ July 16, 2008: Message edited by: Marco Fern�ndez ]

[ July 16, 2008: Message edited by: Marco Fern�ndez ]

[ July 18, 2008: Message edited by: M Fern�ndez ]
[ July 18, 2008: Message edited by: M Fern�ndez ]
 
steve souza
Ranch Hand
Posts: 862
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Due to formatting problems it's hard to read your code. It looks like you are looping through a lot of data structures to find one object. Maybe you have the wrong type of object and should instead be using one that has better direct access such as a Map.

Also, you should have opened a new posting and not hijacked this old one.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic