slow web page response times

[jon london]

Hi We are using jboss-4.0.4.GA and are experiencing very slow web page times... on AIX 5.3.0.0, 2GB RAM, 1 processor (Processor Type: PowerPC_POWER5,Processor , clock Speed: 1656 MHz, CPU Type: 64-bit, Kernel Type: 64-bit)

We use quartz scheduler to run lots of scheduled jobs with a threadpool size of 20. We also use Spring, Hibernate and multiple web services within the Jboss container

I am using jconsole to view memory usage, which seems to peak (at 1280MB) every 20 minutes or so, then GC runs and the memory usage goes back down to around 256MB.

Sometimes just logging in takes around 30 seconds or longer (using BASIC JAAS)... This is even when memory and CPU usage of the JVM are very low.

Is there any way you can give priority to web sessions, so that they can use JVM resources first? It's frustrating to see that memory and CPU are available but its still taking so long to navigate our web application.

Would really appreciate any advice here.

This is our lscfg
====

INSTALLED RESOURCE LIST

The following resources are installed on the machine.
+/- = Added or deleted from Resource List.
* = Diagnostic support not available.

Model Architecture: chrp
Model Implementation: Multiple Processor, PCI bus

+ sys0 System Object
+ sysplanar0 System Planar
* vio0 Virtual I/O Bus
* ent1 U9119.590.518191E-V19-C3-T1 Virtual I/O Ethernet Adapter (l-lan)
* vscsi1 U9119.590.518191E-V19-C42-T1 Virtual SCSI Client Adapter
* hdisk8 U9119.590.518191E-V19-C42-T1-L860000000000 Virtual SCSI Disk Drive
* hdisk3 U9119.590.518191E-V19-C42-T1-L820000000000 Virtual SCSI Disk Drive
* hdisk2 U9119.590.518191E-V19-C42-T1-L810000000000 Virtual SCSI Disk Drive
* vscsi0 U9119.590.518191E-V19-C41-T1 Virtual SCSI Client Adapter
* hdisk11 U9119.590.518191E-V19-C41-T1-L890000000000 Virtual SCSI Disk Drive
* hdisk10 U9119.590.518191E-V19-C41-T1-L880000000000 Virtual SCSI Disk Drive
* hdisk9 U9119.590.518191E-V19-C41-T1-L870000000000 Virtual SCSI Disk Drive
* hdisk6 U9119.590.518191E-V19-C41-T1-L860000000000 Virtual SCSI Disk Drive
* hdisk1 U9119.590.518191E-V19-C41-T1-L820000000000 Virtual SCSI Disk Drive
* hdisk0 U9119.590.518191E-V19-C41-T1-L810000000000 Virtual SCSI Disk Drive
* ent0 U9119.590.518191E-V19-C2-T1 Virtual I/O Ethernet Adapter (l-lan)
* vsa0 U9119.590.518191E-V19-C0 LPAR Virtual Serial Adapter
* vty0 U9119.590.518191E-V19-C0-L0 Asynchronous Terminal
+ L2cache0 L2 Cache
+ mem0 Memory
+ proc0 Processor

and our JVM options:
=====================

/usr/java5/bin/java -Xms1280m -Xmx1280m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dprogram.name=run.sh -Djboss.bind.address=10.40.30.229 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8004 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -verbosegc -Djava.endorsed.dirs=/opt/middleware/products/jboss/jboss-4.0.4.GA/lib/endorsed -classpath /opt/middleware/products/jboss/jboss-4.0.4.GA/lib:/opt/middleware/products/jboss/jboss-4.0.4.GA/lib/endorsed:/opt/middleware/products/jboss/jboss-4.0.4.GA/server/all/lib:/usr/java5/lib:/opt/kpi/runtime/jboss/lib:/opt/middleware/products/jboss/jboss-4.0.4.GA/bin/run.jar:/usr/java5/lib/tools.jar org.jboss.Main -Djboss.home.dir=/opt/middleware/products/jboss/jboss-4.0.4.GA -Djboss.server.home.dir=/opt/kpi/runtime/jboss -Djboss.server.base.dir=/opt/kpi/runtime/jboss -Djboss.server.log.dir=/opt/kpi/runtime/jboss/log -Djboss.server.lib.url=file:/opt/kpi/runtime/jboss/lib -Djboss.lib.url=file:/opt/middleware/products/jboss/jboss-4.0.4.GA/lib -Djboss.server.data.dir=/opt/kpi/runtime/jboss/data -Djboss.server.temp.dir=/opt/kpi/runtime/jboss/tmp -Djboss.server.name=managed1 -Dprogram.name=KPI -Djboss.patch.url=file:/opt/middleware/products/jboss/jboss-4.0.4.GA/patch --configuration=/opt/kpi/runtime/jboss

[Paul Sturrock]

Is there any way you can give priority to web sessions, so that they can use JVM resources first? It's frustrating to see that memory and CPU are available but its still taking so long to navigate our web application.

Have you determined if the bottle neck is memory or CPU bound? It could be IO - have you run a profiler against it?

[jon london]

Is there a profiler that you would reccommend for AIX? How would I determine and resolve I/O issues?

CPU usage is generally low so I don't think the bottleneck is there. Memory is generally available, especially after GC so I don't think the problem is with memory.

[Lester Burnham]

Memory is generally available, especially after GC so I don't think the problem is with memory.

Given the earlier statement

which seems to peak (at 1280MB) every 20 minutes or so

and

-Xms1280m -Xmx1280m

I'd say something is a bit odd memory-wise. There are apparently a lot of objects being created and junked in a short period of time; both creating and GC'ing 1 GB of objects takes time - is all that really necessary?

[jon london]

regards I/O

looking at topas (AIX) I can see that the top two hards disks are continously working very hard...

    
Disk    Busy%     KBPS     TPS KB-Read KB-Writ  PgspIn      370  % Noncomp   3.2
hdisk0   97.0   1834.0   322.0   848.0   986.0  PgspOut     246  % Client    3.2
hdisk8   84.5   1622.0   269.0   636.0   986.0  PageIn      370
hdisk1    2.0     14.2     4.0     0.0    14.2  PageOut     247  PAGING SPACE
hdisk3    2.0     14.2     4.0     0.0    14.2  Sios        617  Size,MB    2048
hdisk2    0.0      0.0     0.0     0.0     0.0                   % Used     74.1
hdisk6    0.0      0.0     0.0     0.0     0.0  NFS (calls/sec)  % Free     25.8
hdisk9    0.0      0.0     0.0     0.0     0.0  ServerV2       0
hdisk11   0.0      0.0     0.0     0.0     0.0  ClientV2       0   Press:
hdisk10   0.0      0.0     0.0     0.0     0.0

unfortunately I know very little about how hard disk utilisation works on AIX.

I have also noticed that our log4j logging is in debug mode and the logs are constantly being written.

Reading here, http://www.jboss.com/pdf/JB_JEAP4_3_PerformanceTuning_wp.pdf, it seems that verbose logging could be one of our issues, which ties in with Paul's comment on I/O issues.

Lester I have also thought that our memory usage cycles are odd... We trigger many jobs via quartz (1000's per day) and these jobs (Extract Transform Load) use alot of objects including RMI, and hibernate via Spring. So the creation of so many objects is not such a surprise, however I think we need to tweak our garbage collection config to stop the memory peaking out so often. Can anyone offer any GC tips for such an application?

[jon london]

Also, is it normal for PgSp to be always increasing and never reducing in size?

    
Name            PID  CPU%  PgSp Owner           ServerV3       0   "h" for help
java         860278  23.4 406.1 kpirun          ClientV3       0   "q" to quit
topas        561272   0.5   2.2 kpirun
lrud          16392   0.2   0.1 root  
gil           57372   0.1   0.1 root  
getty        344242   0.1   0.4 root  
mstragen     499736   0.1   1.3 root  
tecad_lo     757774   0.0   0.9 root