Hi All,
I have configured
jboss clustering using Apache web server with mod_jk load balancer.
I have deployed a sample application in all\farm folder in one of the cluster member
it will automatically deployed in all the cluster members.
I am providing all the configurations I have made
In Apache web server
httpd.conf file
-----------------
added the following lines
# Include mod_jk's specific configuration file
Include conf/mod_jk.conf
mod_jk.conf file
------------------
# Load mod_jk module
# Specify the filename of the mod_jk lib
LoadModule jk_module modules/mod_jk.so
# Where to find workers.properties
JkWorkersFile conf/workers.properties
# Where to put jk logs
JkLogFile logs/mod_jk.log
# Set the jk log level [debug/error/info]
JkLogLevel debug
# Select the log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"
# JkRequestLogFormat
JkRequestLogFormat "%w %V %T"
# Mount your applications
JkMount /application/* loadbalancer
# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
JkShmFile logs/jk.shm
# Add jkstatus for managing runtime data
<Location /jkstatus/>
JkMount status
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Location>
workers.properties
----------------------
# Define list of workers that will be used
# for mapping requests
worker.list=loadbalancer,status
# Define Node1
# modify the host as your host IP or DNS name.
worker.node1.port=8009
worker.node1.host=172.16.1.46
worker.node1.type=ajp13
worker.node1.lbfactor=1
# worker.node1.cachesize=10
# Define Node2
# modify the host as your host IP or DNS name.
worker.node2.port=8009
worker.node2.host=172.16.1.191
worker.node2.type=ajp13
worker.node2.lbfactor=1
# worker.node2.cachesize=10
# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=1
#worker.list=loadbalancer
# Status worker for managing load balancer
worker.status.type=status
JBoss AS nodes configuration
---------------------------------
Node running on application server 172.16.1.46
In JBOSS_HOME\jboss-as\server\all\deploy\jboss-web.deployer\server.xml added jvmRoute attribute
<Engine name="jboss.web" defaultHost="localhost"
jvmRoute="node1">
Node running on application server 172.16.1.191
In JBOSS_HOME\jboss-as\server\all\deploy\jboss-web.deployer\server.xml added jvmRoute attribute
<Engine name="jboss.web" defaultHost="localhost"
jvmRoute="node2">
All above details are configurations I have made
When I deployed the sample web application individully it is working but when access through apache web server it is not working
it is showing the following lines..
Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
My apache webserver is running at port 8080(172.16.1.46)
Jboss AS node1 running at 9090(172.16.1.46)
Jboss AS node2 running at 8080(172.16.1.191)
I am using the following url to access the sample
servlet
http://localhost:8080/application/clusterDemo
I am attaching the log files of the Apache web server
mod_jk.log file
------------------
[Mar 04 15:13:18 2009][3104:2056] [debug] jk_map.c (588): Dump of map: 'worker.loadbalancer.sticky_session' -> '1'
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_map.c (588): Dump of map: 'worker.status.type' -> 'status'
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (242): creating worker loadbalancer
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (146): about to create instance loadbalancer of lb
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (159): about to validate and init loadbalancer
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (146): about to create instance node1 of ajp13
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (159): about to validate and init node1
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2469): worker node1 contact is '172.16.1.46:8009'
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2644): setting endpoint options:
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2647): keepalive: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2651): socket timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2655): socket connect timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2659): buffer size: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2663): pool timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2667): ping timeout: 10000
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2671): connect timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2675): reply timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2679): prepost timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2683): recovery options: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2687): retries: 2
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2691): max packet size: 8192
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2695): retry interval: 100
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2508): setting connection pool size to 64 with min 32 and acquire timeout 200
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (146): about to create instance node2 of ajp13
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (159): about to validate and init node2
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2469): worker node2 contact is '172.16.1.191:8009'
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2644): setting endpoint options:
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2647): keepalive: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2651): socket timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2655): socket connect timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2659): buffer size: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2663): pool timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2667): ping timeout: 10000
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2671): connect timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2675): reply timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2679): prepost timeout: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2683): recovery options: 0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2687): retries: 2
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2691): max packet size: 8192
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2695): retry interval: 100
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_ajp_common.c (2508): setting connection pool size to 64 with min 32 and acquire timeout 200
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_lb_worker.c (1572): Balanced worker 0 has name node1 and route node1 in domain
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_lb_worker.c (1572): Balanced worker 1 has name node2 and route node2 in domain
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_lb_worker.c (261): worker node1 gets multiplicity 1
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_lb_worker.c (261): worker node2 gets multiplicity 1
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_lb_worker.c (342): syncing shm for lb 'loadbalancer' from mem
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (242): creating worker status
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (146): about to create instance status of status
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (159): about to validate and init status
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_status.c (4800): Status worker 'status' is read/write and has css '(null)', prefix 'worker', name space 'jk:', xml name space 'xmlns:jk="http://tomcat.apache.org"', document type '(null)'
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_status.c (4851): Status worker 'status' has good rating for '0000000f' and bad rating for '00ff1010'
[Wed Mar 04 15:13:18 2009][3104:2056] [info] mod_jk.c (3018): mod_jk/1.2.27 initialized
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (116): found a worker loadbalancer
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (293): Found worker type 'lb'
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_uri_worker_map.c (511): Checking extension for worker 0: loadbalancer of type lb (5)
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (116): found a worker status
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_worker.c (293): Found worker type 'status'
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_uri_worker_map.c (511): Checking extension for worker 1: status of type status (6)
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_uri_worker_map.c (171): uri map dump after extension stripping: index=0 file='(null)' reject_unsafe=0 reload=60 modified=0 checked=0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_uri_worker_map.c (176): generation 0: size=0 nosize=0 capacity=0
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_uri_worker_map.c (176): generation 1: size=2 nosize=0 capacity=4
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_uri_worker_map.c (185): NEXT (1) map #0: uri=/application/* worker=loadbalancer context=/application/* source=JkMount type=Wildchar len=14
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_uri_worker_map.c (185): NEXT (1) map #1: uri=/jkstatus/ worker=status context=/jkstatus/ source=JkMount type=Exact len=10
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_uri_worker_map.c (481): Switching uri worker map from index 0 to index 1
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] jk_shm.c (158): Shared memory is already opened
[Wed Mar 04 15:13:18 2009][3104:2056] [debug] mod_jk.c (2903): Initialized mod_jk/1.2.27
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_uri_worker_map.c (1035): Attempting to map URI '/application/clusterDemo' from 2 maps
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_uri_worker_map.c (849): Attempting to map context URI '/application/*=loadbalancer' source 'JkMount'
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_uri_worker_map.c (862): Found a wildchar match '/application/*=loadbalancer'
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] mod_jk.c (2320): Into handler jakarta-servlet worker=loadbalancer r->proxyreq=0
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_worker.c (116): found a worker loadbalancer
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_worker.c (339): Maintaining worker loadbalancer
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_lb_worker.c (649): decay with 2^14
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_worker.c (293): Found worker type 'lb'
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] mod_jk.c (932): Service protocol=HTTP/1.1 method=GET host=(null) addr=172.16.1.46 name=172.16.1.46 port=8080 auth=(null) user=(null) laddr=172.16.1.46 raddr=172.16.1.46 uri=/application/clusterDemo
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_lb_worker.c (1088): service sticky_session=1 id='empty'
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_lb_worker.c (969): found best worker node1 (node1) using method 'Request'
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_lb_worker.c (1133): service worker=node1 route=node1
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_ajp_common.c (2922): acquired connection pool slot=0 after 0 retries
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_ajp_common.c (605): ajp marshaling done
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_ajp_common.c (2246): processing node1 with 2 retries
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_ajp_common.c (1461): (node1) all endpoints are disconnected.
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_connect.c (451): socket TCP_NODELAY set to On
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_connect.c (575): trying to connect socket 1584 to 172.16.1.46:8009
[Wed Mar 04 15:27:45 2009][3104:3920] [info] jk_connect.c (593): connect to 172.16.1.46:8009 failed (errno=61)
[Wed Mar 04 15:27:45 2009][3104:3920] [info] jk_ajp_common.c (922): Failed opening socket to (172.16.1.46:8009) (errno=61)
[Wed Mar 04 15:27:45 2009][3104:3920] [error] jk_ajp_common.c (1467): (node1) connecting to backend failed.
Tomcat is probably not started or is listening on the wrong port (errno=61)
[Wed Mar 04 15:27:45 2009][3104:3920] [info] jk_ajp_common.c (2407): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[Wed Mar 04 15:27:45 2009][3104:3920] [debug] jk_ajp_common.c (2267): retry 1, sleeping for 100 ms before retrying
[Wed Mar 04 15:27:46 2009][3104:3920] [debug] jk_ajp_common.c (1461): (node1) all endpoints are disconnected.
[Wed Mar 04 15:27:46 2009][3104:3920] [debug] jk_connect.c (451): socket TCP_NODELAY set to On
[Wed Mar 04 15:27:46 2009][3104:3920] [debug] jk_connect.c (575): trying to connect socket 1584 to 172.16.1.46:8009
[Wed Mar 04 15:27:47 2009][3104:3920] [info] jk_connect.c (593): connect to 172.16.1.46:8009 failed (errno=61)
[Wed Mar 04 15:27:47 2009][3104:3920] [info] jk_ajp_common.c (922): Failed opening socket to (172.16.1.46:8009) (errno=61)
[Wed Mar 04 15:27:47 2009][3104:3920] [error] jk_ajp_common.c (1467): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Wed Mar 04 15:27:47 2009][3104:3920] [info] jk_ajp_common.c (2407): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[Wed Mar 04 15:27:47 2009][3104:3920] [error] jk_ajp_common.c (2426): (node1) connecting to tomcat failed.
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_ajp_common.c (743): (node1) resetting endpoint with sd = 4294967295 (socket shutdown)
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_ajp_common.c (2850): recycling connection pool slot=0 for worker node1
[Wed Mar 04 15:27:47 2009][3104:3920] [info] jk_lb_worker.c (1347): service failed, worker node1 is in error state
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_lb_worker.c (1358): recoverable error... will try to recover on other worker
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_lb_worker.c (969): found best worker node2 (node2) using method 'Request'
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_lb_worker.c (1133): service worker=node2 route=node1
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_ajp_common.c (2922): acquired connection pool slot=0 after 0 retries
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_ajp_common.c (605): ajp marshaling done
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_ajp_common.c (2246): processing node2 with 2 retries
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_ajp_common.c (1461): (node2) all endpoints are disconnected.
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_connect.c (451): socket TCP_NODELAY set to On
[Wed Mar 04 15:27:47 2009][3104:3920] [debug] jk_connect.c (575): trying to connect socket 1584 to 172.16.1.191:8009
[Wed Mar 04 15:27:48 2009][3104:3920] [info] jk_connect.c (593): connect to 172.16.1.191:8009 failed (errno=61)
[Wed Mar 04 15:27:48 2009][3104:3920] [info] jk_ajp_common.c (922): Failed opening socket to (172.16.1.191:8009) (errno=61)
[Wed Mar 04 15:27:48 2009][3104:3920] [error] jk_ajp_common.c (1467): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Wed Mar 04 15:27:48 2009][3104:3920] [info] jk_ajp_common.c (2407): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[Wed Mar 04 15:27:48 2009][3104:3920] [debug] jk_ajp_common.c (2267): retry 1, sleeping for 100 ms before retrying
[Wed Mar 04 15:27:48 2009][3104:3920] [debug] jk_ajp_common.c (1461): (node2) all endpoints are disconnected.
[Wed Mar 04 15:27:48 2009][3104:3920] [debug] jk_connect.c (451): socket TCP_NODELAY set to On
[Wed Mar 04 15:27:48 2009][3104:3920] [debug] jk_connect.c (575): trying to connect socket 1584 to 172.16.1.191:8009
[Wed Mar 04 15:27:49 2009][3104:3920] [info] jk_connect.c (593): connect to 172.16.1.191:8009 failed (errno=61)
[Wed Mar 04 15:27:49 2009][3104:3920] [info] jk_ajp_common.c (922): Failed opening socket to (172.16.1.191:8009) (errno=61)
[Wed Mar 04 15:27:49 2009][3104:3920] [error] jk_ajp_common.c (1467): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Wed Mar 04 15:27:49 2009][3104:3920] [info] jk_ajp_common.c (2407): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[Wed Mar 04 15:27:49 2009][3104:3920] [error] jk_ajp_common.c (2426): (node2) connecting to tomcat failed.
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] jk_ajp_common.c (743): (node2) resetting endpoint with sd = 4294967295 (socket shutdown)
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] jk_ajp_common.c (2850): recycling connection pool slot=0 for worker node2
[Wed Mar 04 15:27:49 2009][3104:3920] [info] jk_lb_worker.c (1347): service failed, worker node2 is in error state
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] jk_lb_worker.c (1358): recoverable error... will try to recover on other worker
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] jk_lb_worker.c (1101): retry 1, sleeping for 100 ms before retrying
[Wed Mar 04 15:27:49 2009][3104:3920] [info] jk_lb_worker.c (577): worker node1 is marked for forced recovery
[Wed Mar 04 15:27:49 2009][3104:3920] [info] jk_lb_worker.c (577): worker node2 is marked for forced recovery
[Wed Mar 04 15:27:49 2009][3104:3920] [info] jk_lb_worker.c (1399): Forcing recovery once for 2 workers
[Wed Mar 04 15:27:49 2009][3104:3920] [info] jk_lb_worker.c (1416): All tomcat instances failed, no more workers left (attempt=0, retry=1)
[Wed Mar 04 15:27:49 2009][3104:3920] [info] jk_lb_worker.c (1416): All tomcat instances failed, no more workers left (attempt=1, retry=1)
[Wed Mar 04 15:27:49 2009][3104:3920] [info] jk_lb_worker.c (1427): All tomcat instances are busy or in error state
[Wed Mar 04 15:27:49 2009][3104:3920] [error] jk_lb_worker.c (1432): All tomcat instances failed, no more workers left
[Wed Mar 04 15:27:49 2009]loadbalancer 172.16.1.46 4.390625
[Wed Mar 04 15:27:49 2009][3104:3920] [info] mod_jk.c (2476): Service error=0 for worker=loadbalancer
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] jk_uri_worker_map.c (1035): Attempting to map URI '/favicon.ico' from 2 maps
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] jk_uri_worker_map.c (849): Attempting to map context URI '/application/*=loadbalancer' source 'JkMount'
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] jk_uri_worker_map.c (849): Attempting to map context URI '/jkstatus/=status' source 'JkMount'
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] mod_jk.c (3361): no match for /favicon.ico found
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] jk_uri_worker_map.c (1035): Attempting to map URI '/favicon.ico' from 2 maps
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] jk_uri_worker_map.c (849): Attempting to map context URI '/application/*=loadbalancer' source 'JkMount'
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] jk_uri_worker_map.c (849): Attempting to map context URI '/jkstatus/=status' source 'JkMount'
[Wed Mar 04 15:27:49 2009][3104:3920] [debug] mod_jk.c (3428): no match for /favicon.ico found
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_uri_worker_map.c (1035): Attempting to map URI '/application/clusterDemo' from 2 maps
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_uri_worker_map.c (849): Attempting to map context URI '/application/*=loadbalancer' source 'JkMount'
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_uri_worker_map.c (862): Found a wildchar match '/application/*=loadbalancer'
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] mod_jk.c (2320): Into handler jakarta-servlet worker=loadbalancer r->proxyreq=0
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_worker.c (116): found a worker loadbalancer
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_worker.c (293): Found worker type 'lb'
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] mod_jk.c (932): Service protocol=HTTP/1.1 method=GET host=(null) addr=172.16.1.46 name=172.16.1.46 port=8080 auth=(null) user=(null) laddr=172.16.1.46 raddr=172.16.1.46 uri=/application/clusterDemo
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_lb_worker.c (1088): service sticky_session=1 id='empty'
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_lb_worker.c (969): found best worker node1 (node1) using method 'Request'
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_lb_worker.c (1133): service worker=node1 route=node1
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_ajp_common.c (2922): acquired connection pool slot=0 after 0 retries
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_ajp_common.c (605): ajp marshaling done
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_ajp_common.c (2246): processing node1 with 2 retries
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_ajp_common.c (1461): (node1) all endpoints are disconnected.
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_connect.c (451): socket TCP_NODELAY set to On
[Wed Mar 04 15:28:09 2009][3104:3920] [debug] jk_connect.c (575): trying to connect socket 1584 to 172.16.1.46:8009
[Wed Mar 04 15:28:10 2009][3104:3920] [info] jk_connect.c (593): connect to 172.16.1.46:8009 failed (errno=61)
[Wed Mar 04 15:28:10 2009][3104:3920] [info] jk_ajp_common.c (922): Failed opening socket to (172.16.1.46:8009) (errno=61)
[Wed Mar 04 15:28:10 2009][3104:3920] [error] jk_ajp_common.c (1467): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Wed Mar 04 15:28:10 2009][3104:3920] [info] jk_ajp_common.c (2407): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[Wed Mar 04 15:28:10 2009][3104:3920] [debug] jk_ajp_common.c (2267): retry 1, sleeping for 100 ms before retrying
[Wed Mar 04 15:28:10 2009][3104:3920] [debug] jk_ajp_common.c (1461): (node1) all endpoints are disconnected.
[Wed Mar 04 15:28:10 2009][3104:3920] [debug] jk_connect.c (451): socket TCP_NODELAY set to On
[Wed Mar 04 15:28:10 2009][3104:3920] [debug] jk_connect.c (575): trying to connect socket 1584 to 172.16.1.46:8009
[Wed Mar 04 15:28:11 2009][3104:3920] [info] jk_connect.c (593): connect to 172.16.1.46:8009 failed (errno=61)
[Wed Mar 04 15:28:11 2009][3104:3920] [info] jk_ajp_common.c (922): Failed opening socket to (172.16.1.46:8009) (errno=61)
[Wed Mar 04 15:28:11 2009][3104:3920] [error] jk_ajp_common.c (1467): (node1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Wed Mar 04 15:28:11 2009][3104:3920] [info] jk_ajp_common.c (2407): (node1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[Wed Mar 04 15:28:11 2009][3104:3920] [error] jk_ajp_common.c (2426): (node1) connecting to tomcat failed.
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_ajp_common.c (743): (node1) resetting endpoint with sd = 4294967295 (socket shutdown)
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_ajp_common.c (2850): recycling connection pool slot=0 for worker node1
[Wed Mar 04 15:28:11 2009][3104:3920] [info] jk_lb_worker.c (1347): service failed, worker node1 is in error state
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_lb_worker.c (1358): recoverable error... will try to recover on other worker
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_lb_worker.c (969): found best worker node2 (node2) using method 'Request'
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_lb_worker.c (1133): service worker=node2 route=node1
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_ajp_common.c (2922): acquired connection pool slot=0 after 0 retries
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_ajp_common.c (605): ajp marshaling done
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_ajp_common.c (2246): processing node2 with 2 retries
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_ajp_common.c (1461): (node2) all endpoints are disconnected.
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_connect.c (451): socket TCP_NODELAY set to On
[Wed Mar 04 15:28:11 2009][3104:3920] [debug] jk_connect.c (575): trying to connect socket 1584 to 172.16.1.191:8009
[Wed Mar 04 15:28:12 2009][3104:3920] [info] jk_connect.c (593): connect to 172.16.1.191:8009 failed (errno=61)
[Wed Mar 04 15:28:12 2009][3104:3920] [info] jk_ajp_common.c (922): Failed opening socket to (172.16.1.191:8009) (errno=61)
[Wed Mar 04 15:28:12 2009][3104:3920] [error] jk_ajp_common.c (1467): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Wed Mar 04 15:28:12 2009][3104:3920] [info] jk_ajp_common.c (2407): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[Wed Mar 04 15:28:12 2009][3104:3920] [debug] jk_ajp_common.c (2267): retry 1, sleeping for 100 ms before retrying
[Wed Mar 04 15:28:12 2009][3104:3920] [debug] jk_ajp_common.c (1461): (node2) all endpoints are disconnected.
[Wed Mar 04 15:28:12 2009][3104:3920] [debug] jk_connect.c (451): socket TCP_NODELAY set to On
[Wed Mar 04 15:28:12 2009][3104:3920] [debug] jk_connect.c (575): trying to connect socket 1584 to 172.16.1.191:8009
[Wed Mar 04 15:28:13 2009][3104:3920] [info] jk_connect.c (593): connect to 172.16.1.191:8009 failed (errno=61)
[Wed Mar 04 15:28:13 2009][3104:3920] [info] jk_ajp_common.c (922): Failed opening socket to (172.16.1.191:8009) (errno=61)
[Wed Mar 04 15:28:13 2009][3104:3920] [error] jk_ajp_common.c (1467): (node2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Wed Mar 04 15:28:13 2009][3104:3920] [info] jk_ajp_common.c (2407): (node2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[Wed Mar 04 15:28:13 2009][3104:3920] [error] jk_ajp_common.c (2426): (node2) connecting to tomcat failed.
[Wed Mar 04 15:28:13 2009][3104:3920] [debug] jk_ajp_common.c (743): (node2) resetting endpoint with sd = 4294967295 (socket shutdown)
[Wed Mar 04 15:28:13 2009][3104:3920] [debug] jk_ajp_common.c (2850): recycling connection pool slot=0 for worker node2
[Wed Mar 04 15:28:13 2009][3104:3920] [info] jk_lb_worker.c (1347): service failed, worker node2 is in error state
[Wed Mar 04 15:28:13 2009][3104:3920] [debug] jk_lb_worker.c (1358): recoverable error... will try to recover on other worker
[Wed Mar 04 15:28:13 2009][3104:3920] [debug] jk_lb_worker.c (1101): retry 1, sleeping for 100 ms before retrying
[Wed Mar 04 15:28:13 2009][3104:3920] [info] jk_lb_worker.c (577): worker node1 is marked for forced recovery
[Wed Mar 04 15:28:13 2009][3104:3920] [info] jk_lb_worker.c (577): worker node2 is marked for forced recovery
[Wed Mar 04 15:28:13 2009][3104:3920] [info] jk_lb_worker.c (1399): Forcing recovery once for 2 workers
[Wed Mar 04 15:28:13 2009][3104:3920] [info] jk_lb_worker.c (1416): All tomcat instances failed, no more workers left (attempt=0, retry=1)
[Wed Mar 04 15:28:13 2009][3104:3920] [info] jk_lb_worker.c (1416): All tomcat instances failed, no more workers left (attempt=1, retry=1)
[Wed Mar 04 15:28:13 2009][3104:3920] [info] jk_lb_worker.c (1427): All tomcat instances are busy or in error state
[Wed Mar 04 15:28:13 2009][3104:3920] [error] jk_lb_worker.c (1432): All tomcat instances failed, no more workers left
[Wed Mar 04 15:28:13 2009]loadbalancer 172.16.1.46 4.500000
[Wed Mar 04 15:28:13 2009][3104:3920] [info] mod_jk.c (2476): Service error=0 for worker=loadbalancer
Please suggest me the right approach..
Thanks in advance,
Krishna
