Brian Hennessy

Hi Andres
On AIX I've seen the process stop due to oom of native memory segments on a 32 bit system where we had set the max heap size to 2gb. This gave us only 2 segments (512MB) available for java -> native code generated by the VM .


I think in the native_stderr.log I was seeing c based malloc,calloc calls fail(return -1) , the vm shut down. I had verbose GC enabled at the time .

This pointed us to the heap being sized too large for the amount of native code that was being generated.

Not saying thats your issue but i suppose another thing to look for.

Hi
I've used this tutorial before, its for WAS 5 but approach is still relevant

Testing J2EE Security Applications Using a Custom Registry in WebSphere Studio V5

Yes in the admin console (I'm on WAS 6.0)

Enterprise Applications => app Name => Session Management => Cookies

The cookie path will be set to /

change this to /set_value_here , some people use the apps context root so change it to /your_context_root
Make sure though that if you uninstall and reinstall your app you set this value again. Might be worth doing this through a jython deployment script.

Hi Catrine,
I think your post will generate more questions than answers. Can you advise on why you need 3 threads to run on 3 different servers, are the 3 threads doing different jobs whats the problem with them running on one server.

Also some other things you should be aware of. If you are creating threads like
Thread t = new Thread(new MyRunnable())

this is not recommended. In ejb specs its not allowed and while not explicit in the servlet spec I don't think its advised. In websphere unmanaged threads don't have access to context info so jndi looks using an InitialContext may give you null pointers. For scheduling take a look at the scheduling section in this article

I think post some more details on what type of work you need to do with these 3 threads and why they need to execute on different servers and we'll see if there are alternative ways to doing what you want.

Divya
I think you need to post your code that reads the XML file and describe how its being executed for example from servlet , async bean ,ejb etc.

Hi Selva
I would look at a couple of things.
Try identify if you have SQL calls taking longer than your connection wait timeout. If you have 50 connections all executing SQL and each piece of SQL takes 3 minutes to execute you could get this problem.

If its not the database then it could be a call to an integration point thats wrapped in the same transactional call as the db call.
For example

void callQuote() {
start transaction
call really_fast_sql
call http_service
end transaction
}

Note: start and end transaction probably handled by Spring AOP

if the call to call http_service blocks waiting on a response , the connection that is used to make the SQL call will not be released to the connection pool until the whole method ends.
Maybe the external system you call experiences problems at your peak load and its not in the database.

You fix this by removing the remote call from the transaction


Perform a thread dump and see whats executing and what else is waiting[and look at code].

hi Ankit
you can use a mutiple servers in the corbaloc url

corbaloc::server1_boot_strap_port:9812,:server2_boot_strap_port::9813

I'm not sure that it load balances in a round robin fashion but it will failover to the next server in the list if the first is down.

In terms of load balancing this article has some code samples on ejb client calls it randomly picks the corbaloc urls , sort of DIY load balancing

for example
url1=corbaloc::server1_boot_strap_port:9812,:server2_boot_strap_port::9813
url2=corbaloc::server2_boot_strap_port:9813,:server1_boot_strap_port::9812

http://www.ibm.com/developerworks/websphere/techjournal/0412_vansickel/0412_vansickel.html

Hi Mehmet
I'm glad the profiling suggestion is of use to you and I hope your load testing goes well. If for some reason you change your mind on the unmanaged threads or theres some other compelling reason to change your code, another way to implement this could be to use a message driven bean. Instead of running the first select statement you could have each channel(jsp, web services) send a message to an mq queue instead of adding a row to a database. The mdb listeners for a message on that queue and processes it as it receives them. Its possible to configure multiple instances of mdbs which acts like your 12 threads.. You also don't need to worry about the row locking (select for update) as the container services handle the synchronizing access to each message. .
The rest of your logic remains the same
On the remote ejb lookups I know your pain

Best of luck
Brian

Hi Mehmet,

1) Websphere runs on the IBM JVM. What are the odds that the IBM JVM's GC doesn't collect garbage objects created by my custom threads ?

I don't know the exact answer to this , I'd be surprised if they were not garbage collected. Probably a question for an IBM gc programmer, however there are application developer tools you can use to try and determine this.

1. Use a profiler in RAD to see if the objects created by the unmanaged threads are garbage collected ? Run your server in profile mode for x amount of minutes , force a gc then view the profiling output. There should be a column in the profile view of objects collected.
2. Run your server for x amount of time with only your servlets executing (no online users) , force a heap dump analyze the heap using a HeapAnalyzer. Allow the application run for another x amount of hours
force a heap dump . Do this multiple times, if the number of unmanaged object references are growing on each heap dump, you may have a potential memory leak.

Option 1 is more exact as you have proof of these objects being collected or not.


2) Are there any other drawbacks in creating custom threads within a servlet environment ?
IBM don't recommend you create unmanaged threads due to some of the reasons listed in the following linkhttp://www.ibm.com/developerworks/websphere/techjournal/0609_alcott/0609_alcott.html#spring-4 such as preventing graceful shutdown, releasing resources, unable to create an InitialContext (i believe new InitialContext() in unmanaged threads returns null) etc. I think they have an alternative approach to doing custom threading using the WorkManager API see http://www.ibm.com/developerworks/websphere/techjournal/0606_johnson/0606_johnson.html

In terms of the ejb/servlet specs creating threads in an ejb breaks the spec see (page 563 of ejb 2.1 spec )

The enterprise bean must not attempt to manage threads. The enterprise bean must not attempt
to start, stop, suspend, or resume a thread, or to change a thread’s priority or name. The enter-
prise bean must not attempt to manage thread groups.

The servlet spec is not as strict , I think it translates to not having the InitailContext available is unmanaged threads.

In terms of your application design why do you need to create these lists in a thread in an infinite loop ? Are you using this servlet as a sort of batch job scheduler ?

Hi
I am automating deployments using the wsadmin scripting features. I want to try prevent my wsadmin script executing, if someone has logged into the admin console with the same configurater user as my script.

I can potentially use a http call to determine this but would like , if possible, to keep the functionality in wsadmin calls if it exists.

Does anyone know of a way to implement this ?

Thanks in advance

Hi Debopam ,
I cannot answer the question on how it works because I have not looked at the source but I suspect under the covers it does some ldap lookup similar to the articles Deepak has provided. In terms of where the the ldap server is configured it will be in the Global security settings in admin console you need to select ldap as the active user registry. In the same screen it should provide a link to configure the ldap settings (host, port , base dn etc) . I believe this code will look at these configuration details to connect to ldap

Hi Micheal
Is it possible that as you navigate between each application the JSESSIONID is changing (being over written) . This assumes the applications are hosted in different JVMs?

A scenario like this

- Login assign JESSIONID:12345
- Navigate around your application no problems
- Navigate to other company application , this applications over writes JSESSIONID in browser . Browser now has JSESSIONID:54321
- From other application navigate back to yours. Browser sends JSESSIONID:54321 your app doesn't recognise this jsession and your req.getSession returns null ?

Hi sreedhar
I'll do my best to answer some of your questions in relation to my organisations set up

1) Suppose my first request went server1 and will it be possible to go my subsequent request to different server
Its possible depending on what and how you have set up your load balancer. Some load balancers are just Ip sprayers ,however for web applications that use sessions to store state you probably want a web server such as IBM HTTP Server (IHS) or Microsoft IIS that will route requests to the same server that it originated from and if that server is down route it to another server in the cluster (you'll need to have session replication between servers for the failure scenario).
Websphere has an option to generate a web server plugin file called plugin-cfg.xml file. This file has a list of the servers in your cluster their ports and a server identifier. The server identifier is typcially part of the JSESSION cookie id sent in the request. This plugin-cfg.xml can be installed on the web servers . See Install plugin on IIS
The web servers can then figure out how to direct the request based on the unique server id sent in the JSESSION identifier. If the requests are stateless (like some web service requests) then the web server picks a server based on a routing algorithm.
This excellent tech note explains the algorithm for IIS and IHS Understanding IBM HTTP Server plug-in Load Balancing in a clustered environment

2) What are the different load balancing techniqes we have

See answer 1.
Also if you want a just IP sprayer or network dispatcher then look at documentation on IBMs edge server or f5 load balancer. I have not used F5 but I have heard it mentioned in some infoq architecture presentations.

3) What is clustered environment
This is probably not the correct term but for me a clustered environment typically consists of more than one application server logically grouped together. There are operational efficencies to be had with logically grouping servers such as you can deploy one application across multiple servers and configuring fail (over see answer 1).

Hi Ben
A message queue is a data struture used for holding messages. Think of it like a list where you can push items onto the list and pop items off the list.
A message driven bean is a special type of EJB that is used to listen for messages on a messsage queue , take the message from the queue and perform some processing on the message. All this typically runs as a transaction which allows you to roll the message back on the queue under certain failure scenarios.

You create a queue manager in IBM MQ and create queues associated with this queue manager. Applications write messages to these queues by using API calls like JMS. In you JEE application you will have written a message driven bean , this bean will process messages put to the queue. The bean will be configured to use a javax.jms.QueueconnectionFactory which is a configuration setting in WebSphere that represents connection details to the MQ queue manager and a javax.jms.Queue object which is a configuration in websphere that represents the actual queue associated with a queue manager in MQ.

Brian

I'm a little late responding to this, so Subha I apologise for not responding ,but hopefully it comes in handy for somone else. I originally used an application called Portecle to import the cert , I may have found a link to it from the bouncycastle web site. However to get the websphere ikeyman working I following the following instructions from IBM

Unable to import a PKCS12 file that is created by IIS or other non-IBM Web server keystores into a CMS or JKS database