Starting custom threads in Websphere Servlet Container

[Mehmet Gunacti]

Hello,

The web application I develop creates many threads in a "StartupServlet"'s init method and destroys them in the destroy method.

The threads work in an infinite loop unless they are signaled to stop. They have to create a lot of list objects, that is, they produce a lot of garbage object (objects left for the GC).

The reason, I create them within a servlet environment is because they need to invoke remote ejbs. I pass them an InitialContext object prepared within the servlet environment. (and invoking remote EJBs in a standalone application can be quite cumbersome.)

The threads are marked as daemon threads, so that they don't hinder the WAS process (main process) from shutting down.

My questions are :
1) Websphere runs on the IBM JVM. What are the odds that the IBM JVM's GC doesn't collect garbage objects created by my custom threads ?
2) Are there any other drawbacks in creating custom threads within a servlet environment ?

[Brian Hennessy]

Hi Mehmet,

1) Websphere runs on the IBM JVM. What are the odds that the IBM JVM's GC doesn't collect garbage objects created by my custom threads ?

I don't know the exact answer to this , I'd be surprised if they were not garbage collected. Probably a question for an IBM gc programmer, however there are application developer tools you can use to try and determine this.

1. Use a profiler in RAD to see if the objects created by the unmanaged threads are garbage collected ? Run your server in profile mode for x amount of minutes , force a gc then view the profiling output. There should be a column in the profile view of objects collected.
2. Run your server for x amount of time with only your servlets executing (no online users) , force a heap dump analyze the heap using a HeapAnalyzer. Allow the application run for another x amount of hours
force a heap dump . Do this multiple times, if the number of unmanaged object references are growing on each heap dump, you may have a potential memory leak.

Option 1 is more exact as you have proof of these objects being collected or not.


2) Are there any other drawbacks in creating custom threads within a servlet environment ?
IBM don't recommend you create unmanaged threads due to some of the reasons listed in the following linkhttp://www.ibm.com/developerworks/websphere/techjournal/0609_alcott/0609_alcott.html#spring-4 such as preventing graceful shutdown, releasing resources, unable to create an InitialContext (i believe new InitialContext() in unmanaged threads returns null) etc. I think they have an alternative approach to doing custom threading using the WorkManager API see http://www.ibm.com/developerworks/websphere/techjournal/0606_johnson/0606_johnson.html

In terms of the ejb/servlet specs creating threads in an ejb breaks the spec see (page 563 of ejb 2.1 spec )

The enterprise bean must not attempt to manage threads. The enterprise bean must not attempt
to start, stop, suspend, or resume a thread, or to change a thread’s priority or name. The enter-
prise bean must not attempt to manage thread groups.

The servlet spec is not as strict , I think it translates to not having the InitailContext available is unmanaged threads.

In terms of your application design why do you need to create these lists in a thread in an infinite loop ? Are you using this servlet as a sort of batch job scheduler ?

[Mehmet Gunacti]

Hi Brian,

first of all, thanks for taking the time to answer my question.

1. Use a profiler in RAD to see if the objects created by the unmanaged threads are garbage collected ? Run your server in profile mode for x amount of minutes , force a gc then view the profiling output. There should be a column in the profile view of objects collected.

Ok, I'll do that. A colleague once did that in a prev. project, but I never took/had the time to go into profiling WAS.

BM don't recommend you create unmanaged threads due to some of the reasons listed in the following linkhttp://www.ibm.com/developerworks/websphere/techjo...cott/0609_alcott.html#spring-4 such as preventing graceful shutdown, releasing resources, unable to create an InitialContext (i believe new InitialContext() in unmanaged threads returns null) etc.

The document says :

Unmanaged threads are unknown to WebSphere Application Server and do not have access to Java EE contextual information.

Yes, but that's why I use a 'startup servlet' and pass an InitialContext() object to my custom threads, so that they can access the JNDI service.

In addition, they can use resources without WebSphere Application Server knowing about it,

Since I ask WAS for the resources through an InitialContext() created within its process, it shouldn't be a problem (at least, that's what I think).

I think they have an alternative approach to doing custom threading using the WorkManager API see http://www.ibm.com/developerworks/websphere/techjo...0606_johnson/0606_johnson.html

At startup I need exactly 12 threads to be created, that is, I know the number of threads, which is configurable through a properties file. The 'worker' threads wait for a queue object to pass them 'reference IDs' from a database. So they don't wait for user interaction, but rather wait for newly inserted rows into a database table.

so I don't think that I need a 'thread pool', which makes this whole app a lot less complex.

In terms of your application design why do you need to create these lists in a thread in an infinite loop ? Are you using this servlet as a sort of batch job scheduler ?

Exactly. Very simple explained, it looks like this :
There's a database table, which gets input from various channels (MQ, jsp pages, web services, etc.). Now I've one thread per App Server that performs a 'Select' query to this table every 5 seconds. When it detects new rows, it collects the primary keys and passes it to a queue manager object. the 12 worker threads wait for a 'notifyAll()' call of the queue manager, which does that after it gets new work, and retrieve one ID. They then handle the row by performing a 'select for update' query.
Finally a worker thread uses the InitialContext object to call remote EJBs, get database connections from the WAS JDBC Connection Pool and send data to a Websphere MQ server through JNDI.

I've tried to lookup a Home Object of the Remote EJBs residing on another WAS server from within a standalone Java app, but got a lot of errors. When I googled these errors (can't remember the details) I saw that a lot of developers struggled with different types of errors while trying to invoke remote EJBs from standalone Java Apps. (I ran the Java app within RAD and used WAS's JRE, but there were problems caused by the different JVM versions. so I didn't even try to use another JVM).
Putting the right jar files into the classpath was another problem. Now, the EJB app I connect to runs on WAS 6.0, but they're going to migrate to 6.1, and later to WAS 7.
so, getting an InitialContext from a running WAS server is the most convenient way.


I do think that creating unmanaged threads is bad practice, though. And, I'm actually waiting for the performance tests to complete next week.

[Brian Hennessy]

Hi Mehmet
I'm glad the profiling suggestion is of use to you and I hope your load testing goes well. If for some reason you change your mind on the unmanaged threads or theres some other compelling reason to change your code, another way to implement this could be to use a message driven bean. Instead of running the first select statement you could have each channel(jsp, web services) send a message to an mq queue instead of adding a row to a database. The mdb listeners for a message on that queue and processes it as it receives them. Its possible to configure multiple instances of mdbs which acts like your 12 threads.. You also don't need to worry about the row locking (select for update) as the container services handle the synchronizing access to each message. .
The rest of your logic remains the same
On the remote ejb lookups I know your pain

Best of luck
Brian

[Mehmet Gunacti]

Instead of running the first select statement you could have each channel(jsp, web services) send a message to an mq queue instead of adding a row to a database. The mdb listeners for a message on that queue and processes it as it receives them. Its possible to configure multiple instances of mdbs which acts like your 12 threads.. You also don't need to worry about the row locking (select for update) as the container services handle the synchronizing access to each message. .

Wow, I've never thought of using MQ like that (at the beginning of this project, I wasn't even aware we could use MQ at all). Right now we use MQ to communicate with apps running on the mainframe.
Maybe in version 2.0, because right now there are limitations caused by the contract and (of course) a time-limit we already exceeded.

Thanks for the great tip !

Regards,
Mehmet