Socket connection error in Tomcat

[Tushar Madhukar]

Hi,

I have a web application (Tomcat 5.5.23) that talks to an Oracle database. Clients to this webapp simple push/retrieve data from the DB.

I have noticed infrequently that after a few operations, the web app stops responding and the catalina.out prints the following trace:

    
Jan 4, 2008 2:02:58 AM org.apache.jk.common.MsgAjp processHeader
SEVERE: BAD packet signature 18245
Jan 4, 2008 2:02:58 AM org.apache.jk.common.ChannelSocket processConnection
SEVERE: Error, processing connection
java.lang.IndexOutOfBoundsException
        at java.io.BufferedInputStream.read(BufferedInputStream.java:306)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:626)
        at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:583)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.ja
va:691)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSock
et.java:895)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadP
ool.java:685)
        at java.lang.Thread.run(Thread.java:595)

What is the reason for this error? It's not something my application throws. Just FYI, the webapp maintains a pool of DB connections to the Oracle DB.

Thanks

[Akhilesh Trivedi]

Tushar can you go in detail.

First, is your Tomcat start-up is all ok. Make sure if tomcat isn't crashed. Try re-installing it?

I have not done too much of JDBC, but if there is manual connection-pooling done check if its non-erroneous. Handle connections properly, handle exceptions, release resources when not in use.

[Tushar Madhukar]

Hi,

I don't see any error in Tomcat startup. Abt connection pooling, we have our own implementation which is well tested and exercised.

Any idea what can be the source of this error? The message and stack trace makes me wonder if this error is thrown even before my web app receives the request from the container..

[William Brogden]

The message and stack trace makes me wonder if this error is thrown even before my web app receives the request from the container..

I think you are right, an error in the request Thread would show servlet related classes in the stack trace.

If this was my problem I would be looking at the source code for those org.apache.jk.common classes for hints.

Bill

[Tony Stocker]

I'm also getting this error, actually on three different servers, here's sample output:

Apr 13, 2010 6:18:06 PM org.apache.jk.common.MsgAjp processHeader
SEVERE: BAD packet signature 18245
Apr 13, 2010 6:18:06 PM org.apache.jk.common.ChannelSocket processConnection
SEVERE: Error, processing connection
java.lang.IndexOutOfBoundsException
at java.io.BufferedInputStream.read(BufferedInputStream.java:310)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:621)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:578)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:686)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)
Apr 13, 2010 6:18:06 PM org.apache.jk.common.MsgAjp processHeader
SEVERE: BAD packet signature 18245
Apr 13, 2010 6:18:06 PM org.apache.jk.common.ChannelSocket processConnection
SEVERE: Error, processing connection
java.lang.IndexOutOfBoundsException
at java.io.BufferedInputStream.read(BufferedInputStream.java:310)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:621)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:578)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:686)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)
Apr 13, 2010 6:18:06 PM org.apache.jk.common.MsgAjp processHeader
SEVERE: BAD packet signature 18245
Apr 13, 2010 6:18:06 PM org.apache.jk.common.ChannelSocket processConnection
SEVERE: Error, processing connection
java.lang.IndexOutOfBoundsException
at java.io.BufferedInputStream.read(BufferedInputStream.java:310)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:621)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:578)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:686)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)


All three servers are reporting this error.
Server 1 is a CentOS 4 machines running these versions:
Apache HTTP 2.0.52-41.ent.7.centos4
Tomcat 6.0.14
mod_jk 1.2.25

Server 2 is a CentOS 4 machines running these versions:
Apache HTTP 2.0.52-41.ent.7.centos4
Tomcat 6.0.26
mod_jk 1.2.28-httpd-2.0.X.so

Server 3 is a CentOS 5 machines running these versions:
Apache HTTP 2.2.3-31.el5.centos.4
Tomcat 6.0.26
mod_jk 1.2.28-httpd-2.2.X.so

The really odd thing here is that I can't seem to find anything BROKEN in terms of apps/servelets. All the example jsp/servlets seem to be working fine, as does an in-house developed app. However there's a new issue, one that I do not believe is related to this error, but I've been asked to track down the source and fix it.

So what should I be looking for here? (I'm a system admin type, not a developer so my Tomcat/JSP knowledge is generally weak.)

The mod_jk files are up to date, and I'm using the correct binaries for the correct version of Apache HTTP on each system. The few posts on the site seem to indicate that this is a case of an AJP port getting input not in AJP format, but I'm not sure what would be doing that. I can make the following error occur simply by telneting to port 8009 (AJP 1.3 connector port) and typing random characters, but note that the "signature ####" is different when I do that:

SEVERE: BAD packet signature 25196
Apr 15, 2010 7:20:12 PM org.apache.jk.common.ChannelSocket processConnection
SEVERE: Error, processing connection
java.lang.IndexOutOfBoundsException
at java.io.BufferedInputStream.read(BufferedInputStream.java:310)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:621)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:578)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:686)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)

That number seems to be dependent on the characters I type at the telnet prompt, as I seem to be able to generate the 18245 number if what I type on the telnet line is "GET" (case sensitive as "get" generates a 26469.)

So it seems like there is something that is doing an awful lot of GET requests on this port. Is there a likely culprit?

Thanks!

[Tim Holloway]

If it was an ARRAY IndexOutOfBoundsException, I'd suspect some sort of buffer overflow. But it's not. It seems to imply that network packet headers are defective.

You might want to have someone check out the networking hardware, especially any routers and switches in the local segment.

[Paul Jacob]

Hi,
Even I am getting the same error. I have two load balanced tomcat instances on the same machine. The AJP ports are 8009 & 8019. I am getting this error on the one that is on 8019 port :

May 12, 2010 12:56:17 PM org.apache.jk.common.MsgAjp processHeader
SEVERE: BAD packet signature 18245
May 12, 2010 12:56:17 PM org.apache.jk.common.ChannelSocket processConnection
SEVERE: Error, processing connection
java.lang.IndexOutOfBoundsException
at java.io.BufferedInputStream.read(Unknown Source)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:621)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:578)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:686)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Unknown Source)

I have not noticed anything wrong from the application side .... but this error bothers me since we are going live in a week.
Any help is much appreciated.
Thanks.

[Shriram Gopalakrishnan]

Did anyone find solution for the problem, I am having the same problem and if someone suggests the solution used to resolve this that would be of great help.

[Leonardo Jose Gomes da Silva]

Someone solved that?
I'm getting the same error...

Hugs