Win a copy of Functional Design and Architecture this week in the Functional programming forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Rob Spoor
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Henry Wong
  • Liutauras Vilda
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Tim Holloway
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Mikalai Zaikin
  • Piet Souris

Exception opening database connection

 
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello, this is my first post for ~10 years :-)

I would be thankful if somebody could give me some hints on how to solve following problem:

For a long while I've stuck to Apache Tomcat/6.0.53 on my local development computer because it just works. I've started to experiment with new features which require a higher Java version and a newer Tomcat.

Under Apache Tomcat/6.0.53 I've developed and become familiar with MySQL database connections. Everything works well under Tomcat 6.0.53:

-Using Java 1.6.0_45
-Placing mysql-connector-java-5.1.6-bin.jar under $CATALINA_HOME/lib and things have always worked well.

Then I tried to upgrade:
-Installed and switched to Java 1.8.0_281.
-Installed Tomcat 9.0.43 and verified that it starts, can serve JSP pages etc.
-Downloaded and copied "mysql-connector-java-8.0.11.jar" to $CATALINA_HOME/lib according to instructions valid at the time:

https://tomcat.apache.org/tomcat-9.0-doc/jndi-datasource-examples-howto.html

Added to my context.xml:



In localhost.YYYY-MM-DD.log I get the stack trace below. Obviously, there is no database connection available.

Does anybody have hints on how to solve this problem and which JDBC Driver actually works?

BR
Reidar Gjerstad

 
Saloon Keeper
Posts: 24283
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Welcome back!

Best I can tell is that you may have a non-printing or otherwise invalid character in your JDBC connection URL.

Since you blanked out details, I can't tell any more than that, however.
 
Reidar Gjerstad
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:Welcome back!

Best I can tell is that you may have a non-printing or otherwise invalid character in your JDBC connection URL.

Since you blanked out details, I can't tell any more than that, however.



Thank you for the reply!

The blanked out details contain only printable characters.  And the same URL is working under Tomcat 6.0. I could try removing the

I don't remember when it was added.

BR
Reidar Gjerstad
 
Tim Holloway
Saloon Keeper
Posts: 24283
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
In XML, unsupported attributes are ignored, so don't expect that to help.

I cross-check the JRE source code. It's pretty definite. Something was attempting to parse a URL and an escape sequence was found in the URL. For example, the "%20" in "https://coderanch.com/messages/my%20message".

In your case, however, it looks like the URL contained "%$Z" and since "$Z" is obviously not a hexadecimal number code, it threw an internal NumberFormatException.

If you can't see anything like that anywhere in your failing Tomcat files, then you may have hardware problems or a corrupt VM.

One way to find out for sure would be to launch Tomcat as a remote debugging session and set a breakpoint on com.mysql.cj.exceptions.WrongArgumentException. I'd actually recommend catching the NumberFormatException itself, but unfortunately, there are jokers out there who deliberately toss around common exception types in their product code and you might get inundated with them.

Also, if you could post the Realm XML it would help, since that's where the Exception actually displays.
 
Tim Holloway
Saloon Keeper
Posts: 24283
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Incidentally, the inability to load the connection class for the Realm is because of the bad URL. In JDBC, it's the URL that determines which driver JAR will be searched to locate the driver itself. And although the driverName is in the form of an absolute classname, I'm not sure if it's actually required to be a real class - just a way of determining which of several drivers a given driver JAR might contain will be used. So the JDBC connectionURL is very important. If the URL cannot be parsed, the JDBC driver JARs cannot be scanned for the requested driver.
 
Reidar Gjerstad
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:
Also, if you could post the Realm XML it would help, since that's where the Exception actually displays.



Tim, thank you for the reply. Apologies for the stupid question, but which "Realm XML" are you referring to? I made a recursive search for xml files, and I can't see any *.xml starting with "Realm" or similar.



The only additional xml is my web app's web.xml.
apache-tomcat-9.0.43/conf/context.xml is the only file I've modified.

Thanks
Reidar Gjerstad
 
Tim Holloway
Saloon Keeper
Posts: 24283
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
"Realm" is an XML node element that defines a security realm for webapps in Tomcat. It can be applied to a single webapp - in which case, it's normally part of conf/server.xml or - more commonly, it's defined in a Context for a single webapp. Your error message implies that there is a JDBC Realm defined somewhere in that Tomcat system and that it contains an invalid JDBC URL.

There's probably at least one sample Realm definition example (commented out) in server.xml. The Tomcat pre-installed management webapps are secured by a MemoryRealm or something like that.
 
Tim Holloway
Saloon Keeper
Posts: 24283
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Oh, and by the way - why did you modify conf/context.xml?
 
Reidar Gjerstad
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:"Realm" is an XML node element that defines a security realm for webapps in Tomcat. It can be applied to a single webapp - in which case, it's normally part of conf/server.xml or - more commonly, it's defined in a Context for a single webapp. Your error message implies that there is a JDBC Realm defined somewhere in that Tomcat system and that it contains an invalid JDBC URL.

There's probably at least one sample Realm definition example (commented out) in server.xml. The Tomcat pre-installed management webapps are secured by a MemoryRealm or something like that.



Thanks again.

Recursive grep for '%$Z': It doesn't exist anywhere in tomcat-dir or my project's dir.

Recursive grep for '$Z': in tomcat-dir




Recursive grep for '$Z': in my project's dir:



There is a Realm defined in server.xml, but the invalid URL is nowhere to be seen:




Since there's no '%$Z' in my files, can I assume the problem is somewhere else?

BR
Reidar Gjerstad
 
Reidar Gjerstad
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:Oh, and by the way - why did you modify conf/context.xml?



It is for historical reasons. I got it working like that a looong time ago following instructions similar to this:

https://tomcat.apache.org/tomcat-9.0-doc/jndi-datasource-examples-howto.html

Then I never thought about experimenting and finding other ways to do it. Perhaps I should have :-)

BR
Reidar Gjerstad
 
Reidar Gjerstad
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
My Java version:

 
Reidar Gjerstad
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Downloaded and installed another jdk, but it made no difference. The version was

 
Tim Holloway
Saloon Keeper
Posts: 24283
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Uh-oh.

I think you took a wrong turning.

First, let me start by explaining deployment descriptors. Deployment descriptors are the meta-information used to properly deploy webapps. In JEE, each webapp has two: the server-dependent deployment descriptor and the server-independent deployment descriptor.

The server-independent deployment descriptor is the webapp's /WEB-INF/web.xml file, plus whatever dynamic equivalents might have been scooped up from annotations in the webapp's classes.

The server-dependent deployment descriptor is the Context.

You don't actually have to have an explicit Context. If you don't provide one, Tomcat will synthesize one internally. But an explicit Context allows you to define JDBC Connection Pools, JNDI Resources, security Realms, and the webapp URL context path itself. Also it permits you to direct the application's codeBase to an alternate location. Such as the target build directory in my Eclipse project or to a WAR file in /opt/com/mousetech/mywebaps/mywebapp.war

The Context can be located as a file named "META-INF/context.xml" in a WAR file. It can also be located as an XXXXX.xml file in /conf/Catalina/localhost, where the URL context path will be the same as the Context filename (minus the extension). This overrides the Context in the WAR, if there is one and that's handy for keeping a development context in the WAR and overriding it for production.

You can also put a Context as an XML element node in /conf/server.xml, but you should not. It's mostly a relic from old times.

And then there's conf/context.xml. You might notice that there's also a conf/web.xml file. These are the prototypes to which Tomcat will fall back when it cannot find what it needs in your actual deployment descriptors and in general they should not be modified (you'll note that context.xml does allow you to change it to disable the SER file option, though).

Also, the Context is a tomcat-specific (server-dependent) deployment descriptor. Any resemblance to deployment descriptors used by other JEE servers such as Wildfly and WebSphere are purely co-incidental. And in fact, typically different webapp servers use different filenames and/or locations so that they don't conflict. Finally, Context may be generated by the webapp server if you deploy a WAR via the webapp's administrative console GUI/webapp.


The web.xml file is more interesting, since when you submit a URL that doesn't map to a servlet, it's where Tomcat goes looking for guidance. The prototype web.xml takes unresolved URL paths that can map to WAR resource directories and produces a directory listing webpage. For paths that resolve to static resources such as javascript, css or image files, it locates and copies the static resource to the response stream. And if it can't find anything else to resolve with, it emits the "404 Not Found" page.

So, in short most people should only modify the conf/server.xml and leave the other files located in the conf directory alone (Exception noted below).



=====

Anyway on the grep[/search, don't forget that special symbols such as "%" and "$" are often magical unless properly escaped and thus might not match as expected. You might be better off scanning for "Realm", in fact.

Incidentally, the 2 Realm definitions in the default server.xml are the UserDatabaseRealm, which looks up userid/password and userid/role in the conf/tomcat-users.xml file (which incidentally, you can and often would edit), and the LockoutRealm that says that if the Realm(s) it's presiding over reject too many login attempts, that loginid will be locked out for a specified time.
 
Reidar Gjerstad
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:Uh-oh.

I think you took a wrong turning.

First, let me start by explaining deployment descriptors. Deployment descriptors are the meta-information used to properly deploy webapps. In JEE, each webapp has two: the server-dependent deployment descriptor and the server-independent deployment descriptor.

The server-independent deployment descriptor is the webapp's /WEB-INF/web.xml file, plus whatever dynamic equivalents might have been scooped up from annotations in the webapp's classes.

The server-dependent deployment descriptor is the Context.

You don't actually have to have an explicit Context. If you don't provide one, Tomcat will synthesize one internally. But an explicit Context allows you to define JDBC Connection Pools, JNDI Resources, security Realms, and the webapp URL context path itself. Also it permits you to direct the application's codeBase to an alternate location. Such as the target build directory in my Eclipse project or to a WAR file in /opt/com/mousetech/mywebaps/mywebapp.war

The Context can be located as a file named "META-INF/context.xml" in a WAR file. It can also be located as an XXXXX.xml file in /conf/Catalina/localhost, where the URL context path will be the same as the Context filename (minus the extension). This overrides the Context in the WAR, if there is one and that's handy for keeping a development context in the WAR and overriding it for production.

You can also put a Context as an XML element node in /conf/server.xml, but you should not. It's mostly a relic from old times.

And then there's conf/context.xml. You might notice that there's also a conf/web.xml file. These are the prototypes to which Tomcat will fall back when it cannot find what it needs in your actual deployment descriptors and in general they should not be modified (you'll note that context.xml does allow you to change it to disable the SER file option, though).

Also, the Context is a tomcat-specific (server-dependent) deployment descriptor. Any resemblance to deployment descriptors used by other JEE servers such as Wildfly and WebSphere are purely co-incidental. And in fact, typically different webapp servers use different filenames and/or locations so that they don't conflict. Finally, Context may be generated by the webapp server if you deploy a WAR via the webapp's administrative console GUI/webapp.


The web.xml file is more interesting, since when you submit a URL that doesn't map to a servlet, it's where Tomcat goes looking for guidance. The prototype web.xml takes unresolved URL paths that can map to WAR resource directories and produces a directory listing webpage. For paths that resolve to static resources such as javascript, css or image files, it locates and copies the static resource to the response stream. And if it can't find anything else to resolve with, it emits the "404 Not Found" page.

So, in short most people should only modify the conf/server.xml and leave the other files located in the conf directory alone (Exception noted below).



=====

Anyway on the grep[/search, don't forget that special symbols such as "%" and "$" are often magical unless properly escaped and thus might not match as expected. You might be better off scanning for "Realm", in fact.

Incidentally, the 2 Realm definitions in the default server.xml are the UserDatabaseRealm, which looks up userid/password and userid/role in the conf/tomcat-users.xml file (which incidentally, you can and often would edit), and the LockoutRealm that says that if the Realm(s) it's presiding over reject too many login attempts, that loginid will be locked out for a specified time.



Hi Tim.

Thank you for the thorough explanation. I passed the SCWCD exam some time around 2010. I remember some details, but in addition to being rusty my understanding does not go as deep as yours.

I ran the grep over again. It does find the "$Z" in the logs. Doesn't it mean that it would find it if it were present in any xml or jsp files...?



It also finds "$Z" if I paste it into a test html file:



In addition there are binary files (JARs, PDFs) that contain the "$Z".

This makes me puzzled:
-I have written this web app myself.
-I can not remember having put any URL into any file with that kind of "$Z" content
-All URLs that exist are in JSPs and refer to other JSPs or servlet mount patterns (.do links)
-There is only one exception to the above, a simple JSP test page that tests the database connection. It doesn't contain the "$Z" sequence, either.
-The URL for the database connection is specified in conf/context.xml. I can't remember having any other URL for the database.

Am I searching the wrong place for the wrong thing? The offending "$Z" must be in a text file, is it so?

BR
Reidar Gjerstad




 
Reidar Gjerstad
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:In XML, unsupported attributes are ignored, so don't expect that to help.

I cross-check the JRE source code. It's pretty definite. Something was attempting to parse a URL and an escape sequence was found in the URL. For example, the "%20" in "https://coderanch.com/messages/my%20message".

In your case, however, it looks like the URL contained "%$Z" and since "$Z" is obviously not a hexadecimal number code, it threw an internal NumberFormatException.

If you can't see anything like that anywhere in your failing Tomcat files, then you may have hardware problems or a corrupt VM.

One way to find out for sure would be to launch Tomcat as a remote debugging session and set a breakpoint on com.mysql.cj.exceptions.WrongArgumentException. I'd actually recommend catching the NumberFormatException itself, but unfortunately, there are jokers out there who deliberately toss around common exception types in their product code and you might get inundated with them.

Also, if you could post the Realm XML it would help, since that's where the Exception actually displays.



You mentioned that I may have a corrupt VM. I installed another complete JDK, but that did not make any difference:



BR
Reidar Gjerstad

 
Tim Holloway
Saloon Keeper
Posts: 24283
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Please don't quote me wholesale. It takes up lots of space and makes things harder to read.

Get rid of all changes to conf/context.xml. It's the underpinning for all other other contexts and will be applied to every webapp deployed on the server. Also, despite its name, I don't know if it actually gets handled the same way as the "real" Context definitions. so I won't guarantee what happens.

What you need is either a META-INF/context.xml file in your WAR or a Context file in conf/Catalina/localhost/. Put your Connection pool definition there.

The problem with searching for "$Z" is all the false hits you'll get. The Java compiler takes inner class names like "StandOn.Zanzibar" and translates them to "StandOn$Zanzibar", for example. That's why I recommended searching for Realm. Because your stack trace is claiming that it's attempting to set up a JDBCRealm.
 
Reidar Gjerstad
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Sorry for quoting too much. Will cut down.

I will try the things you suggest, one by one.

In addition to Tomcat's default test app, there is only one app on this server (local PC).

Is there a chance that Tomcat 9 simply won't work with the MySQL version that I have locally?



It appears that Tomcat 6 and Tomcat 9 have exactly the same instructions for which versions will work:

"Versions of MySQL and JDBC drivers that have been reported to work:

   MySQL 3.23.47, MySQL 3.23.47 using InnoDB,, MySQL 3.23.58, MySQL 4.0.1alpha
   Connector/J 3.0.11-stable (the official JDBC Driver)
   mm.mysql 2.0.14 (an old 3rd party JDBC Driver)"

BR
Reidar Gjerstad
 
Tim Holloway
Saloon Keeper
Posts: 24283
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
These are the two most important lines in your stacktrace:


The first line indicates that the error was seen while attempting to create a security Realm - not a database Connection pool.

The second line indicates that the MySQL URLDecoder found an illegal URL character sequence ("$%Z") in the connection URL. The actual exception happened when the MySQL ConnectionUrlParser invoked java.net.URLDecoder.decode() which is in the JDK core class collection (not MySQL or Tomcat).

By way of example, this is what a valid JDBCRealm definition looks like:

However, your stack trace reads like it was coded more like this:

Unfortunately, while the full URL that offended is included in internal Exception information, it's not printed out to the log.

if you cannot find the bad Realm using grep, then about the only other suggestion I could give would be to hook a debugger up to Tomcat and set a trap for SQLException, since the Realm will construct one when the MySQL driver throws its  SQLNonTransientConnectionException.
 
Reidar Gjerstad
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Your last post brought me a little forward, but did not resolve the issue. I had of course forgotten to define the security Realm in server.xml (embarrassing) and it was in a different form:



I modified to use your suggested form



Unfortunately, it did not solve the problem. Exception still comes up.
 
Tim Holloway
Saloon Keeper
Posts: 24283
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Break out the debugger.
 
You showed up just in time for the waffles! And this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
reply
    Bookmark Topic Watch Topic
  • New Topic