Missing web.xml

[Roger F. Gay]

Tomcat 8.0.9
java version "1.8.0_11"
Java(TM) SE Runtime Environment (build 1.8.0_11-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.11-b03, mixed mode)

The web.xml file for my ROOT application disappeared. I don't want to start looking for the cause by imagining that someone at the business site came in and deleted it from the machine.

This is an application that's been running for more than a year, so this isn't a basic setup problem. I've more recently upgraded to Tomcat 8 and Java 8; but that was about a month ago. No other problems in that time.

I don't mean that web.xml wasn't found when running. I literally mean that when I looked into why the app suddenly stopped working properly (when it tried to use a servlet) this morning, web.xml was physically not there - it was gone.

I have the whole system backed up, so I got it running again in seconds after seeing that web.xml was gone. But gee-willikers. I have to wonder whether there's any bug or security vulnerability that could be involved here.

I don't know if it matters, but we also had power problems in the building about the time this happened. Since I discovered the problem first thing in the morning, I can't be sure whether there was exact correspondence between these events.

Any thoughts?


ADDED: Checking logs, found that ContainerBackgroundProcessor had been invoked early that morning (while I was sleeping). This apparently is connected to being able to change the app code on the fly, while the server is still running.

[Jaikiran Pai]

It's no longer mandatory in recent versions of Java EE to have a web.xml for a web application (like ROOT.war). Are you sure there was one in that application? Tomcat 8 is the latest version so they might have stopped shipping the web.xml, I don't have one to verify right now.

[Roger F. Gay]

I'm using servlets, so I have a web.xml. There might be a way to avoid using it, but I guess I'm just old fashioned. (I wrote the app.)

[Tim Holloway]

Tomcat does have an annoying habit of deleting the entire webapp if it cannot be started, but no, it isn't going to be selectively erasing files, and especially not the protected files in WEB-INF and its subdirectories. In fact, a properly-designed webapp doesn't even contain application code that does write, update, or delete operations on the WAR.

My best guess is a filesystem hiccup of some sort. Keep an eye out (and maintain your backups!) in case it happens again.

And, BTW, one of the things that Tomcat 7 (and 8) add is a java annotation for servlets, so no, they no longer must be defined in a web.xml file.

[Roger F. Gay]

Thanks. My backups were up-to-date. It's good to be obsessive sometimes. After a year of no problems, while making changes from time to time, I might have needed this to feel the importance.

I'll take some time to look at annotations. I started using Java / Tomcat several versions ago and haven't kept up on absolutely everything.

I think I'll psychologically accept your answer (rather than continue to worry "obsessively"). There were other problems going on around that time. The computer should still be good hopefully.

[Roger F. Gay]

As long as I'm looking into annotations, might be worth mentioning that I'm using the web.xml file for more than specifying servlets. Other uses include:

UrlRewriteFilter - filter path is defined. UrlRewriteFilter from tucky.org is set up to always show the web address without www. (301 redirects)
error-page specification - specifies what to run when certain http errors are encountered
resource-ref

All these things can be handled by annotations?

[Roger F. Gay]

Server went down yesterday. web.xml was missing again, and there's something else. I still have a copy of the whole Tomcat 7 system on the computer, not running. Since I first posted, web.xml went missing from both versions.

This time, we also had an out-of-memory problem. There's still the suggestion that something is going wrong with the computer. But these missing web.xml files don't seem random.

[Roger F. Gay]

15-Aug-2014 23:31:25.805 WARNING [http-nio-80-exec-38] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "tomcat"
15-Aug-2014 23:31:43.106 WARNING [http-nio-80-exec-18] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "tomcat"
15-Aug-2014 23:34:45.735 WARNING [http-nio-80-exec-43] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "manager"
15-Aug-2014 23:36:42.564 WARNING [http-nio-80-exec-18] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:37:02.173 WARNING [http-nio-80-exec-13] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:37:19.739 WARNING [http-nio-80-exec-40] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:37:43.841 WARNING [http-nio-80-exec-38] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:38:09.472 WARNING [http-nio-80-exec-43] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:38:29.986 WARNING [http-nio-80-exec-18] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:38:47.021 WARNING [http-nio-80-exec-13] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:39:03.073 WARNING [http-nio-80-exec-40] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"

[Tim Holloway]

Normally that would indicate that someone's either trying to hack the server or you have automated control processes whose passwords are out of date.

[Roger F. Gay]

catalina stderr has a lot more warnings of a similar nature, starting with about a hundred trying to get into admin

...
14-Aug-2014 19:40:26.855 WARNING [http-nio-80-exec-6] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "admin"
14-Aug-2014 19:40:27.355 WARNING [http-nio-80-exec-8] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "admin"
15-Aug-2014 23:31:25.805 WARNING [http-nio-80-exec-38] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "tomcat"
15-Aug-2014 23:31:43.106 WARNING [http-nio-80-exec-18] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "tomcat"
15-Aug-2014 23:34:45.735 WARNING [http-nio-80-exec-43] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "manager"
15-Aug-2014 23:36:42.564 WARNING [http-nio-80-exec-18] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:37:02.173 WARNING [http-nio-80-exec-13] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:37:19.739 WARNING [http-nio-80-exec-40] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:37:43.841 WARNING [http-nio-80-exec-38] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:38:09.472 WARNING [http-nio-80-exec-43] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:38:29.986 WARNING [http-nio-80-exec-18] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:38:47.021 WARNING [http-nio-80-exec-13] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"
15-Aug-2014 23:39:03.073 WARNING [http-nio-80-exec-40] org.apache.catalina.realm.LockOutRealm.authenticate An attempt was made to authenticate the locked user "root"

[Tim Holloway]

Try putting in a request valve to print out the source IPs of these attempts. If you know where they are coming from, you'll have a better idea of how to fix or firewall them.

[Roger F. Gay]

My first Valve. How'm I doin?

   <Valve
      className="org.apache.catalina.valves.AccessLogValve"
      directory="${catalina.base}/logs"
      prefix="access_log"
      resolveHosts="true"
      fileDateFormat="yyyy-MM-dd.HH"
      suffix=".log"
      pattern="%t %h cookie:%{SESSIONID}c request:%{SESSIONID}r  %a %q %r %u %U %I"
    />

[Roger F. Gay]

Valve: It's a start based on an example I found ... needs a little fixing. Currently generating a new file for every click. I am getting IP information, without domain name lookup at the moment.

Meanwhile: I've removed all the servlets from web.xml by using annotations.
JDBC resource reference can be removed - coding out the resource approach.
That just leaves Tuckey's UrlRewriteFilter, which doesn't come with source code and I don't have a work around.

UrlRewriteFilter rewrites the web address, removing www. any time that it's used. (This is Tomcat)
I could do that in the web pages with JavaScript but that loads and reloads the page ... not nice.

[Roger F. Gay]

Actions taken against attempted break-in:

Reported through abuse email to both our Internet Provider and the telecom company of the offender.

Added the following valve to Context.xml

<Valve className="org.apache.catalina.valves.RemoteAddrValve" deny="79.188.115.98" />

[Tim Holloway]

You're kinder than I am. I would have blocked the IP at the firewall.

If they're trying to break in via Tomcat, they may well be trying to break into other services, too.

[Roger F. Gay]

Tim Holloway wrote:You're kinder than I am. I would have blocked the IP at the firewall.

If they're trying to break in via Tomcat, they may well be trying to break into other services, too.

Done!

[Roger F. Gay]

I'd still like to know how the web.xml files went missing - the first sign that prompted this activity.
What I have in the logs are apparently automated attempts to get into the server via the manager. (many attempts, one second apart)
I don't really see how that would have resulted in missing application web.xml files.

[Tim Holloway]

It's a mystery to me as well. Like I said, Tomcat will cheerfully delete an entire webapp, but selected parts?

What would be nice is if you had a tool that could detect the filesystem delete operation and point back to the offender, but if there's a generally available tool out there for that, I don't know what it is. For Linux, you could probably set up a dtrace, though.

[Roger F. Gay]

Tim Holloway wrote:It's a mystery to me as well. Like I said, Tomcat will cheerfully delete an entire webapp, but selected parts?

What would be nice is if you had a tool that could detect the filesystem delete operation and point back to the offender, but if there's a generally available tool out there for that, I don't know what it is. For Linux, you could probably set up a dtrace, though.

Windows Performance toolkit?

[Tim Holloway]

I'll let you check that one out.

I worked with Windows for the first time in months yesterday. My wife apparently powered off in the middle of about 400MB worth of critical system updates, got the updates into a broken state and damaged 2 IE files.

Some banging and shouting and Safe Mode stuff got the updates back on track, finally got the updates applied (4 hours later), and the system is now allegedly clean. So I noodled around just to make sure it was all usable.

Between anonymous "rundll"-style processes and anti-virus software, I swear that "Windows Performance" is an oxymoron, though.

[Roger F. Gay]

I disabled Windows Search (service) and one other that I found to be real performance killers. Did the same on my girlfriend's machine.
Searches run much slower, but I'm just living with that for the moment. It's a rare occasion when I need to search through everything on a whole disk.

[Roger F. Gay]

RESOLVED (I'm really pretty sure.)
Several huge print jobs originating from a scanner caused the printer to run out of memory and crash. The jobs got stuck in the print spooler.
Coincidentally (seeming to define the laws of probability) a lot of things happened all at once, providing a list of potential problems (like suddenly people were trying to hack the system and others).
The busy circle on the cursor kept flashing over and over and over all the time; but I decided that was just an irritant and lower priority than the web app crashing.
Should have paid attention to the circle. It led me to the problem. It was telling me that something was happening over and over again.
The print spooler kept trying to deal with the print jobs that were stuck; and apparently IT HAS A MEMORY LEAK! (Windows 7)
It was slowing everything down, and would eventually run out of memory and the whole system crashed.
That's why I couldn't get the web application running again by re-starting the system remotely. No memory. Windows crashed. Needed to go in and do a hard re-boot each time. But the print spooler is persistent ... storing jobs on disk, and the cycle started over again after every computer restart.

FIX: Re-started the print spooler service and deleted the job files from the print spooler directory. Everything's back to normal - very fast and reliable web app.
This really looks like a totally confirmed fix - everything is back to normal. So, what happened to the web.xml files? I'm pretty sure they just didn't survive crashes when encountering the out-of-memory problem.


Side note: The length of time with this problem stems from my long career in big companies where everyone is specialized. I'm with a small company now (love it) and we can't just call a specialist when there's a problem. A Windows maintenance guy (specialized) or someone with a history of fixing all their own problems would have nailed this much more quickly.

Almost unrelated side note: Speaking of specialization, I've finally been successful providing external access to the server on my home machine through a cheap, cheesy Chinese mobile router. I've been running web servers on my home machines for years but this is the first time through a cheap, cheesy little Chinese mobile router. Apparently, I did it right the first time - but this is the first time I've encountered the loop back problem. I'm upgrading to a paid account on noip.com. I've always had a high regard for DynDNS, but noip:s support is better. Thanks noip!