Granny's Programming Pearls
"inside of every large program is a small program struggling to get out"
JavaRanch.com/granny.jsp
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Jeanne Boyarsky
  • Devaka Cooray
  • Paul Clapham
Sheriffs:
  • Tim Cooke
  • Knute Snortum
  • Bear Bibeault
Saloon Keepers:
  • Ron McLeod
  • Tim Moores
  • Stephan van Hulst
  • Piet Souris
  • Ganesh Patekar
Bartenders:
  • Frits Walraven
  • Carey Brown
  • Tim Holloway

fake.cfg in tmp directory and lot of out bound traffic

 
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

We are using a Amazon EC2 instance for a Java and Tomcat based web application. Recently we noticed a sudden spike in out bound traffic and it was being generated from a process named "nodewx" which resided in "/tmp" directory. Same directory also contained "fake.cfg" file. On googling around I found out there is a known vulnerability in older tomcat versions which is associated with fake.cfg file.

We are using Java 1.7 update 51 and Apache Tomcat/7.0.27. We have created a separate tomcat user and launching tomcat through JSVC.

I would like to know what should I do to protect tomcat and my Server in order to avoid this issue again?

-----
Edit
-----

I just looked at the access logs and they show this story

173.45.75.58 - - [05/Feb/2014:02:26:23 +0000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 952
173.45.75.58 - - [05/Feb/2014:02:26:23 +0000] "GET /phpMyAdmin/scripts/setup.php HTTP/1.1" 404 952
173.45.75.58 - - [05/Feb/2014:02:26:23 +0000] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 404 952
173.45.75.58 - - [05/Feb/2014:02:26:23 +0000] "GET /pma/scripts/setup.php HTTP/1.1" 404 952
173.45.75.58 - - [05/Feb/2014:02:26:24 +0000] "GET /myadmin/scripts/setup.php HTTP/1.1" 404 952
173.45.75.58 - - [05/Feb/2014:02:26:24 +0000] "GET /MyAdmin/scripts/setup.php HTTP/1.1" 404 952
157.56.93.49 - - [05/Feb/2014:04:21:31 +0000] "GET /robots.txt HTTP/1.1" 404 952
157.56.93.49 - - [05/Feb/2014:04:24:04 +0000] "GET / HTTP/1.1" 200 11264
192.71.151.187 - - [05/Feb/2014:09:17:18 +0000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 952
192.71.151.187 - - [05/Feb/2014:09:17:19 +0000] "GET /phpMyAdmin/scripts/setup.php HTTP/1.1" 404 952
192.71.151.187 - - [05/Feb/2014:09:17:19 +0000] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 404 952
192.71.151.187 - - [05/Feb/2014:09:17:19 +0000] "GET /pma/scripts/setup.php HTTP/1.1" 404 952
192.71.151.187 - - [05/Feb/2014:09:17:19 +0000] "GET /myadmin/scripts/setup.php HTTP/1.1" 404 952
192.71.151.187 - - [05/Feb/2014:09:17:19 +0000] "GET /MyAdmin/scripts/setup.php HTTP/1.1" 404 952
162.243.231.9 - - [05/Feb/2014:13:14:46 +0000] "HEAD / HTTP/1.1" 200 -
69.90.132.223 - - [05/Feb/2014:14:32:24 +0000] "GET / HTTP/1.1" 200 11264
211.141.27.243 - - [05/Feb/2014:15:17:42 +0000] "GET /manager/html HTTP/1.1" 401 2486
211.141.27.243 - manager [05/Feb/2014:15:17:43 +0000] "GET /manager/html HTTP/1.1" 200 19059
173.244.206.13 - - [05/Feb/2014:15:37:39 +0000] "GET / HTTP/1.0" 200 11244
180.140.25.158 - - [05/Feb/2014:15:50:49 +0000] "GET /manager/html HTTP/1.1" 401 2486
180.140.25.158 - manager [05/Feb/2014:15:50:54 +0000] "GET /manager/html HTTP/1.1" 200 17563
180.140.25.158 - manager [05/Feb/2014:15:50:55 +0000] "GET /manager/images/tomcat.gif HTTP/1.1" 200 2066
180.140.25.158 - manager [05/Feb/2014:15:50:55 +0000] "GET /manager/images/asf-logo.gif HTTP/1.1" 200 7279
180.140.25.158 - - [05/Feb/2014:15:50:59 +0000] "GET /favicon.ico HTTP/1.1" 404 952
180.140.25.158 - - [05/Feb/2014:15:51:00 +0000] "GET /favicon.ico HTTP/1.1" 404 952
180.140.25.158 - manager [05/Feb/2014:15:51:02 +0000] "POST /manager/html/upload?org.apache.catalina.filters.CSRF_NONCE=D55BBC344D43A670EE4D4112C193504B HTTP/1.1" 200 19313
180.140.25.158 - - [05/Feb/2014:15:51:05 +0000] "GET /hosts%2Dmanager HTTP/1.1" 302 -
119.147.146.189 - - [05/Feb/2014:15:51:06 +0000] "GET /hosts-manager HTTP/1.1" 302 -
180.140.25.158 - - [05/Feb/2014:15:51:09 +0000] "GET /hosts-manager/ HTTP/1.1" 200 3310
119.147.146.189 - - [05/Feb/2014:15:51:09 +0000] "GET /hosts-manager/ HTTP/1.1" 200 3310
180.140.25.158 - - [05/Feb/2014:15:51:14 +0000] "POST /hosts-manager/ HTTP/1.1" 200 6405
180.140.25.158 - - [05/Feb/2014:15:51:16 +0000] "GET /hosts-manager/?action=command HTTP/1.1" 200 2687
180.140.25.158 - - [05/Feb/2014:15:51:18 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 2720
180.140.25.158 - - [05/Feb/2014:15:52:08 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 2732
180.140.25.158 - - [05/Feb/2014:15:53:24 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 2709
180.140.25.158 - - [05/Feb/2014:15:53:28 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 2704
180.140.25.158 - - [05/Feb/2014:15:53:52 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 2736
180.140.25.158 - - [05/Feb/2014:15:53:56 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 4765
101.226.65.104 - - [05/Feb/2014:16:21:05 +0000] "GET /hosts-manager HTTP/1.1" 302 -
101.226.65.104 - - [05/Feb/2014:16:21:06 +0000] "GET /hosts-manager/ HTTP/1.1" 200 3310
157.56.93.84 - - [05/Feb/2014:16:45:23 +0000] "GET /robots.txt HTTP/1.1" 404 952
157.56.93.84 - - [05/Feb/2014:18:05:24 +0000] "GET / HTTP/1.1" 200 11264
220.194.196.102 - - [05/Feb/2014:19:55:53 +0000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 952
220.194.196.102 - - [05/Feb/2014:19:55:54 +0000] "GET /phpMyAdmin/scripts/setup.php HTTP/1.1" 404 952
220.194.196.102 - - [05/Feb/2014:19:55:55 +0000] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 404 952
220.194.196.102 - - [05/Feb/2014:19:55:55 +0000] "GET /pma/scripts/setup.php HTTP/1.1" 404 952
220.194.196.102 - - [05/Feb/2014:19:55:56 +0000] "GET /myadmin/scripts/setup.php HTTP/1.1" 404 952
220.194.196.102 - - [05/Feb/2014:19:55:57 +0000] "GET /MyAdmin/scripts/setup.php HTTP/1.1" 404 952
209.126.230.78 - - [05/Feb/2014:21:25:35 +0000] "GET / HTTP/1.0" 200 11244
109.68.190.145 - - [05/Feb/2014:22:09:11 +0000] "GET / HTTP/1.0" 200 11244
59.37.154.80 - - [05/Feb/2014:22:24:09 +0000] "GET //cgi-bin/php HTTP/1.1" 404 952
59.37.154.80 - - [05/Feb/2014:22:24:12 +0000] "GET //cgi-bin/php5 HTTP/1.1" 404 952
59.37.154.80 - - [05/Feb/2014:22:24:12 +0000] "GET //cgi-bin/php-cgi HTTP/1.1" 404 952
59.37.154.80 - - [05/Feb/2014:22:24:13 +0000] "GET //cgi-bin/php.cgi HTTP/1.1" 404 952
59.37.154.80 - - [05/Feb/2014:22:24:13 +0000] "GET //cgi-bin/php4 HTTP/1.1" 404 952

Thanks
 
Haris Hasan
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So it seems like Tomcat Manager is being used by the attacker. First step for me would be to remove the tomcat manager from web apps
 
Bartender
Posts: 20838
125
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Welcome to the JavaRanch, Hasan!

If that's your post asking the same question over on stackoverflow, please be advised that we prefer people to announce when they are posting the same question to multiple places. It helps keep things from getting too confused.

I did a little digging on the topic and it looks like what happens in such events is that an intruder has managed to install a webapp of their own for nefarious purposes.

There are several ways to install WARs in Tomcat. One is to use the Tomcat manager webapp/web services. By default, the necessary functions are not authorized, so it's worth checking to see if on this machine they have been authorized and to whom. And to change the passwords of those users.

The "fake.cfg" thing is especially worrisome, since it's not part of a WAR. It means it's possible that the malWAR was installed via OS filesystem access, unless the intruder created it as some sort of marker after installation. I have no idea what use that file could possibly serve other than as an indicator that the system is already infected.

In any event, I'd also check the OS login accounts to see if any of them have been compromised.

Of course, times being what they are, and this being Amazon, it may just be something the NSA added, but that's a different matter.
 
Haris Hasan
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank for the reply. (I have removed the post from stackoverflow)

If I understand you correctly, you are saying that it cannot be concluded from the access log I posted that attacker used Tomcat Manager only for the attack? We are not using tomcat manager for deployment and the IPs in the access log are not one of ours.

I believed it was done through Tomcat Manager

180.140.25.158 - - [05/Feb/2014:15:51:14 +0000] "POST /hosts-manager/ HTTP/1.1" 200 6405
180.140.25.158 - - [05/Feb/2014:15:51:16 +0000] "GET /hosts-manager/?action=command HTTP/1.1" 200 2687
180.140.25.158 - - [05/Feb/2014:15:51:18 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 2720
180.140.25.158 - - [05/Feb/2014:15:52:08 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 2732
180.140.25.158 - - [05/Feb/2014:15:53:24 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 2709
180.140.25.158 - - [05/Feb/2014:15:53:28 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 2704
180.140.25.158 - - [05/Feb/2014:15:53:52 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 2736
180.140.25.158 - - [05/Feb/2014:15:53:56 +0000] "POST /hosts-manager/?action=command HTTP/1.1" 200 4765

We had already disabled password based authentication and we only use SSH for login.

Furthermore, we found that attacker was able to launch a process named "nodewx" and it was sending the out bound traffic. This process was launched by user "Tomcat", which is our dedicated user for handling tomcat. Plus the /tmp directory contained the nodewx file along with fake.cfg. To me all these pieces hints that Tomcat was used for the attack.
 
Tim Holloway
Bartender
Posts: 20838
125
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It's not necessary to delete the stackoverflow posting. We just like to know if there are other places where the question is also being asked. That way people can sort of home in on a single place instead of running around ignorant of each other.

That IP address tracks back to the People's Republic of China, so I think that you can safely conclude it's an unauthorized intruder.

To prevent further abuse, take a look at the security context definition for the tomcat manager webapp - or if one is lacking, the default Realm for Tomcat as a whole. That will tell you where it's getting its account information from, including userids, security riles, and passwords. Once you have that, lock them out.
 
Haris Hasan
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks
 
Rancher
Posts: 43009
76
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Start by removing all web apps that you are not using. The access log indicates that both /manager and /hosts-manager were accessed successfully (response status 200). It also looks as if only "/manager" requires authentication, not "/hosts-manager".

"/hosts-manager" is not a standard Tomcat web app, though - is that part of your software? Or is it just the renamed "host-manager" web app?
 
Haris Hasan
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
We have removed all the websites like manager except our own which is under ROOT. We are deploying without manager any ways. I believe it should fix the issue
 
Tim Holloway
Bartender
Posts: 20838
125
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I'd still go in and make sure that there aren't any unsecured security Realms floating around, though.

Just a little added insurance.
 
Haris Hasan
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Can you kindly shed some light on "unsecured security Realms floating around". I am new to linux and security world.
i really appreciate your help
 
Tim Holloway
Bartender
Posts: 20838
125
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What I was referring to wasn't Linux security, it was Tomcat security.

The J2EE and JEE standards define a container-managed security system and that system is used by the Tomcat Manager webapp. In fact, with very rare exceptions, I recommend that it be used with every webapp, since most user-designed security systems are about as secure as a wet paper wall, and that includes quite a few financial and even military apps I've encountered. Application developers don't have the training or the time to do real security.

In Tomcat, the security is defined by a Realm. Realms are plug-in modules that allow you to keep your credentials and role information in a wide variety of locations, including databases, LDAP/Active Directory, the conf/tomcat-users.xml file and other places, as desired. I've written custom Realms that use web services as their backing store.

There's usually a sample Realm or 2 commented out in the TOMCAT_HOME/conf/server.xml file. If you find a Realm defined in there that isn't commented out, you may want to de-activate it just to be safe. It's not a bad idea to remove any userids that exist in the tomcat-users.xml file, for that matter.

 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!