• Post Reply Bookmark Topic Watch Topic
  • New Topic

Cloud server Eth0 usage shot through the roof; anyone have any advice on figuring out why?  RSS feed

 
Alex Lieb
Ranch Hand
Posts: 61
3
Java Netbeans IDE Notepad
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi!

So uh... I work a place, and at that place we rent out some cloud servers and put things on them. I set up some stuff and put some projects out on these two cloud servers several months ago, and one of them had a problem this morning.

We got an email from the billing department of the server people saying we exceeded one of the resource limit thresholds we'd set; and it gave us this:

Port Eth0 Overusage Alert
Time: 2016-06-07 06:00:00
Eth0 Threshold: 90%
Eth0 Usage: 683813697420.09%

So what I'm getting out of this is we told the server it was allowed to use 1 Internets, and it used 6.8 billion internets. I can't find anything unusual in the log files around that time; there's no successful logins from any users who shouldn't be logging in (or from any users, actually), and in fact there doesn't appear to be any evidence that *anything* unusual happened at or around that time. I've checked the secure log, the message log, tomcat logs, tomcat access logs, httpd logs, mail logs, cron logs, basically every log in the /var/log directory, but I can't find any evidence of anything strange happening.

Does anyone have any idea what might cause this sort of thing? Or for that matter does anyone have any idea what this sort of thing that happened is?
 
Tim Holloway
Saloon Keeper
Posts: 18795
74
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
"eth0" is not a "port", it's an interface.

Interfaces communicate on ports using protocols - for example, port 80/tcp for http, port 53/udp for DNS.

If your traffic stats are being exceeded, then chances are that you are either being DDOS'ed or that your server has been pwned and co-opted into DDOS'ing someone else. Hopefully your cloud ISP would have told you specifically if you were spamming out email, but there are several other nasty attacks that your server could be aiding and abetting. I have been on the receiving end of several types of UDP reflection attacks where the massed zombie armies of the Internet would periodically assault my DNS and NTP services.

To determine what's contributing to this overuse, you need to apply the proper tools. The tcpdump utility can capture traffic logs for analysis by Wireshark. I also like iptraf, which gives me real-time displays - although to avoid distortion you have to be careful NOT to monitor your ssh port with it if you run iptraf over ssh (it's not fatal, but the network traffic used by ssh to keep the iptraf display updated can distort your view).

Once you know the port, ip address and direction of the excessive traffic you can look for the culprit.
 
Alex Lieb
Ranch Hand
Posts: 61
3
Java Netbeans IDE Notepad
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It's an interface... Alright; thanks for clearing that up, that was throwing me off... For the record though I didn't write those details; this part:

> Port Eth0 Overusage Alert
> Time: 2016-06-07 06:00:00
> Eth0 Threshold: 90%
> Eth0 Usage: 683813697420.09%

Is copied directly out of the email they sent us O_o
We haven't been able to find any evidence that our site has been pwned, but I'll keep looking into that... It's possible but not probable that we're being DDoS'ed, I think... I think? Unless someone just really hates our hosting provider for some reason. In any event it happened again this morning, at the same time, and the same "Eth0 Threshold", but the "Eth0 Usage" was at the less absurd level of 561%.

Anyway, the weird thing about this is we can't find any evidence of any unusual activity on any of our logs for yesterday or today, and we checked our Eth0 usage on our hosting company's portal site, and it says usage has been at 0 the entire time. Sooooo I'm confused now... My boss is thinking it might be some kind of measurement error on their side now, and in any case he's baffled as well. I emailed our hosting company's support team about it, so we'll see what they say back when they get back to us on that, and in the meantime I'll look into that iptraf utility again...

Thanks!
 
Tim Holloway
Saloon Keeper
Posts: 18795
74
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
And welcome to the Ranch!

Take it as a given. When you open a door on the Internet, a whole world of Bad Guys will instantly try to charge in. My servers are hardly well-known or, for that matter, doing anything very important, but each of them bounces a couple of thousands of assaults every day - a problem made worse because my ISP only provides a fairly brain-dead Cisco router that I cannot add firewall rules to like I could on my old hookup. So I'm firewalling on a per-host basis.

The problem with infected systems is that a good infection will go to great lengths to hide itself. Which is why it's good to sic a wide variety of tools on it. But if you can narrow the traffic down to a particular port number, it's a lot easier to see what service offends.

One of the most despicable attacks I know of is the DNS Reflection attack. TCP messages establish a session, and therefore are effectively un-spoofable. But DNS operates primarily on UDP, which does not. So a one-shot hit-and-run approach can be taken. In the case of a reflection attack, the zombies hit my server with DNS lookup requests that themselves are short, but can be expected to return a lot of data (in other words, be amplified). By placing the IP address of the actual victim as the return address of the request, the zombies then are enlisting me as part of their attack, since in good faith, I return this amplified data burst to the victim server, not knowing that it isn't the true source of the request. It's not easy to defend against such attacks without turning off my own DNS servers. The best I could do was upgrade to a DNS version that would allow me to reject requests for DNS hosts not served by me. I still get pummeled by the inbound traffic, but at least now I no longer contribute to the problem.
 
Alex Lieb
Ranch Hand
Posts: 61
3
Java Netbeans IDE Notepad
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I see... Well we did narrow it down a little bit;

I'm not sure why it wasn't showing up earlier, but it turns out there is a spike in network traffic and CPU usage immediately after a restart, and we know that something running on Java is responsible for it, but we don't actually know what. I'm thinking I'd like to install that iptraf monitor thing, but I'm not sure if it would actually be able to catch anything because this traffic is going through immediately after a reboot, and I don't know what boot priority it would/should have... I'll definitely look into that now though, now that we've got it isolated down to basically "What Java thing is going nuts and sending out and receiving 70-80 MB of data every time we reboot?"

The fact that our incoming traffic reflects our outgoing traffic is also strange... If it's a reflection attack, it's a bad one, because nothing is getting amplified, it's close to a 1:1 ratio of incoming to outgoing traffic.
 
Tim Holloway
Saloon Keeper
Posts: 18795
74
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well that narrows it down considerably, but if you have it localized to Java, then you should have some idea which JVM is doing it (assuming that you're running more than one). And you should probably be able to start/stop it manually so that you have all your metering tools listening when it does it.

Failing that, there are definitely places you can start tcpdump logging long before any JVMs start up.
 
Alex Lieb
Ranch Hand
Posts: 61
3
Java Netbeans IDE Notepad
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hey!

So two things;

1) We are idiots. Ok, not really, but the reason this is happening is pretty great... So in our server manager panel we have a little thing we can use to notify us if we cross a certain threshold of resource usage. THAT is what is sending us the emails. WE SENT OURSELVES EMAILS AND PANICKED ABOUT THEM. Our hosting company is not sending us emails. Our hosting company is not charging us for resource overusage. Our hosting company *does not care.*

2) Their traffic software has a bug. They did get back to us and let us know what was happening; the thing that keeps track of traffic does it by counting how much stuff you've sent and comparing it every 5 minutes.

So say you start the server at 5:00 on Tuesday, and it sends about 1 byte per minute for some reason, and you want it to alert you if it ever sends more than 10.

At 5:05 the logger will say "The difference between 0 and 5 is 5. In the past 5 minutes, 5 bytes were sent."
At 6:05 the logger will say "The difference between 60 and 65 is 5. In the past 5 minutes, 5 bytes were sent."

You wait three weeks, and restart the server. It resets the statistic for total bytes sent, so the traffic monitor it knows that when it last checked in, it had sent 30240 bytes, but now it's sent 0 bytes.

"The difference between 30240 and 0 is 30240. In the past 5 minutes, 30240 bytes were sent. I think that might be over 10... I should sent a notification. HEY MAN, FYI YOU USED 302400% OF YOUR INTERNET RESOURCES."

Thanks for your help though! I never did install that traffic monitor, but I'll keep it on the back burner.
 
Tim Holloway
Saloon Keeper
Posts: 18795
74
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator


I had to give you a cow for that, just for entertainment value.
 
Alex Lieb
Ranch Hand
Posts: 61
3
Java Netbeans IDE Notepad
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks!
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!