With love, from Mother Russia

Eat a bag of dicks, Russia.

Last evening, I was alerted by one of my web server tenants (Lydia) that the web server may be down. “Impossible!” I thought to myself. My CentOS web server has been extremely stable and I’ve taken great pains to secure it (that’s not a challenge, by the way, leave me in my happy ignorance). In the past, usually this has meant that our connection has gone flakey or the router needed to be rebooted. But this time, our connection checked out as I was able to resolve external web sites, just not our hosted sites. hm

I attempted to SSH to the CentOS VM from within my network, and no love. Again, hm. I popped open a console to the VM and sure enough it was bogged down, unresponsive. Well that’s not great. My first inclination was that the disk could be full, as web devs aren’t particularly careful when uploading JPEGs for their sites. Need an 300x200 JPEG on your wordpress page? Ah yeah, this 3176x1571 image will do. I rebooted the VM and watched it come back up with no issues. I logged in and ran df -h. Only 57% disk utilized - so not the disk. Then before I could run anything else, the VM started locking up again. Whiskey Tango Foxtrot?

Another reboot, this time to runlevel 3, and I quickly tail -f /var/log/messages where I see complaints about httpd eating all my memory, then the VM locks up again. This ain’t good. Another reboot, this time runlevel 1 (shit’s getting serious) and I dig into /var/log/httpd/error_log. Sure enough, I see [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 16 children, there are 41 idle, and 129 total children. I think, “that’s odd, I’m only running four virtual hosts, all of which are not heavily trafficked.” I shutdown port 80 on the external firewall and run init 3 to bring all services up. Server seems stable and I can access sites internally (after a bit of fumbling, since I use the ServerName directive for directing traffic, which works all well and good when DNS resolves and the sites are available externally. Not so much internally without host files). The server remains stable - this stinks of a DDoS. But why? Who would waste the effort in DDoSing matt5lot10.com (although I realize it is one of the hottest sites on the web). A quick peak in /var/log/httpd/access_log gives me my answer. My DDoS attacker is 191.96.249.80. A quick ip-tracker.org lookup shows Moscow, Ruskie. Whiskey Tango Foxtrot, seriously?

My first inclination was to just block the one IP on the firewall, but after having to downgrade my lab after moving last year, I didn’t have PFSense to do so and my firewall device leaves some features to be desired… (DDWRT may be in my future). So looks like we’ll need to rely on good ol’ iptables. And f*ck that IP, I’m blocking all of Russia and China - no matt5lot10 for you! A bit of googling as to how to best blacklist entire countries turned up this Mattwilcox blog post which uses handy, dandy ipdeny.com to create an ipset which can be blocked in iptables. I had to modify his scripts a bit (it looks like he uses a Debian-based Linux and my bash scripts have a bit more error checking to ensure a zone file exists in /etc before attempting to delete it.

Sure enough, once the ipsets were in place and iptables deny rules implemented, Apache is again swimming along and the DDoS attacker(s) can eat a bag of dicks.

So why DDoS matt5’s IP? Who knows - probably some Vodka-soaked Ruskie had too much free time on his hands and is just pointing H.O.I.C at US IPs. What’s more concerning is it could be state-sponsored Vodka-soaked Ruskies carrying out these and other attacks - it probably explains how Trump was elected.

Either way, no more matt5lot10 for China or Russia, they’ve lost their privilege of matt5lot10.com.