Hacked again - lessons and next steps

My server got hacked once again. I’m not sure what I’ve done to attract this kind of unwanted attention – I’m doing all the obvious things like keeping the system patched, passwords secure, moved SSH to a non-standard port, monitoring the logs, etc. – but if you’re reading this, please don’t do it again!

Thankfully, nothing was lost. The upside is that each time is a learning experience, and a chance to harden and tighten things up a bit more.

This time, I took the opportunity to switch from Debian to FreeBSD. From my brief experience with it, I like it a lot better. I thought I was going to miss some of the great GNU tools, since it’s not GNU/Linux, but that hasn’t really been an issue. I also thought I would miss the Debian package manager, but I like the flexibility of the ports system with the option of binary packages a lot better.

The key productivity advantage, though, is that it’s easier to get up-to-date software. When you’re developing in Rails, you tend to want the latest version of software. On Debian, that means you’re always mixing and matching from stable and unstable, and sometimes you even have to go to testing to get what you need. For example, I ended up giving up on trying to get Apache 2.2 installed under Debian. With FreeBSD, there just wasn’t a problem.

From a security standpoint, I’m hoping that the fact it’s a smaller target is going to help some. Plus FreeBSD has this nice daily security email. And finally, I find that the FreeBSD ports system lets me install a set of software closer to just the stuff I need—and less software is always more secure.

Finally, I’m caving in and getting a firewall. The folks who taught me always taught me that firewalls don’t matter if you just close off all the services and ports you’re not using. It seems logical, but I don’t want to take a chance with this anymore, so now I’m getting a firewall put in front. It feels a bit like buying the extended warranty for a fridge, but what are you gonna do?

The most annoying part of this was that we were down for so long. 24 hours in total. Of that, the 15 were spent waiting for HostEurope to get the OS reinstalled. I sent in the request at about 2 AM, half an hour after the breakin, which my SMS alerts had alerted me to. The OS reinstall wasn’t started on until around noon the next day, and wasn’t completed until 5 PM, which makes 15 hours total. Obviously, that’s not acceptable going forward.

So we’re looking at moving to a different host, and while we’re at it, we should probably get one in the US, since that’s our primary market, though the bandwidth we get in Europe is insane compared with the US plans (5.000 Gb of traffic included in all plans starting at EUR 100/month). Compare that to both Rackspace and Tilted, which gives you 150 Gb in the basic plan, that’s 1/33rd of HostEurope.

What are your recommendations for a serious host that can also help us keep our systems secure and optimized?

The next question is when and how to move beyond the current single-box setup. Right now, there’s ample room to grow on that machine. We could easily host 20x the traffic we currently do (which is mostly thanks to Boxes and Arrows), and I haven’t even started optimizing yet. But the problem is downtime if something breaks (or someone breaks in)—the 15 turnaround time on a reinstall is not something we can afford.

One approach would be to move to 2 boxes first, one being web and application, the other being database, but replicate the database, and have them do failover. That way we add both redundancy and distribute the load. Then from there, we can move to 4 when we need it.

What’s your experience with this, what are the pitfalls and things to look out for, and how much should we expect this to cost us? Your help is greatly appreciated.


Hey Lars, Check out ISprime.com. Their plans look pretty nice and I hear good things. Sure, they use Dell, but nobody's perfect ;-) Cheers, Anthony
By Anthony Cimino on Thu, Oct 12, 06 at 17:20 · Reply
Thanks for the link, Anthony, Actually, I'm on Dell already. I don't mind, so long as I don't deal with them directly myself.
By Lars Pind on Thu, Oct 12, 06 at 17:20 · Reply
how'd they get in this time? seems like you were running a pretty tight ship..
By court3nay on Thu, Oct 12, 06 at 17:20 · Reply
I wish I knew. There was nothing obvious about the processes that I could see were running. But then again, they'd installed a root kit, so most command-line utilities were non-functional, and the rest were probably compromised. I'm hoping that a firewall + redundancy + FreeBSD + a bit more tightening, will be the cure. As we know from reading Bruce Schneier, nothing is 100% secure, it's about the cost of protecting vs. the cost of a successful attack times the probability of such an attack.
By Lars Pind on Thu, Oct 12, 06 at 17:20 · Reply
Always use a firewall. At home I have a simple firewall but even so I was getting <a href="http://mark.aufflick.com/blog/2006/10/10/simple-var-log-secure-analysis">hammered by ssh login attempts</a> (I probably should use a non-standard port). For commercial servers I would recommend a serious commercial firewall with as many intrusion detection metrics as you can. I remember in 2000 my friend Russell put an unpatched Linux (or was it Solaris ... can't quite remember) on the net with no firewall to see what would happen. My recollection is it took about 6 days to be hacked. I'm sure there is much more danger these days. The beauty about having a good firewall is that you end up with very few things to worry about. Your firewall, your httpd, your sshd and, of course, your site code.
By Mark Aufflick on Thu, Oct 12, 06 at 17:20 · Reply
Leo Laporte is also looking for a new hosting company: http://www.twit.tv/2006/10/13/all_feeds_are_currently_down
Interesting, I should team up with him :)
By Lars Pind on Thu, Oct 12, 06 at 17:20 · Reply

Leave a comment