Hacked again - lessons and next steps

My server got hacked once again. I’m not sure what I’ve done to attract this kind of unwanted attention – I’m doing all the obvious things like keeping the system patched, passwords secure, moved SSH to a non-standard port, monitoring the logs, etc. – but if you’re reading this, please don’t do it again!

Thankfully, nothing was lost. The upside is that each time is a learning experience, and a chance to harden and tighten things up a bit more.

This time, I took the opportunity to switch from Debian to FreeBSD. From my brief experience with it, I like it a lot better. I thought I was going to miss some of the great GNU tools, since it’s not GNU/Linux, but that hasn’t really been an issue. I also thought I would miss the Debian package manager, but I like the flexibility of the ports system with the option of binary packages a lot better.

The key productivity advantage, though, is that it’s easier to get up-to-date software. When you’re developing in Rails, you tend to want the latest version of software. On Debian, that means you’re always mixing and matching from stable and unstable, and sometimes you even have to go to testing to get what you need. For example, I ended up giving up on trying to get Apache 2.2 installed under Debian. With FreeBSD, there just wasn’t a problem.

From a security standpoint, I’m hoping that the fact it’s a smaller target is going to help some. Plus FreeBSD has this nice daily security email. And finally, I find that the FreeBSD ports system lets me install a set of software closer to just the stuff I need—and less software is always more secure.

Finally, I’m caving in and getting a firewall. The folks who taught me always taught me that firewalls don’t matter if you just close off all the services and ports you’re not using. It seems logical, but I don’t want to take a chance with this anymore, so now I’m getting a firewall put in front. It feels a bit like buying the extended warranty for a fridge, but what are you gonna do?

The most annoying part of this was that we were down for so long. 24 hours in total. Of that, the 15 were spent waiting for HostEurope to get the OS reinstalled. I sent in the request at about 2 AM, half an hour after the breakin, which my SMS alerts had alerted me to. The OS reinstall wasn’t started on until around noon the next day, and wasn’t completed until 5 PM, which makes 15 hours total. Obviously, that’s not acceptable going forward.

So we’re looking at moving to a different host, and while we’re at it, we should probably get one in the US, since that’s our primary market, though the bandwidth we get in Europe is insane compared with the US plans (5.000 Gb of traffic included in all plans starting at EUR 100/month). Compare that to both Rackspace and Tilted, which gives you 150 Gb in the basic plan, that’s 1/33rd of HostEurope.

What are your recommendations for a serious host that can also help us keep our systems secure and optimized?

The next question is when and how to move beyond the current single-box setup. Right now, there’s ample room to grow on that machine. We could easily host 20x the traffic we currently do (which is mostly thanks to Boxes and Arrows), and I haven’t even started optimizing yet. But the problem is downtime if something breaks (or someone breaks in)—the 15 turnaround time on a reinstall is not something we can afford.

One approach would be to move to 2 boxes first, one being web and application, the other being database, but replicate the database, and have them do failover. That way we add both redundancy and distribute the load. Then from there, we can move to 4 when we need it.

What’s your experience with this, what are the pitfalls and things to look out for, and how much should we expect this to cost us? Your help is greatly appreciated.