We got 0wn3d

Our server, which runs this site and Boxes and Arrows, got hacked last night while I was at the christmas lunch party at DR, the Danish Broadcasting Corporation. It happened at 8 PM CEST, 11 AM PST, and it took me about 14 hours to get everything back up and running. The backups were from around noon, so we’ve lost 8 hours worth of data. Thankfully, it doesn’t look like there was any user-contributed content among that, fingers crossed.

The bad news, of course, is that it happened. The good news is that it gave us, shall we say, a very realistic disaster recovery drill. I’ve definitely learnt something from the experience.

I had my hosting provider reinstall the OS from scratch on the server. Once you’re 0wn3d, there’s no saving the software on it. It took a few hours, and so around 11 PM it was ready for me to dive into again. And I took some extra time to make sure the server isn’t running anything it doesn’t need.

One thing I’ve changed is have backups run every hour, in addition to every day. The daily backup goes to a backup server where I can retrieve old versions. The hourly backup goes to another server via rsync.

It was a pretty weird night and morning. The president of DR lost his jacket with his wallet and everything during the christmas lunch party, and when we got up this morning, Caroline couldn’t find hers. Also, I’d just demoed PublicSquare to someone there when it went down. Normally I’d have a few beers at an occasion like that, but this particular night, I didn’t feel like it. Lucky, given that I ended up working till 7.30 in the morning.

So we’re up again, and I’ve learned quite a bit from the experience. Should this ever happen again, knock on wood, I’d be even better prepared to get everything back in place quickly. And no, I’m not daring you.

UPDATE: Friday keeps getting weirder. John Spencer, AKA Leo McGarry, died that night. And I just got a call from Ekstra Bladet who wanted to know more details about the jacket. I declined.