Which Problem do You Want to Solve?

Software engineers have an occupational disease: They fix problems.

Next time you’re with a software engineer, try showing him a chair that’s broken, or a fridge that’s making noise, and I’ll bet you, he’ll start trying to fix it on the spot. Or try telling him that you can’t get your phone numbers from your palm pilot into your cell phone, and he’ll design a system that could: “If only all the companies agreed on a common XML format, and then we could write a stylesheet, that …” Thanks, we got it.

If a software engineer is left without a problem to fix, he’ll find a problem to fix. Fifty engineers will find some really big problems to fix.

A few months after I started working for <a href=”/lars/goodbye-arsdigita”>ArsDigta, I suggested that we establish a small team of people, non-billable, who would be responsible for maintaining and developing our code base, the <a href=”http://openacs.org”>ACS. With about 100 people depending on and finding bugs in that code base and nobody to receive the patches, things were out of hand. Refactorings and clean-ups that were clearly needed didn’t get done. Something had to happen.

One point in my proposal that was quickly forgotten: Membership of the toolkit team should be temporary, on a rotating basis, so they wouldn’t get a chance to forget who they were serving. (Honestly, it was there. I wish I could find the document now.) The toolkit team team happened around March 2000, and we quickly put together a long list of problems to attack.

A crucial mistake was made. Instead of looking at it as a laundry list, a wish list of problems to potentially solve, items to prioritize and cut, estimate, work through one by one, we decided we wanted to do all of them, and at the same time. We started the ACS 4 project, dreamed of all the really tough problems we’d solve (“let’s implement an object-relational database on top of Oracle”), and embarked. Today, about two years later, that product has just reached Beta status over at OpenACS. We pulled a Mozilla.

But things didn’t stop there. The small and tidy toolkit team that I’d envisioned grew and grew during the following year, until it had become a 50-person monster, completely isolated from the annoyance of customers and users. And it became time for another rewrite, this time to Java.

With all this man-power, a whole new world opened up, full of really interesting problems to tackle. The object-relational problem, anyone? Ah, there’s a nice problem to solve. Just because nobody’s done it before doesn’t mean it can’t be done, does it? Let’s solve that one.

When the ACS was rewritten again, this time in Java, the same thing happened, only more extreme. We not only stayed with the object-relational layer on top of Oracle, we added the Object-Relational mapping problem (mapping Java objects to rows in a database), the web user interface framework problem (Java Swing for the web, dubbed Bebop), the dynamic web publishing with XSLT problem, and probably a couple more that I forgot.

With ACS/Java, I quit programming and became the user interface designer. And I, too, got caught by this bug, and wanted to solve the web user interface problem, once and for all. Needless to say, the effort failed, as did the rest of all the great problem-solving initiatives.

Why did we fail? We failed because we underestimated, as engineers always do, how hard these problems really are. If everyone’s aware that there is a problem, and it’s not really solved yet, it’s probably hard. Unless you have a really compelling reason to think that you’re the one who can solve this, you probably can’t. And “hey, but we’re smarter”, or “we work harder”, or “we look better” isn’t a compelling enough reason.

I’m not saying you shouldn’t try to solve problems that haven’t been solved before. Of course you should. But you have to pick which problems to solve.

Unless your customers happen to be engineering shops, who use your software to build other software, your customers don’t care if you solve a really tough engineering problem. Their problem is not an engineering problem. It’s a real-world problem. And real-world problems are hard enough in themselves.

Interesting and not yet solved problems are always hard. Make it a rule to only try to solve one of them at a time. Focus your energy on the problem that you’re most likely to make a contribution towards solving. Stop thinking that you’re going to actually solve it, as in “Solved. Done.” Stop believing that you can squeeze in an “oh, and by the way, I also solved this really hard engineering problem that IBM and Oracle and Microsoft Research have been banging away at for years.” You can’t.

Pick the one problem that matches your talents and skills, as individuals, and as a company. And stay with it. Is that choice going to be the object-relational mapping problem? The web user interface framework problem? Or the computer-supported collaboration problem?

I know which one I’d pick.