Reuse-in-the-large is an unsolved problem: Why I left OpenACS for Rails

A number of folks have asked me why I decided to leave OpenACS in favor of Rails, and with David leaving Denmark for the States, I figured now was as good a time as any to tell the story publicly.

It’s always been clear to most of us in the OpenACS core group, that OpenACS has some things going against it:

• AOLserver was originally chosen because Apache wasn’t very popular and was inferior at the time, but things had changed a lot, and not supporting Apache was a constant frustration.
• Tcl, which was chosen because AOLserver had an interpreter built-in, isn’t exactly the sexiest or most modern scripting language out there.
• PostgreSQL is a very fine database, but almost everybody uses MySQL.
• Code quality was very uneven, with many packages being poorly designed, written and untested.
• Documentation was poor, and things like metadata for tables had several different partial solutions in different places in the toolkit.
• The community is small, which is comfy, but it isn’t growing, and things move very slowly because there’s not enough people.
• Its data model is heinously complex, with more than 100 tables in a normal install, truckloads of stored procedures, many many joins, and two databases to support, making development hard and performance poor.

These are important problems, to be sure, but the big prize, the reusable applications, more than make up for it. Except, over the first half of 2004, I came to realize that they were a pipe dream.

OpenACS’ primary claim to fame was always the reusable applications (see Community in a Box” for example). There’s a forum, a file-storage, a login system, the groups and the permissions system, the portals. Throw them together, and voila, you have an e-learning platform. Or a project workspace. Or anything, really.

That’s the theory, but I had five experiences that all drove home the message that reality doesn’t stack up.

The first was Basecamp. I’d been struggling with how to design web applications for several years, including a period as UI Czar at ArsDigita, but had failed. Basecamp came to epitomize what I think a web application ought to be like, and it wasn’t quite what we’d had in mind. An example: We had spent countless hours creating a form builder that was just right, so you could make every form on the site look the same, and change it at the flip of a switch. But Basecamp’s forms were all hand-craftet, because each had a different purpose, so each should have a different design. Web forms is not as generic a problem as you’d think.

Moreover, they didn’t even use a forum for the forum. They used a blog format. And that, too, worked out quite well. And, well, they didn’t use a file storage, either, files were just attached to the messages. And they didn’t use a fancy login system, they had a very simple one, where you can’t register, you can only have people invite you. And no fancy permissions, either. You can’t design a framework without designing it for some kind of software, and Basecamp changed my perception of what a web framework should be designed for.

The second lesson was an experience with .LRN, an e-learning system built on top of OpenACS, and used by a number of universities around the world. They put together a forum, upload course materials to a file-storage, and glue it all together in a portal. But there was one nasty bug number 900-something, which kept rearing its ugly head. It was about whether a HTML file in the file-storage should be shown in a way so relative links worked, or downloaded, and the bottom line was that we kept changing things back and forth for quite a while, because people simply used the file-storage differently, even though it was all within a university setting. The lesson I took away is that a file-storage is not just a file-storage. Uploading a file and getting it back out is trivial today (it wasn’t when OpenACS was started). It’s the details that really matter.

Another example was the login system extensions we did. It was a joint effort between many clients, and one of them paid specifically to have us do LDAP, because they wanted to use that down the line. First, they ended up never using it, because by the time they were ready, they’d decided to switch to Rails instead. Second, when I did do the actual integration in Rails, it turned out that what they really needed was much simpler and much different from what we’d built, and what they thought they needed at the time. The lesson is, don’t build without a specific need here and now, because what you think you’ll need will be wrong. So much of OpenACS is built on guesswork.

One of the “privileges” we had at Collaboraid, was that most of our projects extended the core in some way. But in the summer of 2004, we got a large project where there would be no core extensions, and where the client had engaged an information architect and a graphic designer to specify what to build in detail. So here was a project to build a well-defined site, as simply as possible. It had a photo-album, a forum, a FAQ, login pages, all the things that OpenACS already had, only the user interface defined by the design team was different from the OpenACS modules. Naturally, we thought that we’d use the existing modules and just tweak them to fit. But that turned out to be more work that just writing exactly what the client needed from scratch. With that, I learned that our fine applications weren’t up to par with a client’s reasonable expectations to a website.

That project also had a pretty tight deadline, so I hired a bunch of people I knew and trusted in the community. I’d read Brooks and knew the dangers, but we had a shared culture. Well, it turned out that the culture was so strong in its belief that a forum is a forum and details don’t matter, that despite the detailed specs, they would deliver something that was the OpenACS forum, instead of what the client asked for. When I pointed out the very obvious differences between the spec and what was delivered, they just couldn’t see what I was talking about. I ended up recoding almost everything myself. The lesson? That the culture is very strong, and you can’t realistically change it.

Finally, at some point during the summer, I also read Fact 16 in this article by Robert Glass:

“Reuse-in-the-large (components) remains a mostly unsolved problem, even though everyone agrees it is important and desirable.”

Bam! There you have it. All the problems I’d seen weren’t because of poor execution. It was a dumb idea to begin with! Reuse-in-the-large is exactly the problem that OpenACS is trying to solve, and it “remains a mostly unsolved problem, even though everyone agrees it is important and desirable”. Can it be said any clearer than that?

Problem is, that when you take away that piece, you’re left with an little known, hard-to-install, overly complex, partially debugged, aging toolkit built on a stack that nobody else uses. It started to look very unattractive. I knew I had to get away.

But to where? I’d been looking around for quite a while: PHP, Zope, .NET, Mono, Java. In fact, I’d done that in 2001, too, after ArsDigita, and before starting Collaboraid. But I didn’t see anything that I liked enough to leave my friends, my customers, and my code. That is, until David showed me Rails.

I had him come in to the office and give a demo to Peter, Yon, and myself, and I was blown away. Not only was it neatly done, but the thinking behind it matched what I had learned, and all the problems that OpenACS had were solved. I knew from that instant that I had to leave OpenACS and pick up Rails.

I did, and I haven’t looked back. I’m glad to see that a number of my old friends and customers have decided to join me.

UPDATE: I’ve been getting a number of emails with comments. Feel free to post them here publicly.