Reuse-in-the-large is an unsolved problem: Why I left OpenACS for Rails

A number of folks have asked me why I decided to leave OpenACS in favor of Rails, and with David leaving Denmark for the States, I figured now was as good a time as any to tell the story publicly.

It’s always been clear to most of us in the OpenACS core group, that OpenACS has some things going against it:

  • AOLserver was originally chosen because Apache wasn’t very popular and was inferior at the time, but things had changed a lot, and not supporting Apache was a constant frustration.
  • Tcl, which was chosen because AOLserver had an interpreter built-in, isn’t exactly the sexiest or most modern scripting language out there.
  • PostgreSQL is a very fine database, but almost everybody uses MySQL.
  • Code quality was very uneven, with many packages being poorly designed, written and untested.
  • Documentation was poor, and things like metadata for tables had several different partial solutions in different places in the toolkit.
  • The community is small, which is comfy, but it isn’t growing, and things move very slowly because there’s not enough people.
  • Its data model is heinously complex, with more than 100 tables in a normal install, truckloads of stored procedures, many many joins, and two databases to support, making development hard and performance poor.

These are important problems, to be sure, but the big prize, the reusable applications, more than make up for it. Except, over the first half of 2004, I came to realize that they were a pipe dream.

OpenACS’ primary claim to fame was always the reusable applications (see Community in a Box” for example). There’s a forum, a file-storage, a login system, the groups and the permissions system, the portals. Throw them together, and voila, you have an e-learning platform. Or a project workspace. Or anything, really.

That’s the theory, but I had five experiences that all drove home the message that reality doesn’t stack up.

The first was Basecamp. I’d been struggling with how to design web applications for several years, including a period as UI Czar at ArsDigita, but had failed. Basecamp came to epitomize what I think a web application ought to be like, and it wasn’t quite what we’d had in mind. An example: We had spent countless hours creating a form builder that was just right, so you could make every form on the site look the same, and change it at the flip of a switch. But Basecamp’s forms were all hand-craftet, because each had a different purpose, so each should have a different design. Web forms is not as generic a problem as you’d think.

Moreover, they didn’t even use a forum for the forum. They used a blog format. And that, too, worked out quite well. And, well, they didn’t use a file storage, either, files were just attached to the messages. And they didn’t use a fancy login system, they had a very simple one, where you can’t register, you can only have people invite you. And no fancy permissions, either. You can’t design a framework without designing it for some kind of software, and Basecamp changed my perception of what a web framework should be designed for.

The second lesson was an experience with .LRN, an e-learning system built on top of OpenACS, and used by a number of universities around the world. They put together a forum, upload course materials to a file-storage, and glue it all together in a portal. But there was one nasty bug number 900-something, which kept rearing its ugly head. It was about whether a HTML file in the file-storage should be shown in a way so relative links worked, or downloaded, and the bottom line was that we kept changing things back and forth for quite a while, because people simply used the file-storage differently, even though it was all within a university setting. The lesson I took away is that a file-storage is not just a file-storage. Uploading a file and getting it back out is trivial today (it wasn’t when OpenACS was started). It’s the details that really matter.

Another example was the login system extensions we did. It was a joint effort between many clients, and one of them paid specifically to have us do LDAP, because they wanted to use that down the line. First, they ended up never using it, because by the time they were ready, they’d decided to switch to Rails instead. Second, when I did do the actual integration in Rails, it turned out that what they really needed was much simpler and much different from what we’d built, and what they thought they needed at the time. The lesson is, don’t build without a specific need here and now, because what you think you’ll need will be wrong. So much of OpenACS is built on guesswork.

One of the “privileges” we had at Collaboraid, was that most of our projects extended the core in some way. But in the summer of 2004, we got a large project where there would be no core extensions, and where the client had engaged an information architect and a graphic designer to specify what to build in detail. So here was a project to build a well-defined site, as simply as possible. It had a photo-album, a forum, a FAQ, login pages, all the things that OpenACS already had, only the user interface defined by the design team was different from the OpenACS modules. Naturally, we thought that we’d use the existing modules and just tweak them to fit. But that turned out to be more work that just writing exactly what the client needed from scratch. With that, I learned that our fine applications weren’t up to par with a client’s reasonable expectations to a website.

That project also had a pretty tight deadline, so I hired a bunch of people I knew and trusted in the community. I’d read Brooks and knew the dangers, but we had a shared culture. Well, it turned out that the culture was so strong in its belief that a forum is a forum and details don’t matter, that despite the detailed specs, they would deliver something that was the OpenACS forum, instead of what the client asked for. When I pointed out the very obvious differences between the spec and what was delivered, they just couldn’t see what I was talking about. I ended up recoding almost everything myself. The lesson? That the culture is very strong, and you can’t realistically change it.

Finally, at some point during the summer, I also read Fact 16 in this article by Robert Glass:

“Reuse-in-the-large (components) remains a mostly unsolved problem, even though everyone agrees it is important and desirable.”

Bam! There you have it. All the problems I’d seen weren’t because of poor execution. It was a dumb idea to begin with! Reuse-in-the-large is exactly the problem that OpenACS is trying to solve, and it “remains a mostly unsolved problem, even though everyone agrees it is important and desirable”. Can it be said any clearer than that?

Problem is, that when you take away that piece, you’re left with an little known, hard-to-install, overly complex, partially debugged, aging toolkit built on a stack that nobody else uses. It started to look very unattractive. I knew I had to get away.

But to where? I’d been looking around for quite a while: PHP, Zope, .NET, Mono, Java. In fact, I’d done that in 2001, too, after ArsDigita, and before starting Collaboraid. But I didn’t see anything that I liked enough to leave my friends, my customers, and my code. That is, until David showed me Rails.

I had him come in to the office and give a demo to Peter, Yon, and myself, and I was blown away. Not only was it neatly done, but the thinking behind it matched what I had learned, and all the problems that OpenACS had were solved. I knew from that instant that I had to leave OpenACS and pick up Rails.

I did, and I haven’t looked back. I’m glad to see that a number of my old friends and customers have decided to join me.

UPDATE: I’ve been getting a number of emails with comments. Feel free to post them here publicly.


Pelle Braendgaard

Good to meet you last night. You basically have come to the same conclusion I have. I myself have strived for the ultimate reuse on several projects and you can achieve it on some very few simple issues, but generally speaking reuse by copy and paste is much more effective than other ways of doing it. With Ruby on Rails as well I find the core infrastructure is so good that I find myself reusing from Memory and experience much more than I would in say Java. An example. For data models it is much easier for me to just generate the models and manually build up the relations than copying from an existing project or extrapolating common frameworks for that matter. A side effect of this as well is that I am not afraid to dump 100% of my code and just redoing it, if I feel there is a good reason to do so.
Read more
Read less
C. R. Oldham

I read your OpenACS departure article almost with tears in my eyes. I have also left the community, but I was forced more by my workplace (new CTO likes Java). I have also discovered that I don't have enough hours in the day to continue to play with OpenACS. We still have an OpenACS deployment, but I suspect it will be going away in the next couple of years. You didn't talk in your article, however, about the OpenACS datamodel--that was really what drew me to the toolkit originally. Especially the groups/relations/permissioning system. It was so powerful, and we really did use it extensively in our applications. That is something I will definitely miss, and each time I consider starting over with something else I keep thinking "gosh, I really don't want to rebuild that, they did such a good job...". Another thing we discovered, though, was that if you went with OpenACS, you really had to do everything with OpenACS. Integration with other systems was hard. This became even more apparent with the drive in the community to move all the logic from the SQL layer into the Tcl layer. When everything was in the SQL layer, you could do just about anything you wanted with any language as long as you could talk to the database. In the Tcl layer, you had to write another layer for integration. Maybe it's hard with anything, and I really tried to avoid the "if all you have is a hammer, everything looks like a nail" trap.
Read more
Read less

IMHO OpenACS datamodel is very nice but is complex. I'm learning Ruby On Rails and the integration between "any" datamodel and the OO language seems simple and nice. Rails invite to the programmer to the right thing.
Read more
Read less

This article has a lot of resonance with me. I only did a handful of OpenACS deployments and in fact the first one was the quickest and the best and it was Openacs 3. Openacs 4 and then Openacs 5, promised more and more but for me became more and more difficult to modify.
Read more
Read less

I was probably ArsDigita's last hire, in Pasadena, about a week before 9/11. Aurelius Prochazka now uses Rails. 'Nuff said...
Read more
Read less
Dr. Chucks Blog » Blog Archive » NASA Factoid from Trek

[...] software – it is more efficient to just write it from scratch. Posted from the Hacker Jam. The original post from Lars Pind Category: [...]
Read more
Read less

Leave a comment