Logging in production with Rails

I use the Rails server logs primarily to dig through a user’s interaction with the site across requests to track down intricate bugs, like how did the user manage to get the system into this state which it shouldn’t be in.

But using the default Rails logs to do this piece of detective work is horrible.

I’m using the Hodel3000CompliantLogger, which prefixes each log line with the process id, which makes it possible to trace through each individual request.

But I haven’t found any tools that help me dig through those large log files yet.

I discovered Loggly today, and I briefly got my hopes up, but it does nothing like what I’m looking for. Or if it is, I haven’t noticed. All I can see that it does for me, is aggregate log files from various servers. I only have one server. There’s some searching, but it still just shows me the plain old log file.

What I want to see is a list of each request served by Rails, along with the IP address, the user agent, the parameters, info from the session such as logged in user ID, the response code, the redirected to URL if redirected, perhaps processing time, and memory consumption.

Then I want to be able to filter by IP address, so I can follow just one user’s path through the system, looking at each request, one by one.

Then I want to be able to dig into each request, one by one, seeing everything my app logged, which includes database statements and my own debugging statements and such.

I think having such a tool would help me tremendously.

Does any such tool exist already?

Possibly a customized Rails logger along with a web-based interface that lets me search and browse my logs.

I really thought Loggly was it, but I can’t see that it does the job for me :(

Your insight is greatly appreciated.