For years, I've been hoping that we'd find the resources to build a testcase manager, something Bugzilla-like that would allow us to store testcases, track test results over time, and most importantly to distribute the testing load by breaking it up into smaller chunks. Zach, Dawn, Myk and I have had many discussions about building this system and I've, at least twice, drawn up some design docs for how it might work.
A couple of months ago, Myk pointed out an existing testcase manager which was designed to be integrated with Bugzilla (how cool is that!) It was originally implemented by Maciej Maczynski and then ported forward to work with a more recent Bugzilla version by Ed Fuentetaja.
It's pretty clear to me that both of these people put a lot of effort into what is a very sophisticated tool but my initial evaluation of the tool was that it didn't seem to fit well with our model for distributing work and that it wasn't "continuous" enough - it appears to be designed for one person to run one set of tests against one particular build where Mozilla is constantly pushing new nightly builds to thousands of nightly build testers and soliciting bug (and other) feedback incrementally and progressively.
Also, while it is a very powerful tool with, for example, sophisticated mechanisms for doing things like diff'ing versions of a particular testcase or assigning testers to specific components or testcases, it is also seemed quite difficult to use and it wasn't the kind of system that I felt would attract a lot of people willing to give a few minutes of time to run a few tests against their latest nightly build.
Well, Myk's been hacking on it for a couple of weeks, streamlining a few of the primary interfaces and generally making it fit a little closer with my ideas for how a testcase manager ought to function. I'm starting to come around. With some help from Marcia, I've got our smoketests plugged into TestRunner and have been running those tests and recording results for the last few days. So far so good.
I'll be posting more information and links to a test installation as soon as we get a little bit further along. If you're interested in helping hack on this new tool or you've got experience with testcase management and you can offer suggestions, please click that "Comments" link and let me know.