Three Monkeys, Three Typewriters, Two Days

December 1, 2005

Layout QA going forward for 1.9

Well, we've shipped Gecko 1.8. Looking back on the 1.8 cycle, I think we ran into a lot more layout regressions than I would have liked. It would be nice to prevent that happening in the 1.9 cycle, especially with the changes we (hopefully) have coming up. As I see it, there are several things that should happen:

  1. We need to have tests. We have a number checked into CVS, but I think we have a lot of layout bugs that have never ended up with a corresponding test checked into CVS. It would be wonderful if we could somehow make a list of fixed layout bugs that have a testcase (so non-patch attachment, maybe of an HTML or XML MIME type?) attached. If bugzilla can give us such a list, it could be compared to the list of bugs we have layout regression tests for; then we could try to look at the remaining bugs (ones without tests for them) and see whether we can check in tests for them.
  2. We need a test framework. The existing layout regression tests take a long time to run and give a lot of false positives ("failures" that are not actually failures). Robert has created a framework that relies on a screen capture approach instead of dumping out internal data structures (which is what the existing framework does); he's had various troubles with us painting nondeterministically (which leads to false positives), but I think he was going to try to get those issues dealt with. So hopefully we'll have this.
  3. We actually need to run the tests. Given the number of testcases involved (several thousand for the tests we have plus some of Ian's> tests and the CSS test suites), running them will never be that fast. So this is something that would hopefully happen once a day automatically. I think Bob has been looking for time to work on it. So hopefully this will happen too.

Which brings us back to item 1 on this list. Luckily, it seems that it should not be so hard to parallelize, especially if someone can generate that list of bugs to consider looking at. I'd be happy to check in the regression tests if people cc me on bugs that needs to happen with (and note that regression tests do NOT need to be minimized, by any means, so any bug with a testcase is good to go). So I guess I'm sort of hoping someone will volunteer to at least make the list, if not to organize people to go through it.... Any takers?

Posted by bzbarsky at December 1, 2005 12:04 AM