Three Monkeys, Three Typewriters, Two Days

2367 fixed bugs - Layout * - attachment mime type contains html - http://tinyurl.com/78v8v
*871 bugs Layout
* 51 bugs Layout: BiDi Hebrew & Arabic
* 78 bugs Layout: Block and Inline
* 11 bugs Layout: Canvas
* 3 bugs Layout: CTL
* 36 bugs Layout: Floats
* 27 bugs Layout: Fonts and Text
*490 bugs Layout: Forms controls
* 70 bugs Layout: HTML Frames
* 17 bugs Layout: Images
* 17 bugs Layout: Misc Code
* 61 bugs Layout: R & A Pos
*532 bugs Layout: Tables
*103 bugs Layout: View Rendering

79 fixed bugs - Layout * - attachment mime type contains xml - http://tinyurl.com/86g9e

May there should be a "testcase" flag in Layout component just like the one in the JavaScript Engine component?

Maybe you could use the comunity and some human eyes to do the tests?

Put all the test cases in a wiki, each case has a png of the desired output.

Then people click the 'random page' link and do a test. Somewhere a score is held indecating the number of tests a person has done. Each time a person scores a test this is counted as a vote (pass or fail) then when a test has more than 10 votes something amazing happens with champaign, chocolate and fairies.

Actually, could you make the png of the desired output 50% transparent, then overlay that onto the web page. Then any mistakes would be easy to see.

People could sign up to do, say 50 tests and get a free Mozilla t-shirt (donated by a sponsor)

monk.e.boy

"We actually need to run the tests. Given the number of testcases involved (several thousand for the tests we have plus some of Ian's> tests and the CSS test suites), running them will never be that fast."

Could you us the community to help with this? Something like SETI@Home or BOINC, where the "running" is distributed over multiple PCs?

"Robert has created a framework that relies on a screen capture approach instead of dumping out internal data structures (which is what the existing framework does);"

Why is that a better approach?

I'm certainly interested in doing this.
I could collect a lot of testcases from bugzilla (from bugs that have the testcase keyword) and put them somewhere online.

Kevin, thanks for making those lists! We should probably try to extract the bug numbers from those and start eliminating the ones that have tests already checked in...

Caleb, the testcase flag might be nice, but people would forget to set it, I bet.

monk.e.boy, we have about 6000 tests last I checked (counting test suites, etc). We want to run them every day, ideally. Doing that by hand is really not feasible -- that's on the order of 500 man-hours a day, and no mistakes allowed... Plus, we want to be able to easily change the "desired output" if needed, and the png approach makes that hard.

Peter, that might be doable, but again we'd have to have zero error tolerance and a lot of coordination. It seems simpler to just have a test machine spend 3 hours a day running the tests.

Daniel, roc's approach catches regressions in painting code, not just in the layout code. It also doesn't start crying about internal data changes that don't affect final rendering (which is a desirable feature, imo).

Martijn, how about we just get you a CVS account and you can just check the tests in? ;)

I believe the checking in of the layout regression test cases is a task that can be tracked down to a very small number of developers. We know who has CVS write permission and its their duty to checkin the tests too. I neglected that task for a couple of months. I always thought I will do that the other day. Well before your post the other day came (Nov. 19) however. It took hours, to check what bugs got fixed, extract the test cases, place them correctly, verify that they are still rendered correctly and incorporate them into the rtest.lst.

Not running the layout regression tests was at the core of my clash in 2003. For me, running the layout regression tests is a question of courtesy, you don't want to break something that other people got working. David's argumentation then (IIRC) went: we should rather test for correctness than trying to hammer our quirky behavior into stone. (As usual he is right.)

The core difference to the Javascript test framework is that they have a trace of professional QA since the NS days. If I think back only Hixie, has done something similar for us. The outcome of Javascript test cases is probably also easier to judge. What we need is to detect *any* change. Verifying regressions is a QA task, so what we need here is similar efficient QA like the Javascript people have. As we are heavily understaffed each capable QA will however rather soon be converted into a dev. Yeah, Martijn this means YOU.

I see one disadvantage with the painting framework, it sees only content that is visible. If a test case is longer than a screen, a lot of our tests are, the test is blind for them.

If somebody wants to run the tests that we currently have, read http://www.mozilla.org/newlayout/doc/regression_tests.html . I will help you, if that is not enough. I have a patch to get the layout debugger in firefox running ( https://bugzilla.mozilla.org/show_bug.cgi?id=278859 ) but some help to get this working without causing a regression in seamonkey is welcome.

Bernd, roc's painting thing paints the entire document, not just its intersection with the viewport. So it'll catch issues in testcases that are longer than the screen. It might not catch issues that happen inside scrollable areas in a testcase, though (e.g. overflow:scroll divs).

You admire the Javascript test framework, but the Javascript engine fails some of the W3C ECMAScript DOM Conformance Test Suites :-)

I opened a bug about that, bug 296391, but didn't find the time to follow and find out exactly what is going wrong, if it's problems in javascript or in the DOM implementation/interpretation.

Don't get me wrong, most of the suite goes OK, it's only a few percents of the tests failing.

jmdesp, if there are issues with the DOM test suite, please file bugs on specific tests we fail; that way someone can look at them and figure out what's going on. That said, I'll bet money that the issue is bugs in our DOM impl, not the JavaScript engine. The JavaScript engine is NOT really what is tested by the DOM conformance suite.

And yes, we need a DOM testing framework too; I believe Bob Clary is working on it.