« Thunderbird 2 Beta 1 has been unleashed! | Main | 12/15: Community Test Day - FF 2001 on Windows Vista / Gecko 1.9a1 »
December 14, 2006
Test Tool Spotlight: reftest
This morning, I'm going to take a few moments out of my struggles with buildbot (which is coming along nicely, thanks, more about that in a bit) to talk a bit about one of our unsung testing tools, reftest by layout superstar Dave Baron who recently landed the reflow branch to some fanfare and accolades (mad props, Dave!).
Short for "reference test", reftest runs through a list of paired html documents, renders both and converts them to a string-based graphical representation for comparison. Depending upon how the pages are expected to compare, (equals, not-equals, expected pass, or expected fail), the test outputs the pass or failure to the console. In the case of a failure, the log emits the data URL of the graphics that didn't compare.
Currently, this test is rather strict. Because the two images have to be identical for them to match, a difference of a single pixel can throw a failure. During the past couple of weeks, with the landing of the reflow branch, I've seen the tests vary between passing everything and failing everything on each platform, to varying degrees between those extremes. Just now, for example, a Linux build passed completely, Mac OS X failed 1 test (351641-1b.html vs. 351641-1-ref.html) and Windows XP failed 3 tests (142233-1.html vs. 142233-1-ref.html, 351641-1a.html vs. 351641-1-ref.html and 351641-1b.html vs. 351641-1-ref.html). This is out of a total of 74 tests in the current reftest manifest.
Here is an example of what a pair of images looks like from the 351641 test:
Can you see the difference? It's subtle, but the cellpadding on the right is slightly less than in the table on the left -- it's off by about a pixel and that's enough to cause the reftest to fail.
Digging further, it appears that dbaron is already onto it. You can view the full story in https://bugzilla.mozilla.org/show_bug.cgi?id=351641, notably in comment 8, "there's actually also a bug in the table sizing behavior, which I think I need to fix in BasicTableLayoutStrategy::ComputeColumnWidths". This is also the bug that is currently causing all of the CSS counter tests to fail that I converted when I first started looking at this test framework. I've been reluctant to check them in while they're all failing, but it seems that since it's a known problem, I probably should.
Where to from here? There's a need for some more flexibility in these sorts of tests. What might be important in one case (table-sizing behavior, for example), might not be for some other type of rendering. The single-pixel error might be difficult or unecessary for certain types of testing where, say, font aliasing comes into play. For those types of cases, a less strict form of image comparison could be useful.
In the meantime, getting more of these tests in place is crucial. They are being looked at, and people like Rob Sayrer, Martijn (mw22) and others are writing more tests regularly. Soon, hopefully within a week or two, we'll have a buildbot reporting these results to a tinderbox on a regular basis. Consider that my holiday gift to you, gentle reader.
Posted by robcee at December 14, 2006 7:14 AM
Comments
No, comment 8 in that bug was about something else, not this issue.
Could you get bugs fixed on these problems? Either the tests need to be fixed or the code needs to be fixed. Many of these tests are designed so that they really should be producing identical results.
Posted by: David Baron at December 14, 2006 3:27 PM
I too think these tests only make sense if you want them to produce identical results. A pixel is still a pixel, and whether it's painted this or that way should be predictable. I can't think of what would be a real false-positive -- could you elaborate on the font aliasing example?
Posted by: Dao at December 14, 2006 4:30 PM
dbaron: I'd certainly like to get these fixed, though I'll probably be knocking on your door for help. These seem to be inconsistencies across platforms in the layout code.
dao: It's true. There's something to be said for an identical, pixel-accurate comparison. I think what I was trying to get at, and missed the mark on was that there may be some cases where slight variations might not matter. I chose font aliasing as an example, but on further reflection, this is an area where you really would want identical results all the time. If my 'A' suddenly started looking slightly different in some cases, I think I'd notice and it would be highly-annoying.
So maybe "fuzzy" matching isn't all that useful after-all.
Posted by: robcee at December 16, 2006 12:35 AM
thanks
Posted by: ticaret at May 19, 2007 8:02 AM
About Short...Demo
Posted by: abrir mi e mail spanish at June 18, 2007 4:05 PM
Great and cool website !!! Keep it up... Best regards... Visit my sites :) Leon-7
Posted by: Robin at June 20, 2007 2:45 PM
Great and cool website !!! Keep it up... Best regards... Visit my sites :) Leon-8
Posted by: foto pornograficas at June 20, 2007 3:17 PM
Leon-9
Posted by: addictinggames at June 20, 2007 3:59 PM
Lion
Posted by: www xxl com at June 20, 2007 4:57 PM
Lion
Posted by: foto pornograficas at June 20, 2007 5:09 PM
Lion
Posted by: foto pornograficas at June 20, 2007 5:18 PM
myfreepaysite
Posted by: Jak at June 28, 2007 2:32 PM
dodge
Posted by: Michael at July 4, 2007 6:42 PM
dodge
Posted by: Anne at July 4, 2007 7:41 PM
dodge
Posted by: Vanessa at July 4, 2007 7:56 PM
dodge
Posted by: Ron at July 4, 2007 8:12 PM
dodge
Posted by: Brytney at July 4, 2007 8:27 PM
dodge
Posted by: Mister at July 4, 2007 8:42 PM
dodge
Posted by: Vanessa at July 4, 2007 9:52 PM
dodge
Posted by: Jak at July 4, 2007 10:06 PM
dodge
Posted by: Rick at July 4, 2007 10:35 PM
dodge
Posted by: Mister at July 5, 2007 12:40 AM
dodge
Posted by: Bob at July 5, 2007 1:10 AM
dodge
Posted by: Lohness at July 5, 2007 1:41 AM
dodge
Posted by: Mister at July 5, 2007 1:55 AM
dodge
Posted by: Mister at July 5, 2007 2:10 AM
dodge
Posted by: Bob at July 5, 2007 2:26 AM
dodge
Posted by: Mister at July 5, 2007 3:10 AM
tramado0l
Posted by: Lohness at August 8, 2007 11:45 AM
tramaeol
Posted by: Mister at August 8, 2007 11:53 AM
tramad0ol
Posted by: Jak at August 8, 2007 12:00 PM
traamdol
Posted by: Mister at August 8, 2007 12:08 PM
tramadokl
Posted by: Lohness at August 8, 2007 12:23 PM
Good site!
Posted by: teryzhijchertuy at November 15, 2007 1:56 PM
Hello http://43.xx48.in/45.html >link
Posted by: weryzhijchertcc at November 15, 2007 1:56 PM
buy tamadol
medical buy tamadol
pharmacy buy tamadol
tramadol
buy tramadol
tablets buy tamadol
tablets tamadol
Posted by: Mister at November 16, 2007 2:26 AM
tramadol
Posted by: Unison at November 16, 2007 2:33 AM
tramadil
Posted by: Bob at November 19, 2007 8:21 AM
tramadlo
Posted by: Jak at November 19, 2007 8:34 AM
tramsadol
Posted by: Luk at November 19, 2007 8:42 AM
trtamadol
Posted by: Lohness at November 19, 2007 8:53 AM
amatuer asian pussy
Posted by: John at November 20, 2007 11:22 AM
anxious pussy
Posted by: Bobi at November 20, 2007 11:27 AM
tramwadol
Posted by: Jak at November 21, 2007 8:50 AM
t5ramadol
Posted by: Bob at November 21, 2007 12:05 PM
tramadoll
Posted by: Bob at November 22, 2007 4:34 AM
pdopecia
Posted by: Jak at November 22, 2007 11:09 AM
somsa
Posted by: Ron at November 22, 2007 12:03 PM
ultramj
Posted by: Mister at November 22, 2007 12:31 PM
ledvitra
Posted by: Ron at December 1, 2007 12:03 PM
lpevitra
Posted by: Guruchel at December 1, 2007 12:16 PM
levitrz
Posted by: Victor at December 1, 2007 12:35 PM
levi9tra
Posted by: Mister at December 1, 2007 12:42 PM
leevitra
Posted by: Luk at December 1, 2007 12:50 PM
pharmacy-25
Posted by: Jak at December 3, 2007 3:40 AM
tablets-50
Posted by: Ron at December 3, 2007 4:04 AM
tablets
Posted by: Guruchel at December 3, 2007 4:13 AM
medical-100
Posted by: Luk at December 3, 2007 4:24 AM
tablets-25
Posted by: Tablets at December 3, 2007 4:34 AM
Levon
Posted by: Xlop at December 3, 2007 1:43 PM
Nikolay
Posted by: Eger at December 3, 2007 1:58 PM
Xolms
Posted by: Fater at December 3, 2007 2:22 PM
Nikolay
Posted by: Kreks at December 3, 2007 2:36 PM
Reks
Posted by: Ron at December 3, 2007 2:49 PM
Sveta
Posted by: Petr at December 3, 2007 3:03 PM
Venera
Posted by: Qwest at December 3, 2007 3:14 PM
Bober
Posted by: Vladimir at December 4, 2007 10:32 AM
Sveta
Posted by: John at December 4, 2007 10:49 AM
Tolik
Posted by: Dmitriy at December 4, 2007 10:56 AM
Tolik
Posted by: Ulia at December 4, 2007 11:02 AM
Bob
Posted by: Hammer at December 4, 2007 11:05 AM
Ivan
Posted by: Irina at December 4, 2007 11:10 AM
John
Posted by: Larisa at December 4, 2007 11:15 AM
Nikolay
Posted by: Lohness at December 4, 2007 11:20 AM
Vladimir
Posted by: Lohness at December 5, 2007 11:25 AM
Victor
Posted by: Weber at December 5, 2007 11:26 AM
Ron
Posted by: Vektor at December 5, 2007 11:43 AM
Levon
Posted by: Jak at December 5, 2007 11:47 AM
Bob
Posted by: Leonid at December 5, 2007 11:51 AM
Olga
Posted by: Minor at December 5, 2007 11:55 AM