« November 2006 | Main | January 2007 »
December 20, 2006
News regarding Upcoming Community Test Days
With the holidays in full swing, we will resume our Community Test Days in January. The focus of our first test day in the new year will be on writing some new test cases for Litmus. More details will follow, but you can find a list of the upcoming test days here.
As always, suggestions or new ideas for test days are always welcome.
Posted by marcia at 12:13 PM | Comments (69)
December 19, 2006
Welcome Tony Chung - new QA engineer
Tony Chung has joined MozCorp as a User Based Testing engineer. He will be working with Marcia, Tracy, Juan, Tomcat and others to test milestone releases, develop user level tests, and work with community members testing the product. He can also help us with the L10n testing as he knows Chinese (Cantonese) some Mandarin, some Japanese!
Tony has 8 1/2 years of testing experience. He is in in the process of moving to Mountain View, CA from Seattle, WA, where is last job of 5 years was doing testing at Expedia.com on Firefox and IE. Before that he worked at Myrio Corp tested Video on Demand, digital TV, and interactive web applications.
He started Dec 4th. IRC nick=tchung. Please join me in welcoming Tony!
Posted by timr at 3:27 PM | Comments (66)
December 14, 2006
12/15: Community Test Day - FF 2001 on Windows Vista / Gecko 1.9a1
Community Test day resumes tomorrow! Please come out and help contribute your testing abilities on the latest Firefox builds:
- Firefox 2.0.0.1, focus on testing against Windows Vista
- Gecko 1.9a1, (aka Firefox 3)
Please log on to irc channel #testday on irc.mozilla.org
More information on 12/15 Test Day can be found HERE
Posted by tchung at 2:34 PM | Comments (58)
Test Tool Spotlight: reftest
This morning, I'm going to take a few moments out of my struggles with buildbot (which is coming along nicely, thanks, more about that in a bit) to talk a bit about one of our unsung testing tools, reftest by layout superstar Dave Baron who recently landed the reflow branch to some fanfare and accolades (mad props, Dave!).
Short for "reference test", reftest runs through a list of paired html documents, renders both and converts them to a string-based graphical representation for comparison. Depending upon how the pages are expected to compare, (equals, not-equals, expected pass, or expected fail), the test outputs the pass or failure to the console. In the case of a failure, the log emits the data URL of the graphics that didn't compare.
Currently, this test is rather strict. Because the two images have to be identical for them to match, a difference of a single pixel can throw a failure. During the past couple of weeks, with the landing of the reflow branch, I've seen the tests vary between passing everything and failing everything on each platform, to varying degrees between those extremes. Just now, for example, a Linux build passed completely, Mac OS X failed 1 test (351641-1b.html vs. 351641-1-ref.html) and Windows XP failed 3 tests (142233-1.html vs. 142233-1-ref.html, 351641-1a.html vs. 351641-1-ref.html and 351641-1b.html vs. 351641-1-ref.html). This is out of a total of 74 tests in the current reftest manifest.
Here is an example of what a pair of images looks like from the 351641 test:
Can you see the difference? It's subtle, but the cellpadding on the right is slightly less than in the table on the left -- it's off by about a pixel and that's enough to cause the reftest to fail.
Digging further, it appears that dbaron is already onto it. You can view the full story in https://bugzilla.mozilla.org/show_bug.cgi?id=351641, notably in comment 8, "there's actually also a bug in the table sizing behavior, which I think I need to fix in BasicTableLayoutStrategy::ComputeColumnWidths". This is also the bug that is currently causing all of the CSS counter tests to fail that I converted when I first started looking at this test framework. I've been reluctant to check them in while they're all failing, but it seems that since it's a known problem, I probably should.
Where to from here? There's a need for some more flexibility in these sorts of tests. What might be important in one case (table-sizing behavior, for example), might not be for some other type of rendering. The single-pixel error might be difficult or unecessary for certain types of testing where, say, font aliasing comes into play. For those types of cases, a less strict form of image comparison could be useful.
In the meantime, getting more of these tests in place is crucial. They are being looked at, and people like Rob Sayrer, Martijn (mw22) and others are writing more tests regularly. Soon, hopefully within a week or two, we'll have a buildbot reporting these results to a tinderbox on a regular basis. Consider that my holiday gift to you, gentle reader.
Posted by robcee at 7:14 AM | Comments (80)
December 13, 2006
Thunderbird 2 Beta 1 has been unleashed!
Pick your platform and get started:
Download Windows
Download Mac
Download Linux
MozillaZine feedback thread: Feedback Thread Established at: http://forums.mozillazine.org/viewforum.php?f=29
Feel free to post comments here as well. Thanks!
Posted by marcia at 3:34 PM | Comments (74)
December 8, 2006
Test Tool Spotlight: Simple XUL Test Harness
The simple XUL test harness was designed to solve a single problem: identifying the search engines plug-ins being used by each localization of Firefox. Firefox supports a huge amount of localizations due to the efforts of the volunteer localization teams around the planet. The work done by these volunteers to properly translate any and all English within the browser is a great contribution, but it does have the side effect of leaving QA with a very large amount of different Firefox locales to test. This test harness was a small step in the direction of automating this kind of broad testing. The search toolbar for every locale contains a list of search engines - we need to confirm that the plug-ins are correct and listed in the correct order. Instead of installing, loading and confirming each candidate build by hand, this tool will do each step for us - leaving us with precious free time to drink more coffee, or nap.
While currently this tool fixes a small problem, the framework itself could be extended to handle other tasks. Other tests could be designed around checking the content of various fields in the browser - creating a basic content confirmation suite. It will take time and creativity to come up with new tests that can be developed and integrated into this system. I'd be happy to hear any suggestions. See the wiki doc for instructions on downloading and running this test harness.
Posted by alice at 2:32 PM | Comments (85)
December 5, 2006
Firefox 2.0.0.1 RC1 candidate builds ready for testing!
Those of you enjoying Firefox 2.0 will be happy to know we are getting ready for our first security/stability update! Firefox 2.0.0.1 is just around the corner and we need your help in making sure it's better than the original! :-)
So please grab the latest RC1 candidate builds and let us know how things go:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2.0.0.1-candidates/rc1
Please leave a comment here or email us at testday@mozilla.org with any feedback.
We will also be having a 1.5.0.9/2.0.0.1 Community Test Day this Friday, December 8th...so join us then to help us test and discuss the upcoming releases. More info coming soon...
Thanks!
Posted by jay at 1:43 PM | Comments (92)
December 4, 2006
Next Community Test Day - Mark Your Calendars!
A hearty thanks to everyone that participated in our Community Test Day on Friday. We had a good turnout to the Gecko 1.9 testing event.
Our next Community Test Day will be this Friday, December 8th. I will post details shortly here. We will be testing the candidate builds for Firefox 1.5.0.9 and Firefox 2.0.0.1, as well as Thunderbird 1.5.0.9 and Thunderbird 2 Beta 1. Yes, lots of releases and lots to test. So tell your friends, and get them to come out and try some testing - we will have something for everyone.
For users that want to have some old fashioned web fun, I highly recommended running our fun smoketest - Surf's Up. That particular suite of tests has been designed to cover basic functionality of the browser, but also to have some fun while doing so. Feel free to suggest new test cases that we can add to that suite.
Scott MacGregor, our lead Thunderbird developer will also be in the IRC testing channel (#testday). This is shaping up to be a great test day!
Posted by marcia at 12:31 PM | Comments (58)