Socorro, Sauce Labs, Selenium: How video support saved me a lot of headache

| No Comments

In addition to co-hosting/sponsoring many Selenium Meetups, we've been really grateful to have a generous and accomodating business relationship with Sauce Labs, which allows us to augment and complement our own custom-built Selenium Grid testing infrastructure with theirs. For our most recent Socorro release, I and the whole team (dev +QA) really came to appreciate one feature in particular: video, and for two reasons.

To get you familiar with Mozilla's Selenium Grid setup, let me back up just a second; Dave Hunt has previously highlighted his pytest-mozwebqa plugin (still in beta) for our WebQA team, which--among all the many other awesome things it does--allows us to very easily use Sauce Labs' API to integrate with their OnDemand service.

Inspired by the wealth of build-reporting information we get from Sauce, Dave set out some time ago to incorporate some of it--particularly capturing screenshots--as well as writing out HTML reports that link us to (if the job was Sauce-run) a Sauce Labs job #, complete with video for each tests. We looked at incorporating video a bit into our own infrastructure (as did David Burns in Lights! Camera! Action! Video Recording of Selenium), but soon realized that--given all the other challenges of shipping, on average, 7 websites per week, and dealing with our scale of 6 Mac Minis, archiving, disk space, etc.--it wasn't something we wanted to pursue, for now.

For this particular Socorro release, we thought we had--immediately after we deployed--a huge performance regression. See bug 718218 for the gory details, and all the fun that ensued (over the weekend!).

(Click on the image to view the test run.)

Typically, video is a "nice to have" feature, for us, as our team is quite familiar with debugging failing Selenium tests locally, and our HTML reports and screenshots go a long way, too; for this release, though, I was playing back-up for Matt Brandt, and, as the requisite pointy-hair, couldn't code or run a Selenium test to save my life (not quite true, but close!) When I began investigating why our Socorro test suite suddenly ran ~20 minutes longer than it ever conceivably should, I discovered it was in one of the search tests, test_that_filter_for_browser_results, which we later realized had an erroneous |while| clause that never eval'd to |true|.

While I first looked on our own Grid instance, to try to watch (via VNC) our browsers running this particular test, Grid's distributed test-execution nature means that it never predictably ran on the same node each time. (In retrospect, I could've used pytest's keyword (-k) flag from a job in Jenkins to run the suspect test in isolation, but I digress.) I needed a way to quickly show our dev team the individual test, without having them or me set up a test environment, and firing off that test. Luckily, since Dave made the default for our Sauce Labs jobs public, all it took was a single run of the job via Sauce, find the individual test, and copy the link, which has nice annotated timestamps for each Selenium command, and quick paste in #breakpad.

Earlier, I mentioned that their video support is great for two reasons: in addition to the above (the ability to view archived videos from past-run jobs), we also took advantage of the ability to watch the individual test--as it ran in real-time--and correlate the currently-issued Selenium command with the hold-up/"perf regression."

We quickly isolated it to the aforementioned, bogus |while| loop, commented it and the accompanying assert out, and certified the release after more careful checking. Because it was the weekend, and folks were available at different times, once each member popped onto IRC, we simply showed them the "before and after" videos from Sauce Labs, and the team regained its confidence in the push.

So, thanks, Sauce, for such a useful feature, even if it's (thankfully) not necessary most of the time!

Mozilla, and WebQA in particular, looks forward to continuing to work with Sauce Labs and the Selenium project, in exploring ways in which we can work together to help further and bolster Selenium and Firefox compatibility.

Leave a comment

About this Entry

This page contains a single entry by Stephen Donner published on January 23, 2012 6:00 PM.

Mozilla WebQA, blessed with awesome contributors like Sergey Tupchiy was the previous entry in this blog.

Testing help wanted on Mozilla Reps! is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.


Powered by Movable Type 5.12