July 2011 Archives

It's been quite a while since I blogged last, and I've made it a goal this quarter to resume blogging, because it helps me focus my thoughts, convey some gleanings, and, in general, I hope to grow the awesome community we have.

So, what *have* we done, since my last post? Let's directly address the "Wish List" I last posted:

1. Again, clearly, we've got to fix the staging-server issues

Input and AMO are slower and less reliable than the rest, still, but we have a new staging server up soon for Input, and AMO's performance is constantly monitored and improved upon.

2. Continue to maintain legacy tests when sensible (i.e. the maintenance benefits outweigh the fix costs)

We're still maintaining our "legacy" AMO tests, which provide us a tremendous amount of coverage, even today. Now, however, they are starting to show their rigidity, as we can't easily update them to accomodate A/B testing or rewrites of search, using Elastic Search. Instead, we're focusing our efforts on amo.rewrite, our new suite.

3. As much as possible, outright rewrite the more fragile, huge tests (I'm looking at you, AMO_category_landing_layout.py) into smaller ones, and remove them wholesale from the old suite

We haven't really removed the old AMO tests -- many of which are still providing great coverage. We continue to build out existing AMO tests, and one piece we're missing from the old suite is the dynamic, data-driven tests (which tend to be more fragile).

4. Continue to augment our production smoketest suites for all projects, replacing legacy, slower-running tests with the newer, faster ones

We trimmed the SUMO tests down quite a bit, and have done the same with AMO, too. And both are eminently more reliable than a few months ago.

5. Figure out the trade-off between coverage and time-to-run, especially for smoketests in production, and BFTs in staging (and, ensure all projects have this nice hierarchy, and that it works well for each)

Actively working through this, especially as we transition to more of an Etsy.com-like, continuous-deployment model.

6. Socially-engineer developers to recognize/fix failures that our test-suites catch :-)

Not quite there, yet; but the culture is catching on, and we continue the discussion with projects that don't always unit test.

7. Build-out our Selenium Grid's hub, for more capacity

Not only did we switch over to Selenium Grid 2, but we added Aurora, Beta.

8. Switch over existing tests to Selenium 2 where it makes sense, and write new ones in it by default

We're investigating this with our extended automation team, and Dave Hunt has already written both QMO and wiki.mozilla.org tests in Selenium 2/Webdriver.

9. Switch from Hudson over to Jenkins (because new butlers are always better)

Already done; and we always upgrade to the latest.

10. Mobile...?:

Not there yet -- also looking into it, and David Burns has a working prototype already.

We're continuing to figure out our shared continuous-deployment strategy with AMO and SUMO, and are already shipping directly from the master branch straight to production, with various features that aren't yet ready to be activated, hidden behind feature flags, using Waffle.

We welcome feedback, and of course any and all help: find our more about what we do at our QMO page, or find us on IRC, in #mozwebqa, on irc.mozilla.org.

About this Archive

This page is an archive of entries from July 2011 listed from newest to oldest.

April 2011 is the previous archive.

August 2011 is the next archive.

Find recent content on the main index or look in the archives to find all content.

Pages

Powered by Movable Type 5.12