February 16, 2011

The case for "longevity/endurance" (session-based testing) in Selenium

In "Hey, Selenium. Hurry The Hell Up," Sean Grove from Sauce Labs makes a great general point:

"Write your tests to only test a single tiny bit of functionality.
Keep the number of steps in your tests under 20. All the parallelism
in the world won’t save you from long-running tests, because a single test can’t be parallelized.
The best written tests we’ve seen generally run in under 10 seconds."

In general, I agree with that; there are real-world cases, though, where longer, "endurance" or "longevity" tests, as I've heard them referred to, are necessary, or just plain helpful.

As such an example, we recently migrated our Firefox Support website (SUMO) from our San Jose datacenter to a much better build-out, infrastructure-wise, in our new Phoenix datacenter. Because we have a pretty good (though certainly not comprehensive) set of non-volatile tests we run in production pre/during/post-push, we simple edited our /etc/HOSTS files on our Grid environments to point to the staged SUMO instance in Phoenix, and ran our tests, tagged with "prod," against that host while load-balancers, master and slave DBs, etc. were configured and enabled.

Our tests kept passing -- logins were working, search was eventually fine (thought we had to re-configure Sphinx a few times), Knowledge Base articles were zippier than ever, etc.

We didn't, apparently, do enough manual testing around one thing, though: sessions. Yes, account-creation was fine, logins were working, posting was fine, etc., through manual testing, but one thing we noticed and eventually figured out was that user sessions weren't being maintained for more than, I think, 15 seconds, sporadically; other times, it was fine.

(iirc, the problem was that a Zeus VIP was pointing at our SJC colo to re-use sessions, which were being created in PHX, most of the time. jsocol will undoubtedly offer a correction if I'm wrong.)

In our postmortem, we noticed that our Selenium tests -- while testing each piece of functionality just fine, weren't set up to verify and assert that user sessions lasted for a certain length of time, since each was designed to be small, independent, and of course, because they're Selenium, get completely new sessions for each test, every time tearDown() gets called.

The beauty of Selenium is that, unlike unittest or other frameworks which use mock browsers rather than the real thing, it offers true end-to-end and environment testing, for which Vishal wrote test_forum_post_on_prod to ensure that future roll-outs (as well as sessions in general in production) are being maintained properly. This time, the seemingly arbitrary session limit was 15 seconds, but his new test, even on a fast production cluster, should run closer to a minute, giving us more confidence that sessions are maintained.

Again, this doesn't negate the "just the essentials for each function" test approach wisely advocated by Sean and others, it just offers an additional viewpoint and use-case for how Selenium might be used; we rely on our production smoketest/sanity suites for our largest projects, for good reason -- and now we have another one: we know that users' sessions are doing fine at a push of the button, and continuously, in our Hudson (soon to be Jenkins) runs.

Posted by stephend at February 16, 2011 8:23 PM
Post a comment