May 2, 2010

Conversion process from Selenium IDE to Python (and its pitfalls/gotchas)

(This post assumes its readership already has a Python environment set up, as well as Selenium RC; a quick Googling should provide ample help on getting both install/configured.)

It's easy to convert Selenium tests written in its native "Selenese" language (through the IDE) over to, say, an object-oriented programming language like Python, but not always so easy to fix them, especially since in OOP there's is a bunch of different ways to trap/report errors.

I'm only going to cover the basics, here, because my Python skills aren't quite up to snuff (and it might take some time, given my other responsibilities); this should, however, give you a good idea of the syntax changes and, more importantly, encourage those of you who might be interested in taking on such an endeavor.

Let's get started; this assumes you've already written a Selenese-based test through the IDE. If you haven't, you should first go through their tutorials, or just take a look at some of our IDE tests to get a better idea.

I'll use searchapi.html as our example (written by the ever-intrepid Dave Dash). Fire up Selenium, open the file, and then do Options | Format -> Python - Selenium RC.

Screen%20shot%202010-05-02%20at%2012.20.18%20PM.png

Your IDE's output should now look like the following:

Screen%20shot%202010-05-02%20at%2012.22.26%20PM.png

Now, just copy and paste that into your favorite text editor/IDE/VIM/whatever; as you can see from this line:

self.selenium = selenium("localhost", 4444, "*chrome", "http://change-this-to-the-site-you-are-testing/"), we're not ready to run the test just yet (for those of you already familiar with the IDE, that's the baseURL you'll find in the HEAD); so, it should be this:

self.selenium = selenium("localhost", 4444, "*chrome", "https://preview.addons.mozilla.org")

  • Save the test
  • Launch your RC (java -jar selium-server.jar from the selenium-server-1.0.3 folder)
  • Drop into a terminal/console, and do: python searchapi.py

You'll get this error:

======================================================================
ERROR: test_searchapi (__main__.searchapi)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test.py", line 76, in tearDown
self.assertEqual([], self.verificationErrors)
AssertionError: [] != ["'YSlow' != u'SenSEO'"]

======================================================================
FAIL: test_searchapi (__main__.searchapi)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test.py", line 32, in test_searchapi
self.failUnless(sel.is_element_present("//searchresults[@total_results=\"0\"]"))
AssertionError

----------------------------------------------------------------------
Ran 1 test in 20.134s

FAILED (failures=1, errors=1)

Obviously, all the tests were passing when they were first written, but to fix these, we now need to do a couple things:


  • Manually run the search API URLs and compare the expected output with the real data (since it changes)
  • Fix our tests not to fatally assert (and stop running the remaining tests) on such outdated data, but propagate the error to the console so we can run them all and fix individual failures

Most of the failing tests are because we open a URL and assert that there should be *no results*, e.g. self.failUnless(sel.is_element_present("//searchresults[@total_results=\"0\"]")); this won't be true now since new metadata (such as keywords, descriptions, summaries) changes, so we should either be more explicit in for what we're searching, or more lenient. In reality, a smart combination of both is needed, so Vishal came up this:



    try:

            sel.open("/en-US/sunbird/api/1.2/search/yslow")

            retVar = sel.is_element_present("//searchresults[@total_results=\"0\"]")

            if retVar == False:

                raise AssertionError

        except AssertionError, e: self.verificationErrors.append(str(e) + 'apps,yslow')

Line by line, it:


  • opens the API URL
  • stores the value of |true| or |false|, depending on whether the total results is indeed 0 or not, and if so,
  • raises an assertion (but doesn't fatally die)
  • adds the assertion to the verificationErrors array and adds "apps,yslow" to the assertion string so we know which add-on failed

If we change all of the calls to that format, the complete test will run (though still spew a ton of failures); the next step is to figure out what the right tests are and remove/edit them; Vishal's steps will at least let us run the whole test and see all errors at the same time:

ERROR: test_searchapi (__main__.SearchAPI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "searchAPI.py", line 183, in tearDown
    self.assertEqual([], self.verificationErrors)
AssertionError: [] != ['type:firebug,dict', 'platform,linux', 'limit, firebug, all', 'mobile filtering']


----------------------------------------------------------------------

Ran 1 test in 45.996s

Now, on to fix these tests; until next time...

Posted by stephend at May 2, 2010 11:03 AM
Comments
Post a comment