May 2010 Archives

A huge thanks to everyone who attended, "How Mozilla and LinkedIn use Selenium: Open Source Customized," here @ Mozilla HQ on May 19th, and, especially, to Sauce Labs (esp. you, John) for driving and co-sponsoring it, and Wade from LinkedIn, who was an excellent co-presenter.

There were about ~120+ folks there, and it was really wonderful to have so many Selenium-interested folks sharing real-world tips and asking intriguing questions.

See you all again, next Meetup!

Photo gallery:

Raymond's slides from the presentation

San Francisco Selenium Meetup group page

This is an *actual* URL from Vignette:

I'm sure that behind those obscure-looking alpha-numeric strings there's a rhyme or reason (hopefully?), but that's not really what I wanted to blog about today :-)

What I did in fact hope to convey today is how to try "breaking" (for various definitions of "breaking") web applications' internal logic (or output) by manipulating their URLs.

Spurred on by our resident web-applications-security guru, Michael Coates, I've been applying some really simple approaches. I'll walk you through just four of them, here (not picking on SUMO, I swear!)

1. Add/remove delimiters/trailing slashes:

Let's take a URL like as an example. Obviously, we make sure that and both return a valid homepage (that's almost a given, but you never know).

The more-interesting test is to add junk after the first trailing slash (which we now know to work), like so: Currently, when you do that on our new version of SUMO, you redirect to http://en-us/forums

Filed bug 566106.

2. Put Unicode/non-ASCII text where it wasn't originally intended:

Prior to a fix, SUMO didn't know what to do with Unicode as a value in its "&tags=" parameter:

Paul Craciunoui found and filed bug 564385.

3. Input non-expected/invalid values for parameters:*Finding+your+Firefox

The above is a long URL, but if you notice, the second parameter is "a=a". The original problem was that SUMO was expecting the value of "a=" to be an integer. So, of course, I fed it an alphabetical character, "a".

The result of which can be seen in bug 565857.

4. Try changing the locale code:

If your web app likes its URLs in a certain format, such as "en-US", try simply omitting the "-US", or "US", leaving, respectively, "en" and "en-".

As a real-world example, both the former, and the latter, redirect us to, which is graceful.

These are, of course, just four really simple examples, but they highlight how quickly and easily testers can begin to help ensure your app won't crash/hang/do something evil with bad data; there are a myriad of other ways, and the URLs in the location bar aren't the only target: try using a add-ons, such as Tamper Data, or Live HTTP Headers (which I've already blogged about for its primary use), to change sent values (on GETs/POSTs, etc.), especially when submitting forms.

Once you know how to manipulate _one_ parameter (irregardless of if you know the expected value, or, maybe in spite of that), you can find some gems.

Feel free to get pathological, too, just be mindful that some constructs are just too heinous for words, and a developer is likely to cast you the stink eye if it's too wild.

Check this out; confidence-inspiring, no?

stephen-donners-macbook-pro:smokeTests stephendonner$ python 
ERROR: test_searchapi (__main__.SearchAPI)
Traceback (most recent call last):
  File "", line 65, in tearDown
    self.assertEqual([], self.verificationErrors)
AssertionError: [] != ['type:firebug,dict']

Ran 1 test in 8.395s

FAILED (errors=1)
stephen-donners-macbook-pro:smokeTests stephendonner$ python
Ran 1 test in 6.174s


Sometimes, we return 33 results, and yet at others, we return 0.

Here's what the tests are actually doing:

  "/en-US/firefox/api/1.2/search/firebug type:dict")
            retVar = sel.is_element_present("//searchresults[@total_results=\"0\"]")
            if retVar == False:
                raise AssertionError               
        except AssertionError, e:
            self.verificationErrors.append(str(e) +'type:firebug,dict')
   "/en-US/firefox/api/1.2/search/firebug type:extension")
             if ("Firebug" != sel.get_text("//name")):
                raise AssertionError
        except AssertionError, e:
            self.verificationErrors.append(str(e) +'type:firebug,extension')


I'll be filing a bug shortly.

(This post assumes its readership already has a Python environment set up, as well as Selenium RC; a quick Googling should provide ample help on getting both install/configured.)

It's easy to convert Selenium tests written in its native "Selenese" language (through the IDE) over to, say, an object-oriented programming language like Python, but not always so easy to fix them, especially since in OOP there's is a bunch of different ways to trap/report errors.

I'm only going to cover the basics, here, because my Python skills aren't quite up to snuff (and it might take some time, given my other responsibilities); this should, however, give you a good idea of the syntax changes and, more importantly, encourage those of you who might be interested in taking on such an endeavor.

Let's get started; this assumes you've already written a Selenese-based test through the IDE. If you haven't, you should first go through their tutorials, or just take a look at some of our IDE tests to get a better idea.

I'll use searchapi.html as our example (written by the ever-intrepid Dave Dash). Fire up Selenium, open the file, and then do Options | Format -> Python - Selenium RC.


Your IDE's output should now look like the following:


Now, just copy and paste that into your favorite text editor/IDE/VIM/whatever; as you can see from this line:

self.selenium = selenium("localhost", 4444, "*chrome", "http://change-this-to-the-site-you-are-testing/"), we're not ready to run the test just yet (for those of you already familiar with the IDE, that's the baseURL you'll find in the HEAD); so, it should be this:

self.selenium = selenium("localhost", 4444, "*chrome", "")

  • Save the test
  • Launch your RC (java -jar selium-server.jar from the selenium-server-1.0.3 folder)
  • Drop into a terminal/console, and do: python

You'll get this error:

ERROR: test_searchapi (__main__.searchapi)
Traceback (most recent call last):
File "", line 76, in tearDown
self.assertEqual([], self.verificationErrors)
AssertionError: [] != ["'YSlow' != u'SenSEO'"]

FAIL: test_searchapi (__main__.searchapi)
Traceback (most recent call last):
File "", line 32, in test_searchapi

Ran 1 test in 20.134s

FAILED (failures=1, errors=1)

Obviously, all the tests were passing when they were first written, but to fix these, we now need to do a couple things:

  • Manually run the search API URLs and compare the expected output with the real data (since it changes)
  • Fix our tests not to fatally assert (and stop running the remaining tests) on such outdated data, but propagate the error to the console so we can run them all and fix individual failures

Most of the failing tests are because we open a URL and assert that there should be *no results*, e.g. self.failUnless(sel.is_element_present("//searchresults[@total_results=\"0\"]")); this won't be true now since new metadata (such as keywords, descriptions, summaries) changes, so we should either be more explicit in for what we're searching, or more lenient. In reality, a smart combination of both is needed, so Vishal came up this:

retVar = sel.is_element_present("//searchresults[@total_results=\"0\"]")
if retVar == False:
raise AssertionError
except AssertionError, e: self.verificationErrors.append(str(e) + 'apps,yslow')

Line by line, it:

  • opens the API URL
  • stores the value of |true| or |false|, depending on whether the total results is indeed 0 or not, and if so,
  • raises an assertion (but doesn't fatally die)
  • adds the assertion to the verificationErrors array and adds "apps,yslow" to the assertion string so we know which add-on failed

If we change all of the calls to that format, the complete test will run (though still spew a ton of failures); the next step is to figure out what the right tests are and remove/edit them; Vishal's steps will at least let us run the whole test and see all errors at the same time:

ERROR: test_searchapi (__main__.SearchAPI)
Traceback (most recent call last):
  File "", line 183, in tearDown
    self.assertEqual([], self.verificationErrors)
AssertionError: [] != ['type:firebug,dict', 'platform,linux', 'limit, firebug, all', 'mobile filtering']

Ran 1 test in 45.996s

Now, on to fix these tests; until next time...

About this Archive

This page is an archive of entries from May 2010 listed from newest to oldest.

April 2010 is the previous archive.

June 2010 is the next archive.

Find recent content on the main index or look in the archives to find all content.


Powered by Movable Type 5.12