Three Monkeys, Three Typewriters, Two Days

September 21, 2008

Poor test writing, part 1: Celtic Kane JavaScript speed test

This is the first in what will probably be a series of posts about poorly-written tests. Writing good tests is hard, especially without having some idea of how browsers work on the inside; these posts are meant to point out some of the pitfalls, not to criticize.

I often see the Celtic Kane JavaScript speed test cited as a performance test people care about. This test has several sub-tests, some of which I would like to examine in more detail.

Layer movement.

This test is simply not testing what it thinks it's testing. Here's the entire test:

  var layer1 = document.getElementById('layer1');
  for( var i = 0; i <= 8000; i++ )  {
    layer1.style.x = i + 'px';
    layer1.style.y = i + 'px';
  }

First, and most obviously, the test is simply testing the speed of setting "expando" properties on a CSSStyleDeclaration object, since there are in fact no CSS properties named "x" and "y". I'm not sure why this is an interesting thing to test, or what it has to do with "layer movement". Even if the test were setting "left" and "top", it wouldn't be testing "layer movement", since all UAs I'm aware of do most of their layout and such asynchronously and this test is not leaving time for that. All the test would measure in that case is the amount of time it takes for the UA to record the information needed to do a layout some time later. Again, I'm not sure why this is an interesting thing to test.

Random number engine.

No attempt is made to check whether the numbers are actually random, or even non-constant. A UA that simply returns 0, or returns poor random numbers (using a poor random-number generator) would automatically do better on this test, though it's not clear that you'd want to use the return value of random() for anything serious in such a UA.

DOM speed.

This test tests setting a single DOM property: innerHTML. It sets it to a string with no actual HTML in it. I'm not sure whether this is a common performance bottleneck in the real world, but results on this test change radically in various UAs once there is an actual tag name in the string (a situation that I suspect is more common than the plaintext one). Once again, the test doesn't actually measure the time needed to lay out or render the newly-created DOM nodes, but this is probably fine for a test that claims to be about "DOM speed". It would be a much more useful test if it tested a wider variety of methods and properties, and in more realistic ways.

Array functions.

In pratice, this test basically tests the performance of reversing an array and sorting an array that is in exactly reversed order. There is an Array.push() call, but the time it takes is overwhelmed by the sort() and reverse() calls. This seems like a very uncommon and quite synthetic workload. In particular, there are plenty of array-sorting algorithms that would perform quite well on this workload and poorly on more random arrays, and vice versa.

String functions.

This test tests string concatenation and String.slice(). There are two lines in this test that are no-ops:

  str.toLowerCase;
  str.toUpperCase;

I'm not sure whether the test author thinks he's testing toUpperCase() and toLowerCase() performance here; I hope he doesn't.

Ajax declaration.

This test tests the performance of creating an XMLHttpRequest object and doing absolutely nothing with it. I'm not sure why this is a useful test, since it can be easily gamed by deferring all initialization work until the open() call. That would make the time from creation start to open() complete longer, but speed up creation time.

Before someone suggests it, I did try contacting the test author several weeks ago. There has been no response of any sort.

Posted by bzbarsky at September 21, 2008 11:52 AM | TrackBack
Comments

I doubt you'll get any response. The test results don't cover the most recent versions of any browser, I suspect the test has been long abandoned.

Posted by: Ben Basson on September 21, 2008 1:10 PM

Oh well, forget that, I somehow missed the "update" link at the top.

Posted by: Ben Basson on September 21, 2008 1:11 PM

Interesting breakdown. In particular the DOM. The other day I blogged about how performance benchmarks will move beyond JS to the DOM and rendering soon enough. Though I think much more than just innerHTML will be at play. To be perfectly honest, sometimes I use innerHTML because it's faster than other more DOM-centric approaches.

Posted by: Robert Accettura on September 21, 2008 5:41 PM

Nothing wrong with innerHTML; I just think that labeling an "innerHTML with plaintext speed" benchmark as "DOM speed" is pretty weird.

Posted by: Boris on September 21, 2008 6:24 PM
Post a comment