Three Monkeys, Three Typewriters, Two Days

June 30, 2009

Performance testing pitfalls

Someone recently asked why it is that SFX (e.g. in Safari 4) scores 1000runs/s on the Dromaeo CharAt test while Spidermonkey (e.g. in Firefox 3.5) scores something closer to 80runs/s. Naturally, I pulled out my profiler and looked. I discovered some interesting things:

  1. About 2-3% of the time the Dromaeo harness measures for the charAt test is actually spent running the str_charAt function in Spidermonkey. The rest is spent inside the harness itself.
  2. If I pull the test out of the harness, and instead of the harness' runs/second measurement just directly measure how long it takes to run the test 1000 times, I get 430ms or so in Firefox and 750ms or so in Safari. That corresponds to scores of 2325runs/s and 1333runs/s for Firefox and Safari respectively.
  3. If I rerun the same standalone test in Firefox with jit disabled, I get numbers closer to 7300ms.

Conclusions: The harness is measuring pure JS-execution overhead in both browsers, not actual charAt performance. The harness causes us to somehow not trace the test when running inside of it, leading to the numbers seen above.

Update: Filed a bug on this.

Posted by bzbarsky at June 30, 2009 10:13 AM