Three Monkeys, Three Typewriters, Two Days

June 30, 2009

Performance testing pitfalls

Someone recently asked why it is that SFX (e.g. in Safari 4) scores 1000runs/s on the Dromaeo CharAt test while Spidermonkey (e.g. in Firefox 3.5) scores something closer to 80runs/s. Naturally, I pulled out my profiler and looked. I discovered some interesting things:

  1. About 2-3% of the time the Dromaeo harness measures for the charAt test is actually spent running the str_charAt function in Spidermonkey. The rest is spent inside the harness itself.
  2. If I pull the test out of the harness, and instead of the harness' runs/second measurement just directly measure how long it takes to run the test 1000 times, I get 430ms or so in Firefox and 750ms or so in Safari. That corresponds to scores of 2325runs/s and 1333runs/s for Firefox and Safari respectively.
  3. If I rerun the same standalone test in Firefox with jit disabled, I get numbers closer to 7300ms.

Conclusions: The harness is measuring pure JS-execution overhead in both browsers, not actual charAt performance. The harness causes us to somehow not trace the test when running inside of it, leading to the numbers seen above.

Update: Filed a bug on this.

Posted by bzbarsky at June 30, 2009 10:13 AM | TrackBack
Comments

Actually, looks like in SFX Dromeo *mostly* measures time spent in charAt, since you're getting 1000runs/s vs 1333runs/s without the harness. So the harness is just adding a 30% overhead.

But I agree, it seems like the Dromeo test harness is overly complex, IIRC we've had issues with it before.

Posted by: Jonas Sicking on June 30, 2009 2:30 PM

Jonas, there's still js-execution overhead in SFX outside the harness; there's just a bit less than inside the harness. That is, the harness makes us fall off trace, but doesn't make SFX do the equivalent.

Posted by: Boris on June 30, 2009 2:44 PM

SFX was renamed Nitro.
Now there's a Nitro Extreme branch in webkit svn.

Posted by: RichB on June 30, 2009 3:17 PM

Nitro is just a marketing name for the engine:

http://www.nabble.com/Re%3A-Differences-between-Nitro-and-Squirrelfish.-p22211722.html

Posted by: Kalle Vahlman on June 30, 2009 11:18 PM

It'd be nice to get this into 3.5.1.

Posted by: Ian M on July 1, 2009 5:30 AM

Ian, I suspect the changes required to make this sort of closures-that-modify-things-they-close-over stuff to trace are going to be far too invasive to land in 3.5.x... I can't speak for the js team in terms of how they'll weigh the risk, though.

Posted by: on July 1, 2009 10:47 AM
Post a comment