October 29, 2010

some sunspider numbers

I mentioned in a previous post that I'd get some Sunspider scores in the full browsers (Are We Fast Yet is testing just the JS shells with no browser wrapped around them). Testing full browsers allows me to add Opera to the list.

The tests were performed on a brand new Windows XP install. For each browser I restarted between each run. I ran the test 5 times for each browser, discarding the two scores with the highest error bars and then tossing the high and low score to come up with the run that made it into the chart below. For all browsers I tested the most recent shipping version and the most recent pre-release version (Chrome offers nearly daily updates on the Canary channel, Opera offers "weekly" updates, Webkit seems to make a new Windows build every few days, and Firefox has both mozilla-central (m-c) nightly and the JS branch (tm) nightlies.) For Firefox I included the last shipping version, the last beta, the current m-c nightly and the current tm nightly.

I did test IE 8 but the scores were so slow that it completely obscured any distinctions between the top several browsers so I removed it from the chart. The score for IE 8 was 3357.4 ms. IE 9 won't be available for Windows XP so it is not included in this chart. (I will follow-up with a Win 7 test, but given that most Web users are still on XP, I figured I'd start there.)

Here's the chart:

So there you have it. The top browsers, Chrome, Opera, and Firefox are all within about a hundredth of a second of each other, but the latest Mozilla tm build is in the lead.

update A few people around the office and in email asked and so i'm sharing my thoughts with all of you. Yes, all of the modern browsers are reaching the limits of this particular set of tests.

Sunspider was designed before any of the browsers had these truly modern JS engines with just in time compilers and because of that and all the progress each browser vendor has made over the last several years Sunspider is no longer particularly useful as a JS benchmark. This is kind of obvious when you see that all of the top scores are pretty much tied. One one hundredth of a second (across 26 tests) separates the slow from the fast and that's just not particularly meaningful.

This has been known by all the browser vendors for some time, but it was more difficult for me to say it when Firefox was behind on the test -- it was just too easy to call that sour grapes. But now that we're leading, I think it's a good time to say that yes, the difference between the browsers at the top of Sunspider is so small, and the headroom for improvement so shallow for all of us, that this test is no longer interesting.

It's time we all turn our attention to V-8 and Kraken which still challenge all of the browsers in ways that will provide meaningful benefits out on the actual Web. If we're all really going to enable the next generation of powerful Web applications, all of the browser vendors need to start to focus on the kinds of tasks where JavaScript is not yet a viable replacement for native (compiled) code.

Kraken starts to shed light on some of these areas and V8 isn't done yet highlighting others. But the truth is that after the next round of browser releases, Firefox 4, IE 9, Safari 6, Chrome 8/9, and Opera 11, Sunspider just doesn't help anyone any more and we really should treat it the way we do the original CSS box acid test -- don't regress, but don't bother checking up on it with every new release. We've all won and it's time to move on.

Posted by asa at 2:32 PM