performance testing javascript

| 20 Comments

John Resig just posted on some of the problems and a proposed solution for the current sets of public JavaScript benchmarks. It's well worth a read and even a mostly non-technical person like me got through it just fine and came away much better informed.

One thing did strike me, though, as I read through John's post. John talks about Apple's SunSpider, Google's V8 Benchmark, and Mozilla's Dromaeo. Where are Microsoft and Opera in this game (besides falling way behind, I mean.) Why haven't they put their JavaScript benchmarks out in the public space so we can all benefit the way Apple, Google, and Mozilla have?

Is this just another case where open source proves superior? You've got three open source projects all developing JavaScript engines with public benchmarks and lots of public discussion and idea sharing. And you've got two closed and proprietary companies developing JavaScript engines completely in secret without public benchmarks or any sign of public conversation.

The open source engines are rockin' the speed game and the closed engines are falling very far behind very quickly. It looks to me like the closed and proprietary guys just can't keep up.


Photo "No longer rocking" by Gary Simmons used under a Creative Commons license.

It's actually a shame that they're falling so far behind. It would be helpful for the Web if the IE team was able to make some serious JS improvements for 8 and whatever comes next because there are still a significant number of users stuck on Microsoft's technology.

Their falling behind won't completely stop the Web from moving forward though. There's enough momentum behind the truly modern browsers, Firefox leading the way with 20% of the market, that the Web will get better without them.

IE market share will continue to deteriorate, probably at a somewhat faster rate, as users flee for a better experience. And MS will no doubt slow things down at least some so I really do hope they improve. The less of a drag they are the better.

20 Comments

"Why haven't they put their JavaScript benchmarks out in the public space"

My guess? Microsoft doesn't have a JavaScript benchmark. If you look at IE, or any other Microsoft product for that matter, can you see very much evidence that Microsoft cares about performance? I sure can't...

Jake, I'm willing to give Microsoft and Opera Software the benefit of the doubt. It really would be odd, with the emergence of a JavaScript-powered Web 2.0, for two browser vendors to not be building and using JS performance tests.

I think it's more likely that they don't have the same "for the betterment of all" approach to the Web that we do and so it just doesn't occur to them to make their performance tests available.

There's some evidence against that, though. They have both released some compliance tests for CSS and HTML I believe. Perhaps they're too concerned about the potential negative PR of releasing a test that they do poorly on. That would be a real shame but it seems like another possible explanation.

That would fit with their policies of not disclosing security flaws that were found "internally" to keep the public count of known (even fixed) flaws as low as possible. PR does drive a lot more in those companies than it does for Mozilla.

Then again, Google and Apple, while both of them are making their code available to others under open source terms, are also keeping quite a bit secret as long as they can.

Really hard to say in the end. Opera and MS could help us out by telling us all what's going on with their JS performance testing :-)

- A

Does accusing other browser vendors of keeping their tests for themselves count as "giving them the benefit of the doubt"? That sounds like just the opposite to me! :p
Giving them the benefit of the doubt would be guessing they have nothing to hide, and that they just don't have any in-house JavaScript tests.

That wouldn't be so surprising: the IE team doesn't care, and the Opera one isn't that big, and they already have plenty of available tests to check out.

Anyway, yeah, open source is the clear winner, and that's good!

sry, but this text is settled somewhere between fanboyism and open-source-nazi-ism.

Micrsoft had a post on the IE blog that for the top 100 sites javascript made up less 10% of their performace figures and even on heavy javascript sites it was only about 25% of the performance footprint.
http://blogs.msdn.com/ie/archive/2008/08/26/ie8-performance.aspx

We should really wonder why FF and Chrome are measuring their JS performance only.

And where is Mozilla in the acid 3 rase? Dont they care about standards? Was that just a example where Mozilla was inferior? So can't you keep it up?

Asa stop being a troll!

blabla: I think you're the one being a troll here. Firefox (Minefield) is currently at 85/100 on the Acid3, that's not a shameful score to me (same score as the latest Opera 9.60 build), and certainly doesn't amount to "not caring about standards". And it's slowly but surely going up. Go and bitch to IE for barely reaching 20.

Hey Asa

The improvement in Javascript performance is great. But Firefox has abysmal DOM performance compared to other non-Microsoft browsers.

Just take the basic test on here which repeats the DIV tag over and over again:
http://nontroppo.org/timer/progressive_raytracer.html

You can't even take the full test on this without expecting your browser to be frozen up for a good few minutes, the latest Opera does the full test about 12x quicker.

I love Firefox performance, but it's really time DOM got some TLC from the Mozilla crew :D

Damian: yeah... That's this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=395635
I wish it would get attention.

I don't think that ctitisizing other browser method when, it's not fully alleged they are wrong, is a good way to help them improve.

By the way, 1 Godwin point for Klaus ;)

Hi Stifu

It's also this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=424715

Which I wrote, I have full bugedit status, I'll mess about merging them in a minute.

Damian: alright, thanks. I had actually already voted for both bugs, but I didn't realize the dupe.

This is the current state of things, but I really wouldn't assume it won't change.

Strategically, very important things are happening this month.
Google has started a new browser war and JavaScript performance is a significant pivot point it intends to use as a weapon in this fight.

Microsoft has to choose between :
- do nothing and watch Google progress toward its aim of making the OS irrelevant, hoping some miracle will make them fall and disappear into some deep hole on the way, or
- fight back.

Now the trouble for Microsoft in fighting back with a better browser is that it will help making the OS irrelevant. It's a catch 22.
But still it's better to own the most popular browser than to see Google impose it's own. Gaining 1% market share in one week for a completely new browser is simply huge. IMHO it rules out for Microsoft any option of trying to ignore the threat.

So I thing it's very likely we'll soon see Microsoft put all it has in fighting back Chrome.
And one of Microsoft strongest assets is it's compiler and code optimization team. VC++ has consistently beaten gcc for speed (and size) for years. So on that point they know how to beat open source. So I wouldn't bet too much on the situation staying the same.

Complement for my previous comment : Your other post "javascript ascendent" shows what kind of story Microsoft is likely to put behind this.

IMO It'll be : "If you to develop fast and easily, use C# and Silverlight and not this odd, freaky, weird JavaScript language that definitively is painful to use and a hindrance to fast development ; but if you do want to use JavaScript, take advantage of our browser, platform, VM to make even faster than Chrome and Firefox"

Generally speaking, Microsoft looks for profit, and it will have improved Javascript engine when its time that it influences bottom line (I guess).

If you want to improve Javascript in browsers, just make some killer open source web application and you'll got it. Otherwise, why would you improve Javascrpit that no one uses? (most probable Microsoft's view, and pretty fair for for profit company)

Hi Ivan

"Otherwise, why would you improve Javascrpit that no one uses?"

You think no one out there has AJAX intensive websites and would like to push the boar further? In fact do you not remember when Microsoft first flirted with AJAX for Hotmail that they quickly discovered that it worked too slowly in IE?

Internet Explorer 8 has a profiling engine built into it. If the community builds a suite of tests based on John's reasoning we should be able to use firebug or IE8's profiler to do determine the performance differences. Is that a good use of the profilers? Maybe. But we have to start somewhere. What I find most interesting is that IE8's profiler and firebug return very different results to the point where I don't feel like I can make one improvement in the code that will benefit both browsers. The engines are definitely unique creations.

Why do you think JavaScript performance is the main argue when people choose the browser? As a creator of many good languages/compilers, a bright .NET Platform and many very good applications, Microsoft can not answer to your JavaScript optimization, am I right? Man, please be gentle, and don't say Mozilla the best. I think, Microsoft is not concentrating on performance. It is the best answer for your question: "Why haven't they put their JavaScript benchmarks out in the public space".

Why do you think JavaScript performance is the main argue when people choose the browser? As a creator of many good languages/compilers, a bright .NET Platform and many very good applications, Microsoft can not answer to your JavaScript optimization, am I right? Man, please be gentle, and don't say Mozilla the best. I think, Microsoft is not concentrating on performance. It is the best answer for your question: "Why haven't they put their JavaScript benchmarks out in the public space".

Why do you think JavaScript performance is the main argue when people choose the browser? As a creator of many good languages/compilers, a bright .NET Platform and many very good applications, Microsoft can not answer to your JavaScript optimization, am I right? Man, please be gentle, and don't say Mozilla the best. I think, Microsoft is not concentrating on performance. It is the best answer for your question: "Why haven't they put their JavaScript benchmarks out in the public space".

Monthly Archives