Three Monkeys, Three Typewriters, Two Days

September 2, 2010

Peacekeeper benchmark weirdness

The Peacekeeper benchmark runs its tests by doing the operation 10,000 times, then dividing one million by the time spent to determine a runs/second number. Unfortunately, the accuracy of this approach is terrible: the operations they're timing don't take that long to do, so they're measuring times in the 0-20ms range for those 10,000 iterations. Given the millisecond accuracy of JS timers, that introduces a lot of noise; furthermore some browsers don't actually update their expeditiously; those would look better than they really are in this benchmark.

It looks like the futuremark folks had some code to try to run for 2s instead of 10,000 iterations (only in IE, although the comment says in non-IE), but they messed up the scoping so that the code is a no-op.

On a separate note, for their Array.splice benchmark even 10,000 iterations is nonsense. The benchmark starts with an array of 100,000 elements and removes 20 elements for each call. After 5000 calls, the calls become no-ops, and the benchmark then times those no-ops.

If anyone knows how to contact futuremark about this benchmark, I'd really appreciate it. I have yet to find useful contact info for that.

Posted by bzbarsky at September 2, 2010 12:27 PM | TrackBack

It looks like they have an office in SF. Maybe we can get a meeting to discuss the issues?

Posted by: Gen Kanai on September 2, 2010 5:18 PM
Post a comment