I haven't looked at this very closely but I know people are always asking these kinds of questions, so here's a neat site called Browser speed comparisons. Let me know what you think.
Posted by asa at February 15, 2005 04:27 PMhmmm... I don't really like it I can say that much. And it's not even so much that firefox/mozilla come out not in the lead. But that It lacks reproducability and scientific rigor.
First he fails to give us the tools to repeat the browser start up tests ourselves. He used a JS speed test I've never felt stood up very well.
In the end I just don't really get it. I don't believe that there can be a study that comes out like this that hands us the truth. It's to individual to each machine and set up.
For example on my machine running opera 7.5.4 and Firefox 1.0 testing the same CSS rendering page. I get number almost precisely backwards of his (800ms for firefox and 1300 for opera). This doesn't exactly inspire within me the kind of confidence that would be required for me to feel like this study said something about relative speeds.
Other things strike me as interesting
1) why did he choose to display 2500 divs in the CSS rendering page? what signifigance does that hold and what would change if you were to alter the test?
2) What happens at other web JS testing sites such as this one http://www.world-direct.com/mozilla/dhtml/funo/jsTimeTest.htm
3) how much of an effect does the machine have? (that is, if I run a machine with 1 gig of ram and a 2.0 ghz cpu does it change which browser is faster?)
4) Why does he discredit the "feeling of a browser"? While I understand that many times it's familiarity that taints this notion of browser speed. I really feel that lots of times the numbers were talking about here are SO tiny that what it boils down to is a feeling of speed.
The relative closeness at which the browsers render these very specific tests (and the variability when changing variables, like machines, and distros, and OS's) makes us question not the validity of the results, but the usefulness. In then end it's not about the numbers it's about which feels fastest. Which browser when put to the daily rigamarol makes browsing easiest while continuing to display the pages are a decently quick rate.
Anders
Posted by: Anders on February 15, 2005 05:43 PMAlso notice how the optimized build comes out slower than the normal firefox, makes no sense, he must have messed up.
Posted by: Poningru on February 15, 2005 10:18 PMI must admit I would be interested to see someone replicate the expirement on a similar machine (or as close as can be achieved). There are deffinantly bits of his data that don't feel right to me. And I'm not saying he's lying, just that... it doesnt seem right.
Anders
Posted by: Anders on February 16, 2005 12:16 AMOr maybe optimized builds are not really helpful for real-use situations?
It's easy to discredit and ignore results when they are not to your liking, and pass around the URLs for those tests where your favorite browser shines.
In the end, it comes down to the feeling of fastness. Your subjective feeling can be even be influenced by reading such test results :)
The page nicely leaves the conclusions to the reader.
For a direct comparison of Opera beta and Firefox features and speed, see:
http://www.scss.com.au/family/andrew/opera/firefox/ (scroll down)
It's from an Opera fan, but tries to be as factual as possible.
I have made several posts about firefox and my personal opnions about some issues that have hit news. I posted a link to this article when it came out any my friend had coniption fits when posting comments because of what the site claimed.
Although many may aruge that the tests may not have been proper ways to test consumer usage, I think it is still a good example of speeds for the items he tested (I am captian ovbious)
I laughed that in his tests that Mozilla Suite came up faster than Firefox in most cases. I was really proud of that, beings as I am a Suite user.
PLZKEEPSUITEFASTBBQ
Posted by: larfnarf on February 16, 2005 01:40 AMMan, this Firefox 1.0.1 nightly sure is slow, I've been waiting for hours for my gmail page to load ;-)
http://forums.mozillazine.org/viewtopic.php?t=218095
Posted by: Phil Randal on February 16, 2005 03:04 AMWhat again the previous two posters failed to recognize is the reproducability of these results... Which is as far as I can tell... shakey at best. I don't think that these results can really be discussed as providing any clue into the nature of the browsers unless a big group of people all go out and preform the same tests on their machines and report the results. And to top that it would be great to see if they modified the tests what kinds of results are posted.
As I've mentioned before, on my machine, at least in the css rendering department, the results are nearly OPOSITE his own results. What does this mean to me? Not that he's wrong, or lied, but that his results aren't reproducable. Which simple goes to cast doubt in tossing them around like science.
Anders
Posted by: Anders on February 16, 2005 10:25 AMI have to admit that the results reported are inline with my experience. I use Firefox on a daily basis (and use it exclusively, save for Windows Update and Office Update) because I enjoy its features and extensibility, and I support web standards. I also use it because Gecko has a decent share of the market and is supported by my bank, supermarket, etc., whereas other browsers are not. I've never found the performance especially compelling: one only has to see a complex page not designed for Firefox (http://www.bungie.net/) or a large, simple page (http://www.lcdf.org/c++/clause14.html) to compare... try resizing the browser window on those sites.
On my mobile phone I use Opera, but I don't use it on the desktop because it's not free (beer) and has/had a disgusting and bloated interface, even though I was in awe of the zooming, rendering speed and history performance.
Even IE is faster than Firefox, but I'm tired of seeing bugs in web pages (and would need a shell with tabbed browsing and mouse gestures). I'm not a web designer, but Firefox just seems more consistent. Some things that happen in IE are very obviously not the page's fault (guillotine effects, etc).
If people have reservations about what's been reported, can they point to another series of tests that get different results?
Posted by: James on February 16, 2005 03:19 PM