Asa Dotzler: Firefox and more

Paul Thurrott, in his somewhat late review, takes a look at Firefox 2's new features. Unfortunately, and something I didn't expect from Paul, it's either the most intentionally misleading review of Firefox, or it's a completely unserious piece of writing. Which of those two, I can't say, but either way, it's a disservice to his readers.

Rather than go through the whole review and try to correct it all, something that could take far more time than I'm willing to give it, let's just look at Paul's analysis of the Firefox 2 Phishing Protection feature.

He starts his review of Firefox 2's Phishing Protection stating reasonably that "Unlike with IE 7, Firefox's phishing filter is enabled by default, which might seem like a good idea." We happen to think it is a great idea and, after lots of Firefox testing and user feedback, we determined that many of the people most in need of this feature were the least likely to be able understand why they needed it or how to enable it. We made sure that this default implementation didn't have any privacy issues and didn't require the user to take extra steps to learn what the feature does or how to make it work. This way, we could give Firefox users a safer browsing experience right out of the box. Microsoft's approach was different. They shipped with the feature mostly turned off by default and those IE users don't have a safer experience until they jump through several hoops to enable it.

We thought our approach was a better way to go. Well, not according to Paul. "But the Firefox Phishing Protection feature is a joke: It uses a blacklist of known dangerous sites, which isn't an effective protection against modern electronic attacks." It's not effective? All of our hard work designing, coding and testing this feature and our approach isn't effective? That sucks! Oh. Wait a minute. Why isn't it effective?

According to Paul, "What you need is something that can adapt to threats and update itself automatically." I get it now. Either blacklists in general, or our blacklist in particular, are incapable of adapting and updating automatically. I'll bet the team of designers, engineers, and testers who implemented our adapting and updating blacklist are scratching their heads at this point. How, one might ask, could they have failed to see that a blacklist of known dangerous sites can't be effective? Well, thankfully it turns out that local blacklists, without too much difficulty, can be updated regularly to adapt to the changing attacks.

Maybe Paul didn't have time to actually investigate (do journalists do that any more?) the feature before writing about it. Had he done a Google search for "firefox phishing protection", the first result would tell him that Firefox 2's Phishing Protection "works by checking the sites that you browse to against a list of known phishing sites. This list is automatically downloaded and regularly updated within Firefox 2 when the Phishing Protection feature is enabled." So, Firefox's Phishing protection feature does "adapt to threats and update itself automatically". Good to know, cuz we put a lot of effort into that part of the feature.

Next, Paul goes on to explain "Alternatively, Firefox lets you enable Google's phishing filter, which is more effective because it is updated regularly and provides more advanced functionality." Ahh, confusion again. So Paul's readers should apparently ignore the explanation Mozilla offers on its site and all of the testing that Mozilla did to demonstrate that the two methods are nearly identically effective. Or. I've got an idea. Maybe we should re-engineer the feature so it matches Paul's reporting and save him the trouble of actually learning how it works.

Paul wraps up his analysis of this feature with a paragraph that basically claims that Firefox's Phishing protection is not as effective as IE's anti-phishing feature. His "proof" comes in the next few sentences: "Incidentally, Mozilla initially responded to news that their solution was ineffective by publishing a paper on its Web site that seeks to prove that Firefox Phishing Protection is, in fact, more effective than IE 7's Phishing Filter. However, a third party study--commissioned by Microsoft--had already reported this not to be true. So who's right?"

That just seals it, doesn't it. It's definitive. Microsoft's "third party study" debunked Mozilla's paper.

What's not wrong with that paragraph. Paul either very carefully constructed it to be as misleading as possible, or he really hasn't done even the least bit of serious research for his article (or, I'll offer up, he got incredibly unlucky and accidentally managed write something that will be misunderstood by 100% of his readers.) Let's look at this paragraph in detail for a moment. It's a real gem.

First, Paul suggests that Firefox 2's Phishing Protection was deemed ineffective and there was "news" of that. But, he doesn't provide any links to that "news" -- because it doesn't exist. The only articles he would have been able to find that made any claims of efficacy at all, had he looked, were from alpha and beta releases of Firefox 2, long before the Phishing Protection feature was fully implemented. Other than Paul's article, there isn't a single news report available from Google search that calls Firefox 2's Phishing Protection ineffective. Not one.

Second, he claims that the report published by Mozilla was a response to his imagined (mis-remembered? mis-understood?) bad news about Firefox's feature. It wasn't a response to any news, good or bad. The report was a normal part of our quality assurance and testing process. We test all of our features extensively before shipping them and this was a part of that process. Given that we were commissioning the report, the first of its kind, and that its published results were likely to be scrutinized, we decided to have the testing methodology and results audited by an independent auditor. You can find the results of that independent audit in this PDF. This was not some PR response, as Paul would have readers believe. It was solid data about a major threat online and the top two browsers' responses to that threat.

Next, by labeling the Microsoft commissioned report as a "third party study" while failing to use the same description for Mozilla's report, Paul seeks to lend additional credibility to Microsoft's study while leading his readers believe that the Mozilla report shouldn't be taken as seriously -- it was probably just a PR exercise in response to bad press. The implication that Microsoft's report has more credibility than Mozilla's is completely disingenuous or horribly uninformed. Mozilla's report was the result of a very careful, independently audited, third-party conducted test and what it showed was that both modes of Firefox 2's Phishing Protection, the regularly updated local blacklist and the live remote checking, did significantly better than IE's most effective mode and completely destroyed IE's mostly useless local checking. Further, the report confirms that the two approaches available in Firefox 2 offered very similar levels of protection, completely contradicting Paul's initial assertion that a local blacklist is "ineffective".

An additional note on the "third party study" that Microsoft commissioned. I don't see any evidence that there was any independent audit of that study and without that, there's no definitive way to know with certainty that it wasn't simply a Microsoft press release in disguise. But that turns out not to matter, as you'll see.

Paul's fourth major error, or deliberate attempt to mislead, is his implication that Microsoft's oh so authoritative report debunked Mozilla's report. It didn't. It doesn't. It couldn't. Microsoft's report wasn't a response to the Mozilla report at all. The study he references was conducted months before Firefox 2 was even released and doesn't claim at all to cover Firefox 2's Phishing Protection. The report, available here[PDF], doesn't mention Firefox 2 even once. It is an evaluation of 8 anti-phishing products: IE 7 Beta, EarthLink's ScamBlocker Toolbar, eBay's AccountGauard Toolbar, GeoTrust's TrustWatch Toolbar, Google's SafeBrowsing Toolbar, McAfee's Site Advisor button, Netcraft's Toolbar, and AOL's Netscape 8.1. Where is Firefox 2's Phishing Protection in that list? Oh, it's not. That's surprising. How, then, can Microsoft's report, which doesn't even mention Firefox 2 or Firefox 2's Phishing Protection, have debunked a then unwritten Mozilla report on the efficacy of a then significantly incomplete Firefox feature?

Finally, Paul's last sentence "So, who's right?" set's up that fair and balanced excuse so popular these days. He's saying that we've got one Firefox paper and one IE paper and even if he considers the IE paper as more authoritative, there can be some room for disagreement and it's all just too complicated to figure out so why bother. But this is not some case of he-said-she-said. Not even close. Even a cursory reading of the two papers would make it completely obvious that Paul is making an apples to oranges comparison. And the only conclusions one could reasonably draw from the two reports are that some older anti-phishing toolbars are somewhat less effective than IE 7 beta's feature and that Firefox 2's Phishing Protection knocks the stuffing out of IE's anti-phishing feature.

OK. So let's back up and look at this last paragraph again, just for fun. Like I said, it's a gem. We have study that Microsoft paid for, with no independent auditing, that was conducted months before Firefox 2 shipped, and that doesn't test any version -- even a beta version, of Firefox 2's Phishing Protection, this report somehow disproves a third-party, independently audited test that directly compares the feature in shipping versions of Firefox 2 and IE7. Nice.

This was just one feature Paul Thurrott covered in his "review" of Firefox 2. Other than tossing his credibility out the window, what has Paul Thurrot accomplished with this article?