Asa Dotzler: Firefox and more

First, I want to apologize to you all and to Jesse for taking so long with this. I was slow to gather up all of your questions a slow to get it published. I think, however, that it was well worth the wait. Here is your interview with Mozilla's Jesse Ruderman.

“How did you get involved with Mozilla?”

When I was in high school, I read Slashdot and was a fan of the open-source movement. Slashdot ran an article pointing out that having a good web browser was a crucial part of the effort to make Linux succeed on the desktop. I realized that I could contribute to the Mozilla project even though I used Windows, so I downloaded M13 and started reporting bugs. Soon I was helping to triage bugs and make reduced testcases for broken sites. People like Eli Goldberg and Asa Dotzler really made me feel appreciated early on, which fueled my enthusiasm for contributing.

Since then, Linux hasn't really succeeded on the desktop. Mozilla's effort to keep the Web open may have kept Linux from being completely marginalized, but the Mac may have benefited more.

“Was security research something you were interested in before getting involved with Mozilla?”

When I was in high school, I spent a lot of time on IRC and found a few holes in both clients and servers. I even exploited a few of the bugs I found. But around the time I started college, I realized that reporting bugs felt a lot better than disrupting IRC conversations.

During my freshman year of college, Mitch Stoltz asked me to do a summer internship at Netscape working on security. I was surprised; he pointed out that of the hundreds of bug reports I had made, the security bug reports were especially valuable to the project.

“How do you feel about full disclosure?”

I don't like to put pressure on people to fix bugs quickly. I know that demanding a fix within a week or a month is likely to take time away from other worthwhile activities. Some of these other activities are related to security: for example, addressing other security holes, or making sure the patch doesn't cause other problems (so that users will trust future security updates).

In my experience, if a software maker agrees that what I reported is a security bug, it usually gets fixed within three months even if I don't threaten to disclose or prod for updates. I only used the threat of full disclosure once, against the Mozilla project, when I had already waited for a fix for over a year. Luckily, I haven't felt the need to do so again since I became a Mozilla employee ;)

Some people who disclose early are hungry for publicity, angry at a specific software company, or just impatient. Others have better reasons. Some do it when they think it's likely that black hats will also discover the bug, so they want to the vendor to rush a fix to protect users who may already be under attack. Others do it because they see vendors putting less effort into security than they should. The costs of spam and fraud due to operating system flaws are borne by users, so profit-driven companies may have the wrong incentives when it comes to security. A security researcher might force a vendor to fix a specific bug quickly because he thinks the bug is more important than the vendor does, or because he wants to punish vendors that don't put enough effort into security when writing new software.

“Are you working [at Mozilla] full time? What programming languages do you know?”

I've been working at Mozilla full time since 2005. I use C++, JavaScript, and Python regularly. My favorite programming language is SML, but sadly I have not had a chance to use it since I graduated from college.

“Which one do you say "helped more/have more share" in finding security bugs: 1) people/researchers 2) automatic tools?”

People and tools find different kinds of bugs, so it's hard to make the comparison. Most memory safety bugs in Firefox are found by fuzz-testing tools, while other types of bugs are mostly found by people.

Gecko is far too large and complex for me to review at the code level, and the attack surface for memory safety bugs is every line of code involved in the use of pointers. A fuzz-testing tool can test large parts of code fairly well with little human input, and it's easy to tell when a fuzz-testing tool has found a bug -- Firefox crashes in a way that looks exploitable.

Most other security bugs, such as those that involve user interface issues or JavaScript privileges, are found by people. Finding these bugs requires guessing what a developer might not have thought of, or how components might interact, or how a browser's user interface might be subverted. It would be very difficult to design a tool that understood these things, or even knew when it had found a security hole. But the attack surface is limited to parts of the software that deal with permissions or privileged data in some way, so it's possible for a human to reason about the issues.

When I fuzz, I find more bugs and they tend to be more severe. But when I look for other security bugs, I'm more likely to find the same bug that someone else finds, which might impact users' day-to-day security more (especially if that other person is a black hat).

“[Do you have] any interesting story about security and automated tools? e.g. a problem that is found both by someone and a tool at the same time, etc.?”

I found an inconsistent layout bug while looking at some output from one of my tools. Uri Bernstein commented in the bug, saying that it was likely to affect real web sites. A few days later, someone filed a bug about jiggling text on Wikipedia, and we considered delaying Firefox 3 Beta 5 to make sure we'd get enough testing on RTL sites. The two bugs turned out to be the same issue -- and both were a regression from a third bug, which I had found with another tool. We ended up not fixing the bug for Beta 5, causing it to collect five duplicates, but we did fix it for RC1.

Another example is the bug that forced Firefox 3 RC2, a crash on motortrend.com. Martijn Wargers had found the bug months earlier through fuzzing, but at the time, it was assumed to not affect web sites.

“What is the hardest part about securing a software product?”

Complexity, combined with bad programming languages.

The Web's layout model and scripting language are complex and demanding of high performance. Its security model is complex and under-specified, partly because the security model evolved over time against the constraints of compatibility and the discovery of new attacks.

C++ makes performance tricks possible, but it increases complexity by being unexpressive, forcing programmers to wrap fundamental complexity in additional unnecessary complexity. And it makes bugs (which arise more often due to complexity) worse by encouraging the unsafe use of pointers, turning many logic bugs into memory safety bugs. Having better "safe" programming languages would help to the extent that it could be used -- for example, it might be possible to use it for layout code but not for a JavaScript engine.

My ideal solution is allowing programmers to prove things about their code. The first step is to declare your invariants, which act as suggestions for lemmas. If a portion of your code implements garbage collection using raw pointers, you should be able to and work with the compiler and an automated theorem prover to show that your use of pointers is safe. If a portion of your code deals with privileges, you might want to prove that it is correct in some way. And if a portion of your code needs to run faster, you might be able to help the compiler optimize it by merely adding assertions and invariants!

Some people think static analysis is hopeless, citing the the undecidability of the halting problem. Maybe it is hopeless for existing C++ programs. But I agree with maxwell demon: Undecidability doesn't make verification of software obsolete any more than Gödel's incompleteness theorem makes proofs in mathematics obsolete.

“In terms of building software, how much is security a matter of a set of best practices (e.g. validating input) versus the need to consider it at an architectural/design level?”

I think Mozilla tends to get the easy stuff, like avoiding buffer overflows and validating/escaping input, right. But we don't do very well at "Don't make your data structure invariants so complex that you cannot keep track of them."

“If you used a Mac, would you run anti-virus software?”

I use a Mac and I don't run anti-virus software. But I didn't use anti-virus software even when I used Windows, because I'm careful with what I download and I don't trust anti-virus software to be effective. It's not worth the price or the system-wide slowdown to protect against a small percentage of attacks.

“Have you been doing anything interesting with in-browser page modification (bookmarklets/greasemonkey) recently? Know anybody who is?”

I've been writing scripts for Mozilla developers, such as TidyBox. But I haven't done much general-purpose page modification lately.

“Do you believe in the 'Many Eyes' theory of security for open source, or is overstated? Are people outside the mozilla organization actively looking and helping find security vulnerabilities in the source?”

I think the theory is a bit overstated. Having the source code available doesn't cause all the code to be looked at. The most complicated and poorly-understood code, which is the most likely to have serious memory safety bugs, probably gets looked at the least. I've found ancient bugs in the JavaScript engine, for example. (The second one was a security hole.)

What source code availability does do is allow people who want to look at the code to do so. Academic security researchers love it because it's a real-world software project they can play with without getting special permission beforehand. Companies making static analysis tools love it for the same reason, and because it can help them promote their products. If someone notices glitchy behavior while looking for bugs, she can examine the code and determine whether it's a security hole before reporting it.

Having the source code not only viewable but also reusable also gets more people looking at the code -- not because they're looking for security holes, but because they're building software with it.

Being open-source helps, but not enough to make "all bugs shallow".

“I rarely hear about people getting spyware while running Firefox, but I still hear about it with IE. If the market share of Firefox and IE where reversed and Firefox was the [dominant] browser, do you still believe spyware installation on/through Firefox would still be scarce? Why or why not?”

Firefox has some features that will continue to protect users. Firefox 3 can block known malware distribution sites and block versions of Flash and Java that are known to be vulnerable. Firefox's automatic update system uses clever compression techniques to ensure that even dial-up users get the new version quickly.

So far, Firefox's window of exposure has been much smaller than IE's. We do better on both components of window of exposure: the number of bugs that are made public before they are fixed, and the length of time each one lasts until it is fixed.

I think we're able to do better on patch speed in part because we have a lot of nightly testers. Our open bug-tracking system also helps, by encouraging nightly testers to report bugs they find, which lets us find out quickly if a security fix causes problems.

It's harder to know why we have fewer security holes disclosed before they are fixed. It could because (1) Firefox actually has fewer holes, (2) fewer people are trying to find holes in Firefox, (3) it's harder to find the holes in Firefox despite the code being visible, or (4) people who find the holes are more friendly to us. I only have insight into the fourth item.

First, Mozilla's fast patch time may also test researchers' patience less. Bug reporters who are initially friendly to both Mozilla and Microsoft may become unfriendly toward Microsoft simply because Microsoft takes too long to fix the bug.

I've heard from several researchers that they are angry at Microsoft for delaying a fix indefinitely, blowing off a bug as not being a security hole, or releasing a patch that fixes one exploit but not others. As a result, they no longer give Microsoft the benefit of the doubt. There is pretty strong evidence that Microsoft did things like this in 2002 and 2004, but I don't know if it still happens. Mozilla isn't perfect, of course, but these problems seem to occur less frequently for us.

There are also reasons beyond reputation that researchers are friendlier toward Mozilla. Again, Mozilla's open bug-tracking system helps. (Bugs marked as security holes can't be viewed by the general public, but they can be viewed by the reporter.) Bug reporters can see very quickly that we agree that the bug is a security hole, then that we are discussing a fix, then that we have a patch and we're waiting for a release vehicle. This is much more comforting than only finding out when the bug is fixed. People who report security bugs in Firefox can even contribute beyond the initial bug report, such as by explaining why a proposed fix wouldn't really work, or by posting a patch.

Note that it's possible to have open-source code without an open bug-tracking system, or vice versa. For example, until recently, security holes in WebKit had to be reported through Apple's Radar, which would result in little feedback until the patched version of Safari was released.

“How's the collaboration with other browser manufacturers on fuzzing (and other security related) tools going?”

Other browser vendors appreciate jsfunfuzz, but I haven't been able to share my other fuzzers yet. I'm hoping that other browser makers will follow my example and share the fuzzers they write, ignoring competitive advantage in favor of improving the overall health of the Web.

“What sort of things are you working on these days now that most of the fuzzers you wrote have been used enough to discover most of the easy bugs to find? More fuzzers? :)”

I still spend a lot of time reporting bugs found by my fuzzers and other automated testing tools. At this point, most of the bugs I find are regressions.

A few months ago, Mozilla Messaging intern Gary Kwong asked me to show him how to use my fuzzers. Teaching him has helped me find many opportunities to make the fuzzers easier to use. He has also pointed out some areas that I had been neglecting, so my fuzzers are also more effective at finding bugs than they were before he joined the project.

I'm still doing things that give me ideas for what to fuzz and what to poke at, such as triaging bug reports and writing Burning Edge updates.

“What's your favorite Firefox feature or add-on?”

Greasemonkey. I love being able to fix broken web pages, and Greasemonkey makes it easy.

I want to thank all of you for your great questions and especially thank Jesse for taking the time to give such informative answers.

Stay tuned for the next installment of Ask A Developer!