The Inside Track on Firefox Development.

« Safari, KHTML, Perfection | Main | 1.0.4 »

May 13, 2005

Some More on Perfection

I realize the brevity with which I justified my "cutting corners" statement in my previous entry would lead some people to freak out. Let me recap:

The Mozilla team takes security, dataloss and topcrash bugs very seriously.

To ship software on time (where on time equals this month, this year, this decade), a set of bugs (usually features, or bugs that are harder to reproduce) fall off the boat. This is hard. Some of the bugs you may personally care a lot for and really want to fix. The original Firefox 1.0 plan called for a comprehensive rework of the somewhat crusty Bookmarks manager, but in the end we decided shipping in 2004 was more important, and we should save ourselves something for 2.0.

I understand this conflicts with some folks' dogma, but this is the way product releases work. This is OSS, but it's OSS that we intend to have high impact, so sitting around for years polishing trying to reach some kind of unattainable "perfection" (see last post) serves no one.

Posted by ben at May 13, 2005 11:24 AM

Comments

Curious if you have a set criteria for making the decision, or if it's somewhat more arbitrary.

Posted by: Robert Accettura at May 13, 2005 12:02 PM

What people dont realize is that this criterion for release (perfection/release time) is only a big deal if the software is going out to the linux world primarily. In the windows world people expect things not to work the first time you command a software to do something, in linux (I dont know about mac) if you enter a command and it doesnt work its usually still considered beta (unless its user fault ofcourse). Now firefox certainly was more than ready when deployed even under the linux standards.

Posted by: poningru at May 13, 2005 12:29 PM

.. or, to reduce the workload, just don't build half the software. Email, Composer, just remove them all and watch the project simplify itself dramatically. Hey, maybe at Work we can do the same thing, and just not include any X libs with unix.

Hey, in another year, you should reduce that even further, and peel off the UI so you can work on the engine without distraction. Fast forward 4-5 years, simplifying each year, and the entire project will be incredibly easy because it will have been pared down to a single line of code.

Which is commented out.

Posted by: Bish at May 13, 2005 1:23 PM

Many years ago, some Python folks had a module that they wanted to open up. "But we need to polish it first". I said, "Just release it. If you wait to polish it, then it will never see the light of day. Even if it isn't in good shape, there will invariably be *somebody* who can use it, learn from it, or fix it up."

The same situation applies here: get the thing into people's hands. If you sit around "cleaning it up", then it is useless since nobody has it in their hands to *use*.

Posted by: Greg Stein at May 13, 2005 1:31 PM

First of all, Mac people damn well do expect software to work: first time, next time, every time. Fatalistic acceptance of disfunctional software is for other folks. Windows masochists, software testers -- that kind of riff-raff.

However, in the real world, it's simply not possible to squash every bug. It's nearly impossible to finish all the features originally spec'd for a major project.

Given these messy intrusions of real life into the development process, the trick to offending as few people as possible is to do periodic triage on the feature set and the bug list.

I worked on the Mac Office team at Microsoft for a time; what follows is a greatly simplified outline of how they go about the triage process. Any omissions or errors in the description are mine alone and should not require the presence of a mob of angry villagers:

Milestone 1 - Spec Complete

The simplest way to set the scope of the project is to ruthlessly weed out features at the end of the spec phase, before serious coding starts. Ideas that just aren't gelled yet can get rescheduled for a later release; ideas that started smoking when exposed to sunlight can be quietly staked through the heart.

By this time, the vision of what you intend to ship should be solid. Repeat - "should be".

Milestone 2 - Course Corrections

You're half way through the project. Serious coding work has been done, the testers are starting to find the facehugger nests, and it's time to revise your game plan to match. Check your schedule against the available resources. Did you lose Sergeant Apone to the Alien queen in that last tunnel? Who's got the ammo bag? How do we get out of this chickenshit outfit?

In other words, make your best estimate of the time to get to Code Complete, step up the pace on bug discovery, and axe any features that are in sufficiently bad shape that they jeopardize the release schedule.

Milestone 3 - Code Complete

All elements of all features that have survived to this point are in place and allegedly working. It's Bug Hunt time.

There's usually no question of feature triage at this point. All efforts go to finding, prioritizing and fixing bugs. The primary focus for the leads and PMs is to keep an eagle eye on the find/fix rate for their feature(s), and to get the high-priority bugs swatted hard and fast.

BTW, the MacBU uses a four-level priority system.

Pri 1: crashers or any bug that has a serious global impact on multiple features
Pri 2: 'Must' bugs; high-visibility, high-impact bugs that essentially or actually render some part of the feature unusable.
Pri 3: bugs that are cosmetic or minor in nature and which do not seriously impede the user. Also reserved for bugs that will only affect a very small number of customers.

Pri 4

(There will be a test on this list at the end of this post.)

Milestone 4 - Major Triage & Zero Bug Bounce

As the project heads into the end game, it's time to decide which unfixed bugs will actually get fixed before the ship date. The PMs, assisted by Test and Dev, start postponing, closing or re-prioritizing bugs. Meanwhile, the standard for posting bugs goes up, to shut off the noise-level Pri 3s and 4s. At first, the triage is weekly, then daily, then twice a day. At some point, the feature's bug list hits what's called Zero Bug Bounce (AKA 'ZBB'). Zero active bugs and zero stale bugs at the end of the day for three or four days. That's usually the sign that the feature is shippable.

Release To Manufacturing

When all features have hit ZBB and all PMs and testers for all features have signed off, it's time to hand off to manufacturing. Party time!

So -- that's one way to do triage. I'd be interested to see more detail on how Ben's team does it.

Posted by: Will Parker at May 13, 2005 2:25 PM

Will, I wish software generally (and Mozilla specifically) worked like what you described...

What I have seen over and over again with Mozilla are new "this is so cool!" features being added after Milestone 1 (either in the "course corrections" phase or sometimes even after Milestone 3). Part of the problem is that "Mozilla" has a number of separate modules, being developed on somewhat divergent schedules, with far too little communication and cooperation going on. Part of the problem, from my vantage point, is just lack of discipline. The new features are just _so_ cool and all.

Posted by: Boris at May 13, 2005 2:41 PM

I'm very glad to read this. When I read your earlier entry as posted on zdnet (and they of course take everything out of context) I got a little scared that you were favoring doing things wrong to get them done quicker.
Obviously nothing is perfect, but there is a pretty visible line between right and wrong (even though the complete right is unachievable, save "Hello World").

Posted by: Chris at May 13, 2005 2:55 PM

Boris:

I'll comment on your post in greater detail later tonight. For now, just one thing - adding features after the specs are deemed complete is pretty much forbidden in the MacBU. After the specs are complete, "This is so cool!" is always met with "How many dollars cool is that?".

Sure, feature elements can be modified and improved during the current cycle, but whole new feature ideas? That's a Pri 4: Let's think about doing it next time.

I recommend that you get religious about the Pri 4 idea on your current projects.

Posted by: Will Parker at May 13, 2005 2:55 PM

Where did you get the idea that khtml's devs were "sitting around for years" ? They release in sync with KDE, which means a shorter time between 2 major versions than Firefox. How could you judge them when you don't seem to be really aware of their work ?

I am quite astonished to see you reacting like a /. commenter.

Posted by: JRM at May 13, 2005 3:05 PM

Face it. No matter what you say, if it's remotely connected to another web browser, and isn't 100% positive, it'll be interpreted as a vicious personal attack. Asa has the same problem every time he mentions Opera.

People see what they want to see.

Posted by: Kelson at May 13, 2005 4:28 PM

Will, I wish I could. Unfortunately, I'm not running my current projects.

Posted by: Boris at May 13, 2005 10:29 PM

Boris wrote: "Unfortunately, I'm not running my current projects."

If you're completing tasks of any sort, you're running projects. You're just not getting an equal voice in how the projects are supposed to be run.

Agitate. Advocate. Subvert. Expropriate.

Route around the damage.

Posted by: Will Parker at May 13, 2005 11:51 PM

Absolute nonsense! We give Microsoft a hard time because they cut corners. Are we forgetting why we rain insults on IE?

The KDE, KHTML team, have very high standards with regards to code cleanliness, readability, structure, maintainability and documentation. Heck, there is a reason Apple chose KHTML over Gecko.

Now, one of the most respected open source hackers, is implying KDE developers should accept undocumented large globs of unreadable patches that introduce more bugs and breaks KHTML in several nasty places, and, *GASP*, cut corners. Why? Well to meet Apple's deadlines and please Apple's users.

The software industry is going down the drain because people blatantly advocate cutting corners. Instead of praising the KDE developers for upholding high software engineering principles, standards and ethics, one of the most respected open source hackers, decides to return us the the dark ages of software engineering, cutting corners.

Yes, that's right, the reason you don't hear of many KHTML security vulnerabilities is because the KDE developers decided not to cut corners. If Joe Coding Monkey, from Microsoft said it is okay to cut corners, he'll be forgiving, after all it is Microsoft. But when a respected open source hacker publicly advocates such abomination, one only wonders what message he is propagating the next generation of open source hackers out there.

The KHTML developers deserve better treatment than such insults. They are doing the rarely done *right* thing, their code has been exploited beyond belief and all they get for their contribution is "KHTML sucks, Webcore Rocks! you guys are LUSERS, haha!"

Your rebuttal reminds me of why software engineering is a joke among engineering disciplines. Go ahead, cut corners while building that sky scrapper, or bridge, just to meet your deadlines or to please your customers.

Posted by: Mystilleef at May 14, 2005 1:52 AM

In general, other engineering disciplines differ from software engineering in many more respects than just not cutting corners. Things like sharing your findings and practices. Okay , open source does that, somewhat, although one might wonder how much �the code is public, find it out yourself� can be called sharing. Or like not reinventing the wheel, something software development likes do do a lot, not in the slightest because of the miriad of different incompatible licenses out there.

If you advocate engineering-quality standards in software development, do it properly then :).

In general, although engineering-quality development methods are nice, they don�t apply as much to software engineering because software usually operates in an environment where failure is not dangerous to human lives. On the other hand, development methods like that can be very ardious and cause long delays, something which is usually undesirable when developing software and can make or break a product in the software world.

I�m not saying that software development can�t learn from other engineering disciplines, but I am saying that it isn�t as easy as just stating that they are doing it better and we should follow the same practices.

By the way, with all the talk about KDE being better... Why isn�t there a Windows version of it then? I�d love to try it out, at the least to test whether my webpages look as they�re supposed to, but...

~Grauw

Posted by: Laurens Holst at May 14, 2005 4:30 AM

A nice parallel: the reasons why engineering practices don�t apply well to software development are in a way quite similar to the reasons why patents don�t apply well to software.

You can�t apply the same methods to things which on the surface seem to have much in common, but in practice differ on very important aspects, like the development cost model, or how critical failure is.

~Grauw

Posted by: Laurens Holst at May 14, 2005 4:35 AM

Instead of praising the KDE developers for upholding high software engineering principles, standards and ethics, one of the most respected open source hackers, decides to return us the the dark ages of software engineering, cutting corners.

While I understand your sentiment, why hasn't it borne fruit? By all accounts, KHTML is a well-written rendering engine that does not perform nearly as well as WebCore and is not nearly as good at standards compliance.

So while your software development principles are sound, where's the benefit to me as a user? All that readability, maintainability, etc has to turn into top-performing product for me as a user to see any benefit.

There's no way I'm going to choose KHTML if it underperforms WebCore just because I like the value statements of the developers.

Posted by: Wade Williams at May 14, 2005 7:29 AM

There's no way I'm going to choose KHTML if it underperforms WebCore just because I like the value statements of the developers.

Apple has a whole team of fulltime employees working on their KHTML enhancements. The KHTML team work on KHTML in their spare time. I don't think there is one single KHTML developer that works on it fulltime.

Finally, KHTML as it is today is a damn good rendering engine. I don't know where you get your "under-performance" data from, or who told you KHTML isn't standard complaint. Statements like that is what pisses of the KHTML developers.

Note that you won't be half as happy as you are today, if KHTML sucked, or was an unmaintainable pile poultry droppings. You owe a lot of gratitude to the KHTML developers as a satisfied Safari user. Safari rocks because it is founded on solid and clean code, KHTML.

Posted by: Mystilleef at May 14, 2005 8:29 AM

Let's not kid ourselves that other engineering disciplines are perfect either. One of the things that they teach you in engineering school (I went - took a range of classes from civil, mechanical, chemical and resource before settling on EEE, did you?) is that there are a lot of tradeoffs you make when you build something. You always put people's safety and security first, or you don't finish your project, but beyond that you find the best compromise of customer satisfying content vs time/cost/etc. If you doubt this, go look at cars. Sit in a few and look deeply and see where they've cut corners. Oh no, they've used cheaper plastic on the parts you don't touch so much. Shock horror!

Posted by: Ben at May 14, 2005 11:20 AM

If you're not shipping software, you're spending an awful lot of training and intellect on a useless activity. No piece of software can be perfect, so the question of when to ship must come down to what compromises you're willing to accept in order to ship.

I'll accept any process that provides additional functionality for the user with minimal risk of doing harm to the user. I also want a process that delivers "packets" of additional functionality to the user with reasonable frequency.

Releasing on a set schedule forces the programming and design teams to make hard decisions and address problems.

Can we actually do this? Is it meeting the quality imperatives on which we've agreed? Do we need to revise our architecture? What's the hold-up? Can we ship a partial solution without screwing the user and breaking her currently-installed software.

An earlier commenter said "Safari rocks because it is founded on solid and clean [KHTML] code". I have to ask: why then is it that KHTML currently fails the Acid2 test? Clearly, there were some problems with the code. Indeed, the KHTML community would not be yelling so loudly about Apple's delivery methods for improvements to the KHTML code if those improvements did not address serious problems with KHTML rendering.

Using the words 'solid' and 'clean' to describe any piece of software implies a level of invulnerability that is simply impossible to achieve in the real world. Programmers, whether individually or in teams, must make an informed GUESS when they decide to release code. Is it good enough? Sometimes, the assumptions on which they base that guess are simply wrong.

At least with a set ship date and frequent public triages, developers and designers are forced to re-examine their assumptions.

Posted by: Will Parker at May 14, 2005 12:33 PM

Ben, your mistake here was touching the third rail of open source politics, KDE.

See Ben, it doesn't make a difference whether you were actually making a nuanced point about the inevitable imperfection of software. It doesn't matter that you really weren't bashing anyone else, and that your blog entry was actually very tame. Hell, it doesn't even matter that this particular blog entry doesn't mention KDE or KHTML or any "K" product at all, and reads like an obvious attempt to steer the discussion back towards sanity. And it certainly doesn't matter that everyone but the KHTML team sees that (with a few concessions from Hyatt and Apple) simply adopting WebCore might be the best option (hint: Nokia has done most of the work for you).

It doesn't matter: the electrocution will continue. Can't wait to see it continue in the comments section of your next blog entry, about how spinach is in season.

Touch that rail once, you pay for life.

Posted by: Nobody at May 14, 2005 2:18 PM

*sigh*

This whole KHTML/Apple thing has left me pretty depressed, because the whole discussion/argument has been blown up so that it hardly relates to the original disagreement. Note before I continue that although I develop for KDE, I have very little knowledge of KHTML, except that it was relatively easy to add to the KDE application I use to post on my blog.

The whole *point* with Zack Rusin's initial rant is that WebCore is, by now, a de facto fork of KHTML, which has diverged too far for WebCore patches to be generally applicable to KHTML. As such, when neato patches come out for Safari, people shouldn't hold their breath waiting for them to get merged into KHTML.

Everything else that's been said about this is just fluff that reflects back onto that statement. It's fluff that explains why there was a fork, or how people feel about the fork, or what people would have liked. In the end, however, the KHTML devs weren't pissing and moaning so much about Apple as they were about everyone who said (and keeps saying) that applying such-and-such a patch should be easy.

In trying to explain the fork, KHTML devs pointed to several things:
*) Lack of developers for KHTML to vet and convert patches to apply to KHTML.
*) Lack of context information about how such and such a diff actually fixes a bug. Many fixes were sent to KHTML with no more description than "fix 23498", where 23498 referred to an internal Apple bug db that KHTML devs can't access.
*) Diffs that were otherwise difficult to piece apart. I believe the canonical examples are the megabyte-sized diffs that the KHTML devs have been sent.

What I *don't* remember seeing is KHTML devs claiming that Apple was breaking the license, or that Apple should be breaking every diff down into a code-by-numbers affair while David Hyatt flossed their teeth. No, the reason the KHTML devs were putting that out was because trolls on Slashdot and other places were trying to claim that the KHTML devs were just sitting on their asses for two years. That's not true, but it doesn't matter, because whether we like it or not that fact is that the codebases have diverged.

Now Apple has proposed 'fixing' the problem by replacing KHTML with WebCore. And people seriously expect that this is going to happen? I mean, picture this: Say IBM forks Gecko for some l33t browser, and the Gecko fork eventually drifts so much that IBM fixes no longer apply. Would Mozilla adopt IBM's Gecko fork just because it's developed faster, and lose control of their HTML rendering engine?

KDE could adopt WebCore as an alternate rendering engine, but KHTML won't be going anywhere. And I'm glad that's the case. KHTML has by now become a top-flight HTML engine. I'll be the first to admit that it doesn't support some features that Firefox does, which is why I have Firefox installed as my backup browser. (I even donated to the NY Times ad. :P). But, where KHTML does work, it works better, and faster, than Gecko (at least on my system). KHTML even has better support for some W3C standards than Gecko (my favorite being text-shadow). In addition the text rendering is better for some reason.

I'm *not* trying to rail against Gecko here. What I am trying to do is point out that KHTML isn't the hopelessly outdated, useless piece of crap that some people keep claiming it is.

Ben: As to the original comment, about "cutting corners"... I actually agree with you. Your criticism is misguided though. KDE *has* to cut corners sometimes just to make feature and release freezes, and this applies just as much to KHTML.

But would you apply a patch to Firefox from $FOO_CORP that neither you nor any of your fellow devs understood? Or a patch that broke some piece of software used in the rest of Mozilla but not by $FOO_CORP's particular product?

I mean hell, Firefox went through a release candidate for even a rather severe security vulnerability, so you surely know the use of testing changes rather than just applying patches all willy-nilly. ;)

Posted by: Michael Pyne at May 15, 2005 12:35 AM

Now Apple has proposed 'fixing' the problem by replacing KHTML with WebCore. And people seriously expect that this is going to happen?

Unfortunately, you're right, nobody seriously expect the KHTML developers to be wise enough to give this the proper consideration it deserves.

I mean, picture this: Say IBM forks Gecko for some l33t browser, and the Gecko fork eventually drifts so much that IBM fixes no longer apply. Would Mozilla adopt IBM's Gecko fork just because it's developed faster, and lose control of their HTML rendering engine?

In all fairness, you're probably right: the Gecko people would probably make the same stupid mistake. Call it hubris, call it NIH syndrome, call it sour grapes, call it the inability to take a step back from one's ego and see the big picture... I'll just call it stupidity, because the option is being dismissed without any serious consideration.

"Losing control" is simply not a serious factor. If the license is open source, KDE (or Mozilla) would always have the right to fork (re-fork? un-fork? fork-back?) if things don't go as they like. And they'd be right back where they started, if not farther along. Let's be honest: losing face, not control, is what they're worried about.

Sure, the company involved would need to be willing to make certain concessions. Maybe Apple isn't prepared to make those, maybe they are. (That recent leaked email suggests they are.) But I doubt we'll ever find out - this common-sense path forward has been ruled out without the slightest bit of honest consideration.

But we've veered quite far from Ben's post. Sorry to contribute to the noise, I'll move on...

Posted by: nobody at May 15, 2005 8:33 PM

Open source needs to think three times before criticizing companies that use open source and abide by the license. Forking is not unlikely, at some point, and if that means a storm of bad publicity, corporations will think twice before going the open source route, feeling they'll be locked in, and everyone will lose.

As I understand it, KDE can't claim at this point that their choice has led to a superior browser, only try to lay the blame for the difference at another door. Maybe Apple could have been more helpful. But we shouldn't be shocked when corporations decide a deals a deal, or punish them too quickly for situations that took two parties decisions to create. It's not professional.

Posted by: Russell Johnston at May 15, 2005 10:12 PM

In reply to Russell Johnston:
Open source needs to think three times before criticizing companies that use open source and abide by the license. Forking is not unlikely, at some point, and if that means a storm of bad publicity, corporations will think twice before going the open source route, feeling they'll be locked in, and everyone will lose.

Well, I hardly see how a company would fear getting "locked into" something just by using code. That hasn't stopped the commercial use of other forms of open source software, why should it stop WebCore?

As I understand it, KDE can't claim at this point that their choice has led to a superior browser, only try to lay the blame for the difference at another door.

I'm not sure what you're talking about here. Do you mean KDE's choice regarding ditching KHTML for WebCore? Or the sequence of decisions leading up to the fork?

I'm assuming you're talking about the events up to the fork. One thing I'd like to point out is that we're not really trying to discuss which browser is more superior. Everyone knows that Gecko and Safari work with more websites. They also both have developers behind them paid full time to work on them.

KHTML has no corporate daddy, and no KoFo devoted to it. ;) And before you start going on about Apple, please realize that there is nothing we'd love more than to have other people doing our work for us. Trust me, if we had had the ability to merge WebCore patches on the tree at anywhere near the rate they came in, we would have.

Please re-read the original rant that started all of this: http://www.kdedevelopers.org/node/view/1001

Note that zrusin wasn't trying to rail against Apple. He even admitted that everything that Apple has done has been well within their rights.

Also, no one's trying to claim that WebCore shouldn't be using OS X-specific features. What zrusin's point was that such changes are obviously not going to apply to KDE. They need to be either ported or rewritten. This requires developer time. KHTML has 0 paid developers. Perhaps you can start to see why KHTML would diverge from WebCore...

But we shouldn't be shocked when corporations decide a deals a deal, or punish them too quickly for situations that took two parties decisions to create. It's not professional.

No, it's not professional. And indeed, David Hyatt and the rest of the Safari team have been holding talks with the KHTML devs. The 'punishment' that's occurring is coming from elsewhere, generally from people who read a blurb and immediately decide to make a comment. Apparently few people know the full story. I guess I shouldn't be surprised, because the full story is decidedly more boring than the Slashdot and blogosphere stories...

Posted by: Michael Pyne at May 15, 2005 11:36 PM

Posted by: ruiski at June 1, 2005 12:52 PM

--------

acheter logiciel
logiciel 3d
gratuit logiciel
logiciel caisse
logiciel cao
logiciel comptabilite
logiciel anglais
logiciel dvd
logiciel educatif
logiciel francais
logiciel ftp
logiciel graphisme
logiciel jeu
logiciel mp3
msn logiciel
logiciel graver
logiciel son
logiciel traduction
logiciel windows
logiciel montage video

Posted by: crebuler at June 17, 2005 5:36 PM

©1997-2006 Ben Goodger. All Rights Reserved.
Opinions expressed here are my own, and not those of any organization that I may be affiliated with.
Reload icon is © Stephen Horlander;
Firefox logo is by Jon Hicks, and is a trademark of The Mozilla Foundation.
GetFirefox buttons are from rakaz