firefox thread at top weblog || MAIN || a fox in bill's henhouse

January 15, 2005

making releases happen - answers

I promised I'd get to all of your questions from my making releases happen post and I've finally got a few minutes free to take a stab at it.

Some of the questions overlap and some would take a lot more time to answer than I've got for this post but I've done my best to put together replies to each of the questions. I'm put my responses together in the format of direct replies to each of the people who had questions. Maybe later I can distill this into a more condensed post about the release processes at Mozilla.

This took a good bit longer than I expected :-) (hours, not minutes) so I haven't done any serious proof reading; please excuse grammar or spelling issues. Also, this is just one man's opinion and if I've got something wrong here or misrepresented anything, pin that on me and not on drivers@mozilla.org or other members of the community.

Peter van der Woulde was the first to post questions, so he gets the first set of answers.

  1. How do you decide what to plus and what to minus ?

    This is probably one of the most frequently asked questions and our process isn't based on a set in stone list of rules. As I'm sure you know, there are two types of flags we evaluate and set. The first is the "blocking" bug flag and the second is the "approval" patch flag. The blocking flag is for bugs which should stop the shipping of that particular release and the approval flag is for patches which should land during the freeze for that release.

    It's probably also important to realize that there are about a dozen of us on drivers@mozilla.org and we all come with different expertise. I am particularly focused on quality and how our testing and user communities will respond to each of the product releases. Others have a focus on things like Web standards, security, freezing and maintaining APIs, or the Mozilla application platform. We all approach the plussing and minusing of bugs and patches with a somewhat different set of criteria.

    There are a lot of factors that come in to play, but I think over the years all of drivers have come to settle on some basic tenants.

    First, our criteria for alpha, beta, and final cycle changes are all different. We're going to have a different approach to determining whether or not a bug should block an alpha release than we would for a final release. We're also going to be a bit more relaxed about the kinds of patches we approve to land during an alpha freeze than we would for a final release freeze.

    Alpha cycles can absorb more risk than beta and final cycles so we're likely to try to use drivers@mozilla.org's influence to get high risk patches (including features, major code rewrites or reorganizations) landed in the alpha cycles. I'd certainly rather hold an alpha release for a day or two in order to get the added testing on a major new feature before beta and I'd be willing to approve the landing of that change during the alpha freeze when I might push out lower-impact changes to the beta. It is sometimes a bit odd that we approve higher risk changes and deny lower risk changes, but there's some sanity behind it :-)

    In a beta cycle, we are working to get our features in and all of our localizable resources frozen so we chase down any last feature changes, especially those with localizable strings. Our blocking and approval flag setting during beta are heavily influenced by our desire to get those changes all settled so that our testing and localization communities have a clear idea of what we plan to ship by final and we can get good testing coverage and as many complete localizations as possible.

    During the final cycle, usually on the branch, we're all about minimizing risk and preventing changes that would break localizations. In addition to those criteria, we are looking for spit and polish changes, stability improvements (any low-risk topcrash fixes) and those routine changes to our application identifiers like user agent strings.

    During the alpha cycles, drivers operate more as individuals and we often make blocking and approval flags without consulting all of the other drivers on the team. During beta, we usually get more focused and try to ensure that we've got more than one person making the decision. In a final cycle, we often seek a "triple handshake" where we have at least three drivers agree to a change, to make sure that we've looked at the bug or the patch from various different perspectives before plussing or minusing it.

    Lots of other factors come into play when you start talking about specific bugs and much of this is handled on a bug by bug basis. It is not at all uncommon that we make decisions outside of the criteria listed above.

  2. Is that mostly bases on what is fixable with the "limited" resources you* have or based on what you feel must/should be fixed ?

    It's extremely difficult for me to generalize here. In some cases it's about the available resources. In other cases, it's about the value of the change. There have been many driver decisions to not take a fix that was ready to go or to even remove a feature that was already implemented. There have been other cases where we've influenced developers, paid or unpaid, to stop work on other things and tackle something that was really important. If drivers feel it really must be fixed, we'll do what we can to make that happen. Sometimes that is a specific bug that can be tackled during a single cycle or even freeze period. Other times, with large features or architecture, it's a much longer-term effort.

  3. Why are bugs that are plussed for e.g. FF1.0, but didn't make it, automatically plussed for the next release (create the new version flag before the current release is out)? This way testers don't get the idea (impression/fear) that bugs are forgotten.

    Maybe I misread your question, but I assume you meant "why aren't plussed bugs that fail to make it into a release automatically plussed for the next release?" Again, this is hard to answer in the general because bugs get plussed for a lot of different reasons. Many bugs which were plussed for a release but, for whatever reason, didn't make it into the release are plussed for the next release. Some bugs which were thought to be a priority are no longer as high on the list when compared with new issues in the new release. Other times we were "abusing" the plus flag as a "we really want to get this" or "let's be sure to look at this closely before we ship" but it's not technically something we'd block the release for. It's often the case that when one of us uses a flag in this way, we comment something to that affect in the bug. The flags by themselves simply can't convey all of the purposes we've used them for so it's also important to read a bug and any comments from drivers, developers and testers in that bug.


jens.b asked quite a few questions and so I've tried to get to each one, but somewhat more briefly.

  1. Which is the general rule (or: the blocking=+ chances) for blocking=? bugs that... a) have a patch that is not yet reviewed, or have a review=- and need a fixed patch? b) don't have a patch, but the assignee promises to make one and let it review in time? c) have no patch, and no active developer? e.g. "we never set blocking=+ on such bugs", "only if we think we really need it", "that depends (on...)"...

    In general, jens, we really try to reserve the blocking+ flag for bugs which we would actually block the release on. If a bug has a patch, then the request should be for approval rather than blocking -- unless the problem really should block the release. Occasionally, there's a patch that we're very interested in and just to make sure it doesn't get overlooked, drivers will set a blocking flag, but the normal case is that we will often give approvals to patches on bugs that aren't necessarily release blockers so the patch flag should be used rather than the bug flag.

    I certainly believe that we'd be more likely to set the blocking+ flag on a bug that had a chance of being fixed than one that didn't but the blocking flag is mostly about whether or not we think that level of problem should block the release, not whether or not we've already got a developer commitment to fix it. We often plus bugs as blockers with no developer commitment and then we go looking for help. Other times, if the bug isn't terribly important to that particular release, we'll minus it as a blocker even if there's a patch in the bug (in those cases, I'll usually say something like "not going to block for this but we'd consider an approval request for a fully reviewed and tested patch." or something like that).

  2. Do the "blocking-flag guidelines" differ for alpha, beta and final releases - if so, how?

    Yes, they do vary some. I covered a piece of this in one of the responses above, but to try to explain it more clearly and with hypotheticals, there are lots of issues for which I'd say "if this doesn't make alpha or beta, it's not going to ship in the final" and for those bugs they might get labeled alpha or beta blockers but if they didn't make it, they wouldn't be final blockers. There are other bugs which seem bad to hand to end users, but that I believe our testing community can work around. Those might be final blockers but not alpha or beta blockers. As I mentioned above, we don't have hard and fast rules, but our general guidelines do provide the latitude we need to get alphas out with minimal developer interruption (tree closure) while maximizing new code testing, and to produce final releases of much higher quality.

  3. Would drivers try to find someone (from module owners/peers/MoFo staff/mozilla.org) to work on a bug that's been approved as a blocker, but that has no (or no longer has an) assignee? Or are bugs that are not yet being worked on a near-automatic blocking=- ?

    I think that depends on why the bug became a blocker. There are no automatic or "near-automatic" minuses given for bugs that drivers have determined to be real blocking issues. The only semi-automatic minusing that happens usually happens when the bug flag is still in the request state, not in the already plussed state. I, for example, semi-automatically minus ancient feature requests as blockers because in most cases, an age old feature request hasn't blocked us up until now and probably shouldn't block us going forward -- that is, unless something has changed that equation.

    To make a long answer somewhat shorter, yes, we do regularly go seeking help from all of our developers from module owners to paid Mozilla Foundation staff, for critical blockers.

  4. During the code freeze, are there usually approved checkins that are not because of regressions and not for blockers? E.g. features that were finished last-minute and should be in alpha/beta to be mass-tested, or even go into the release because they're considered so important... If so, can you think of (current or older) examples?

    Yes, during freezes, we often approve patches to land from bugs which were not labeled as blockers. If we determine that the risk to the release is minimal and that there is significant value to the change, we regularly approve patches to land that we would never label as blockers -- or that we had even evaluated and explicitly minused as blockers. You could probably construct a fairly simple Bugzilla query that would give you all of the cases where a patch was approved to land for a bug which was either not plussed as a blocker or even minused as a blocker. It all depends on the risk versus the reward tradeoff for the patch.

  5. Do you sometimes need to discuss a bug with other drivers/module owners in order to decide whether it's a blocker? If so, does it take the form of "need your advice", or is it even a consensus thing?

    I mentioned a little bit above about this. Some folks might get the mistake impression that I'm the one making all of these decisions. Given that I'm the guy that does most of the marking in Bugzilla and that I often talk about releases as though I'm solely responsible for everything ;-) this isn't the case at all. There are about a dozen of us that drive releases and we all play a role in the decisions. Some decisions are easy for one person to make, others require more than one person, or even most of the full group.

    The decision whether or not to block a final release for an unimplemented feature request from three years ago is pretty easy for me to make alone. The decision to turn on SVG in the default builds is one where we'd want a lot of driver (and other interested parties) participation. I very often turn to the experts in particular code areas -- whether they be drivers or not, before making the decision about a blocking or approval request. Sometimes that's me asking Brendan about a JavaScript engine change or David Baron about a style system bugfix. Other times it's me asking Darin about the impact of a networking change or asking Sarah to test a Mac fix before a decision is made.

  6. Not being a developer, do you find it sometimes difficult to estimate the risk a bug introduces?

    Definitely! That's why I'm so quick to request that information from the developers who requested the approval. I'll also consult with other developers, other drivers, and get actual testing from QA. Evaluating risk was much harder for me several years ago. It's gotten a lot easier though as I get to know which areas of the code are more or less stable, which developers have reputations for breaking or never breaking things, etc. There's no doubt, however, that I'm not as capable as almost everyone else on drivers and most of our developers at judging code change impact so I am very quick to ask for that information. As a rule of thumb, if you're requesting approval for a change to land, you'll get a lot faster response from drivers (and especially me) if you include a risk assessment and any information about the testing you performed on the code you'd like to land.


z had only one question.

  1. Why is Mozilla release schedule as muddy and delayed as Microsoft. And what are you guys doing to improve the on-time success rate? [these days, almost NO ONE believes that Firefox 1.1 is due by march end or so.] I know it is tough to predict in software etc, but, i would be glad to know, if something is done to see that we don't lapse like we often happen to do :(

    I'm not sure I'd agree with your characterization but maybe I can shed some light here. Yes, we are often delayed in our releases from what we publish in the developer roadmaps. First thing, though, is that the developer roadmaps, aren't necessarily commitments to consumers. They're developer roadmaps and they are designed to try to coordinate the contributions from a massive team of contributors from all across the globe. As we grow and our consumers, especially our distributors, and other large-scale customers, become more dependent on our releases, we're working to communicate our release plans better. In the end, though, we care a lot more about quality than we do about a few days or even weeks slip in our development schedule and I'm not ready to switch those priorities around. As for individual consumers waiting to download the next release, if you're looking to the developer roadmap for specific release dates, that's probably not a good idea. That page was designed as a resource for our developers and for other projects and organizations who were developing or deploying Mozilla-based applications, not as a public release date commitment.


Bernd asks a great couple of questions that I've done my best to answer honestly.

  1. Some bugs require more than one day work, do drivers also encourage long term fixing well in advance of a release, if so how are these bugs selected. Could you show some examples where that steering did work nicely. When does the release preparation begin relative to the closure of the three.

    Usually when a release comes very close there is some hardcore reminder of bugs that don't get fixed even if they are marked as blocking and linked from tinderbox, how do you deal with them, does MoFo throw its internal resources on this?

    Yes, we try, but this is an area where we really need to do better. First, we need to surface these problems sooner, then we need to start rounding up resources to make their solutions happen sooner.

    All of drivers have other workloads and contribute time driving as only a fraction of their Mozilla workload but we do need to put more time in during the release cycle and not just in the last few days.

    We're not completely falling down here, though :-) There have been quite a few large-scale changes that had driver involvement from very early in a release cycle which helped to get the issues sorted out in time for a release. Much of this used to be tracked in our branch/patch landing tool. Now that we have a better Bugzilla feature set, a lot of that has moved to bugs. A few quick examples would be things like getting the Firefox 1.0 and Seamonkey 1.7.5 Geckos synced up, getting the new Plugin scripting API landed, coordinating the code and other components for getting simultaneous localizations for Firefox 1.0 shipping in a couple dozen languages from Mozilla build machines, finding resources to fix the "mangler" crash problems, coordinating the enabling of xtf, SVG, plugin finder service, and more. I'm sure I could read back through my drivers mail and find lots more examples.

    Still, we definitely need to be more proactive early in the development cycles and I'll be putting in more of my time to try to make this happen. If you've got suggestions of specific ways to improve this, please send mail to drivers@mozilla.org. I'm always looking for help :-)


Jilles had a number of questions. I've tried to answer them but if anyone would like clarification or follow-up, please comment here and I'll try to add more in the comments.

  1. I have great respect for the innovative style the Mozilla project has been managed, which IMHO is the main reason for the success several of its products is currently enjoying. A problem area however is roadmaps.

    Mozilla releases appear to be mostly time driven (release every x weeks) with the exception of major releases which follow a less predictable scheme (and are often delivered later than planned: e.g. Mozilla 1.0, Firefox 1.0 took the best part of a year to pin down after being announced in a roadmap. Do you agree?

    I don't agree that we're mostly date driven. We certainly do publish milestones and shoot for hitting certain date targets. Without that, we'd have no way to coordinate the contributions from so many people and organizations. There were about 1,000 people who contributed code last year. That's a lot to coordinate and it comes from people all across the world, working on a very large codebase. Without target dates, we'd never release anything.

    We've also learned over the last half decade that there are limits to how much we can change without getting the more widespread testing that we get from our alpha and beta releases. We have hundreds, sometimes thousands of people downloading and testing our nightly builds each day, but until we reach tens of thousands or hundreds of thousands of testers that we get with our milestone releases, some problems might go unseen. If we didn't have these milestone releases regularly and on roughly the cycle lengths we've been using, our code wouldn't get the depth and breadth of testing it gets in time to make any regression fixing easier to do and we wouldn't be able to make necessary improvements to new features without the widespread feedback we get from those milestone downloaders.

    So, yes, we do care about getting our alpha and beta releases out on some kind of a regular schedule, but we're not doing it because we must hit certain dates. We're doing it because we've learned that we benefit from that regular testing, and talkback data.

  2. However roadmaps are about features and technology and there seems to be a fairly consistent inability to predict/steer the delivery of features in a particular roadmap. Many people are no doubt interested in Mozilla 2.0 and its relation to the various Mozilla projects. Could you comment on the current plans for moz 2.0 (both time and features)?

    I think I disagree with your characterization of our roadmap. It is about more than just features and technology. It's about coordinating a massive project of distributed resources. Yes, it's about planning for our projects and products which necessarily includes feature and technology planning, but it wouldn't serve us well to just have a roadmap that described a linear series of code additions.

    I think I understand what you're getting at, though, that drivers haven't been responsible for describing all of the coming changes and how and when, precisely, they would land and be available in product releases. I can't argue with that. I do think that it overlooks a lot though. First, if you go back and look at the push to Mozilla 1.0, you will see that there was a lot of work put in to build the requirements list from a technology standpoint, from a product feature set and usability standpoint, and from a quality standpoint. I know because I put in a lot of hours on that effort and I think it was quite successful. We've been ramping up to get the 2.0 requirements and plan in place now. You can follow that in the newsgroups, in the various roadmaps, and at the developer wiki.

    I think it's worth pointing out, though, that the Mozilla projects aren't as hierarchical and top-down as I think a lot of people expect. There are lots of feature and technology decision that are done at the Module Owner level and even some that happen at the individual developer level. A lot can and does happen without drivers@mozilla.org leading or getting in the way. Drivers can and should do more to surface major changes so that they'll be visible to the broader Mozilla developer community.

  3. Is there going to be a realistic, long term technology roadmap or are you planning to stick to the proven release every x weeks and we'll see what will happen (which is perfectly fine with me).

    We've had several major roadmaps that I think were mostly all realistic and long-term technology plans. The first one described the major changes around the movement to the new layout engine, the new networking libraries, the new xp frontend, a new build system, etc. I think we met all of those goals at the same time as "releasing every x weeks".

    The second major roadmap update outlined our push through the Netscape 6 release to a new milestone schedule, a new review system, and a new project management group (drivers@mozilla.org). For those people who weren't paying a lot or close attention back then, this was a critical time as Mozilla began to differentiate itself from Netscape and started to develop an independent technology and product plan with a focus on performance, stability and correctness rather than new features. I consider our weathering the Netscape 6 release and setting the stage for a Mozilla 1.0 release to be a great success.

    The Mozilla 1.0 Manifesto was the next update and I think was one of the most specific and realistic technology roadmaps we've had. After 1.0, the next roadmap was introduced along with a new milestone plan that moved away from the release every 5 weeks to an alpha/beta/final plan. This plan had a focus on how to handle our first long-lived branch and a major push for embeddability on the trunk with a focus on footprint, memory usage, and performance. We made awesome gains in those areas during the 1.1-1.4 cycles. I'm not sure how that can be characterized as anything but a great success. It was during this time that I also made a big push to get community nominations going, which I think has been one of the most important improvements we've made in our project management process.

    The next roadmap introduced the plan to move our focus to the stand alone applications, Firefox and Thunderbird. We were ambitious about the goals and the time frame. I think we delivered on these goals with our most successful product releases ever. We didn't make it on the time frame we'd hoped, but I'm hoping you'll cut us some slack given the amount of time and resources that were required to set up an entirely new organization, find the funding, find and hire the people, achieve non-profit status, and all while continuing to deliver several major updates to the legacy Mozilla suite.

    It's time for a new roadmap (well, maybe a little bit past time) and Brendan, drivers, and our development leads have been working out the details since last year's Developer day where Brendan outlined the big picture (slides available online). Discussion continues on the Wiki, in drivers meetings, in meetings with module owners and even representatives from other browser makers and major Internet technology builders. This document doesn't exist yet, but much of the work that will be reflected in it has been going on for quite a while. You can see lots of great work going on in areas like XULRunner, WebForms 2.0, SVG, APNG, Canvas, NPRuntime, Cairo integration, an entirely new storage system, etc. When Brendan gets free from other commitments, he'll post the new roadmap document to the web site, but all of this work (and more) isn't happening without coordination and direction -- just because we don't have that page updated yet.

    I hope this didn't come across as too defensive, and I hope that it answers some of your questions. I'm happy to try to elaborate more or answer more specific questions if you have them.

  4. Could you comment on the relation (or lack thereof) between Bugzilla and the various roadmaps

    We've done quite a bit to track our roadmap plans and progress in Bugzilla. In the early roadmaps, we used dependency trees to track our blockers and I think that was quite successful. It was painful, but successful. Next I added the Bugzilla keywords for tracking nominations and approvals and with that we saw more people getting involved in the process. We also integrated the branch/patch landing tool as a half-way point between the roadmap and Bugzilla. It was extremely valuable for tracking specific steps for getting bugs or bugsets finished off to meet the goals of a milestone plan. With the advent of Bugzilla flags, I think we've taken a great leap forward in to opening up and expanding the participation of developing the roadmap and product plans to a much wider group of participants. Literally thousands of people have played a role in nominating bugs. (somewhat off-topic, but I did some queries about a month ago which suggested that nearly half of all the bugs that have ever been nominated as a release blocker by our community have been resolved as fixed.) We haven't done the best job in using the Target Milestone field (with the exception of the Thunderbird team who used it quite well in their push to 1.0) but I'm hopeful that this can be improved. Do you have suggestions for improving the relationship between Bugzilla and our other tools?


Darin Grimm asked two questions.

  1. What role do "support/issues from users" plan in determining what issues are considered blockers for the next release. The Mozillazine forums seem to be at least the most public way of gauging issues that users are having with a product like Firefox. What role do those experiences play in shaping the "critical" issues that need addressed and in the 1.1 release and beyond in order to improve the end-user experience?

    I think that these support groups play a couple of roles. First, there are a number of people from the groups that file, identify, or nominate bugs which impact large numbers of users. We could use more of this. If you have suggestions about how to improve this system, I'd love to hear them. The second is that I try to read as much feedback as possible in the forums, the newsgroups and blogs as so that as I'm evaluating bugs with drivers@mozilla.org, I can help to represent those users and to gauge the user impact (as opposed to code or technology impact that is the focus of some of the other drivers).

  2. This issue may not be completely appropriate here, but it has come up a lot in recent discussions of Firefox development with those of us on the outside looking in and it would be nice to get some comments from someone directly involved in the process. When the aviary branch landed on the trunk, there were a fair number of significant regressions and issues that appeared on the trunk builds, making them less usable for daily testing. Many of these have been there for weeks now with no apparent move towards fixing them, why is that?

    Yeah, I hear you :-) A couple of things happened here. The landing was mostly successful but it was interrupted by the holidays when many at the Mozilla Foundation (including me and Tracy and Marcia and Sarah who all do regularly daily smoketesting) were taking vacations and recovering from the Firefox and Thunderbird releases. When we got back, it was time to jump right in to the 1.8a6 release work and that meant that Firefox trunk builds had to take a backseat for a couple of weeks. I hope to be able to get back focused on the state of the Firefox and Thunderbird trunk builds soon.

  3. Thank you Asa, your willingness to field questions from those simply trying to learn more about this process is appreciated!

    Thanks. I hope to be doing more of this in the future. It's somewhat time consuming and with all of my other responsibilities, it's easy to let this kind of thing slip, but I believe that it's critical for our success that we remain as transparent and engaged with our community as is possible. It's very easy to crawl into a shell and put your nose to the grindstone when pressure is high, especially around things like major releases, but in retrospect I always realize that was the time when being open and engaged with a much larger group would have been most beneficial.


Going in order of the questions, there were a couple more form jens.b.

  1. Beyond the "continue to maintain and release security updates for the last stable branch of the suite, 1.7.x" statement, will there be further releases of Mozilla that incorporate the latest Gecko improvements - e.g. 1.8, 1.9, 1.10(?)... What are the prerequisites for that?

    I believe that there will continue to be trunk based releases of the Mozilla application suite because there are members of the community who want it to continue and will contribute the resources to make that happen.

  2. When you posted a draft roadmap graphic, you spoke of the "Gecko 1.8" release - is this just an alias for the Seamonkey 1.8 release? What form will a "Gecko release" have - e.g. a Seamonkey build, a XULRunner version, a Gecko SDK...

    Gecko is the core component shared by all of our applications and as we have more and varied products all shipping from the Mozilla codebase, we'd like to see them all ship with compatible Gecko versions. Rather than tracking all of our products releases based on "Seamonkey" branches, we're going to start looking at this as "Gecko" branches. If everything goes as it's planned today, we'll be shipping a Mozilla (Seamonkey) 1.8, a Firefox 1.1, and a Thunderbird 1.1 from the Gecko 1.8 branch.


Doug's question is about the Aviary branch landing.

  1. There are currently 104 bugs (Fx+Tb) that are marked fixed-aviary, but aren't fixed on trunk. Is it safe to assume that these are all 1.1 blockers?

    We'll have to evaluate all of them. There are probably some of those which are fixed on the trunk or which never existed on the trunk and they just need to be closed. Overall, my goal would be that 1.1 doesn't regress functionality over 1.0 so if it's genuinely a bug on the trunk that was fixed for Firefox 1.0, we should get that fixed for Firefox 1.1.


The final set of questions was from Wally. I'm not sure what kinds of answers you're looking for here. It sounds to me more like a checklist from a textbook than a genuine inquiry into how we make releases happen. If I'm wrong and there is more specific information you're interested in, please let me know in the comments.

  1. Is there a release process? (Evidence for a process could be a document or a web-page describing it).

    Yes, see the Mozilla Release Checklist.

  2. Is the process under control? (Evidence for this could be a date or version issue control system, and approval and review system etc.)

    Yes, see the Roadmap and Bugzilla.

  3. Are there release criteria? (Which would be listed in the process).

    Yes, see the Roadmap, the Mozilla Release Checklist, and Bugzilla.

  4. Are these criteria based on objective measurements of the software under review, or are they arbitrary? (e.g. date driven, or decided by an individual in an ad-hoc manner).

    Our criteria are based on lots of things and are rarely arbitrarily date driven or decided by an individual.

  5. Are there metrics for the process?

    Yes. See Bugzilla.

  6. Is there a process owner? (Someone who has responsibility for monitoring the process and initiating improvements).

    Yes.

  7. Is the process effective? (Is the released software meeting the criteria).

    Yes. We can always improve.

  8. Is the process efficient? (Does it take too much time and effort to operate).

    Yes. We can always improve.

  9. Are there methods for improving the process?

    Yes.

  10. Are the stakeholders (all those affected) in the process involved in the definition/implementation/measurement/review of the process?

    As much as they're willing to be.


Well, this was fun :-) Let's do it again sometime. Maybe I can get it done in less than 5,000 words next time. But, seriously, I hope this was a valuable read for you all, especially the folks that put time into asking some great questions. If I didn't answer your question to your satisfaction, or misunderstood what you were asking, feel free to follow up in the comments here and I'll do my best to respond. Posted by asa at January 15, 2005 07:06 PM

Comments

Asa, you rock! Great answers.

Posted by: Basil Crow on January 15, 2005 09:33 PM

wow... I'll have to read that one in segments.

Anders

Posted by: Anders on January 15, 2005 11:30 PM

Ha, I have great respect for you for typing out that much for a journal entry.

Posted by: larfnarf on January 16, 2005 02:25 AM

Asa, this is really a outstanding post, posting it at mozillazine would probably a good idea.

Bernd

Posted by: Bernd on January 16, 2005 06:26 AM

Thanks for the insight Asa... great reading!

Side note: you might want to find a way to summarize for RSS... your whole multipage post appears on http://feedhouse.mozillazine.org/

Marc

Posted by: Marc Randolph on January 16, 2005 06:44 AM

Excellent post - definitely valuable. Thanks Asa :)

On the point about the fixed-aviary bugs - I would imagine that it's most, rather than some, that are fixed. Any fixes in /browser and /toolkit will have been landed with the branch landing. Unless they have some bad interaction with another trunk change, or made back-end changes only on the aviary branch, the bugs should all be fixed. Of course that doesn't change the fact that they all need evaluating.

And in terms of missing questions - I posted a question a related question on a different post which it would be nice to have a response to (although I think it overlaps a bit with the stuff about managing releases early in the cycle). A practical example would be the smoketest blocker - bug 274903 - that's existed in Firefox trunk since the beginning of December. It's now got a tested and reviewed patch ready to land, but the patch author has so far failed to find anyone that has the time to help (i.e. check it in).

Posted by: michaell on January 16, 2005 07:09 AM

That was fairly comprehensive I'd say...

Posted by: Doug Wright on January 16, 2005 10:05 AM

Let me take just one of the responses to my queries (not from a textbook but an off-the-top list from someone with ten years in SQA):

Q: Are there release criteria? (Which would be listed in the process).

A: Yes, see the Roadmap, the Mozilla Release Checklist, and Bugzilla.

Well I read those documents and frankly I'm none the wiser. One or two of the responses in this blog help define things like the +/- tag, but I see no criteria. Deciding bug by bug what is to be included in a release and what is not is an un-planned, ad-hoc way of developing software. In the case of Thunderbird, widely criticised on mozillazine for going to version 1.0 too early, there are lots of unresolved issues about new features, and about some long standing feature requests that have not made it so far into the release.
There's nothing in the release checklist or the roadmap that tells us when these features will be added (if I missed it, sorry, please correct me). I think the relase process is date driven, and the story smells like 'ship to time, ready or not'. Reclassifying inconvenient bugs from one release to the next is another symptom of this. Criteria should be something like "Features to be included: ; percentage of priority bugs to be cleared: N%; failures on regression test spec (name) permitted: Y% of category test cases; documentation required for release : complete and approved."

Enough from me; it's good to see these issues being opened out, MSFT are certainly improving their development process and Moz can't affort to lag in this area.

Regds, W.

Posted by: Wally on January 16, 2005 02:26 PM

Sorry, the blogging software mangled the criteria list (!), should read someting like:

"Features to be included: [list of features] ;

percentage of priority (high) bugs to be cleared: N%;

failures on regression test spec (name) permitted: Y% of category (whatever) test cases;

documentation required for release [list of docs] : complete and approved."

Posted by: Wally on January 16, 2005 02:32 PM

Wow, thanks Asa for your very detailed answers. It wasn't my intention to put you on the defensive though I admit being slightly provocative :-).

My main observation simply was that every few months or so a a revised roadmap comes out listing some specific dates and milestones and sometimes with some technology details. And then the dates just slip. Usually for good reasons but they do slip.

I watched the firefox 1.0 release date slip from beginning 2004 until the final release in november. During most of this time 1.0 always was a little more than three months away, which appears to be some sort of natural boundary beyond which we are not very good at planning things (unless we can break it down in smaller steps). In other words if we plan it beyond this boundary that really means we're not sure how long it's going to take.

Therefore it didn't surprise (or worry) me that these dates slipped because they always do (I remember thinking that no way are they are going to pull off a 1.0 in spring 2004 somewhere in december 2003). The same happened to mozilla 1.0, which depending on how you look at it, took more than one and a half year from the moment netscape thought is was ready (6.0) until everybody agreed it was ready (the 1.0 release). During this time extra milestones were being inserted and the product improved significantly (so it was not a bad thing). But the key point is that nobody ever planned that this would happen.

Roadmaps are not that important other than for communicating and stimulating discussion about future technological direction. The overall development process mozilla uses ensures a degree of progress and stability I and many other users are very happy with. From milestone to milestone the progress is visible and ever impressive. And the milestones come at regular intervals so there's something new and exciting every few months.

The current development process is excellent for mozilla and arguably the most effective way of coordinating the huge development and testing community. An OSS project like mozilla doesn't really have critical deadlines (no angry customers banging on the door). The only thing that is critical is quality and quality is what we are getting from the current process. In short, you guys are doing a great job!

I'm looking forward to some nice 1.1 nightlies in the near future.

BTW thanks for the historical overview of roadmaps, very insightful.

Posted by: Jilles on January 16, 2005 03:22 PM

Wow! You must have been trying to see if this blog s/w has a limit on posting length, or they are paying you per-word. ;-) That was the longest single post I've seen...

Posted by: Jeff Wilkinson on January 17, 2005 09:49 AM

Asa, thanks for this. It's very valuable as it encourages people to understand how Mozilla works in the inside.

Great job,


--Tristan

Posted by: Tristan Nitot on January 17, 2005 10:22 AM

Very interesting post, thanks for sharing

Posted by: nsk on January 18, 2005 05:42 AM

Nice read.

Posted by: Azmeen Afandi on January 18, 2005 05:49 AM

Hey,

At times I read blogs but only if they appear to be a good read. Not the daily chit chat about how someone changed his pants or something.

THIS IS a good read.

Great questions and very good answers.

Thanks for taking the time to publish this :)

Posted by: Floris Fiedeldij Dop on January 18, 2005 06:01 AM

And now you are about to get slashdotted...

Posted by: Doug on January 18, 2005 07:16 AM

Nice work Asa.

I'm not sure that feeling pressure from MSFT just because they alledge to be in the midst of improving their software development practices is anything Mozilla staff should loose sleep over.

After all they've already seen hundreds of friends lose jobs due to MSFTs illegal practices, right?

And there are plenty of other issues of the day.

Posted by: CaseyD on January 18, 2005 10:50 AM

Post a comment