We're in the middle of a release wrap-up and I'm pretty busy with that but it also feels like an ideal time to bring up this topic.
With literally hundreds of thousands of lines of code changing every cycle, it's clear that there's a lot of engineering and testing work that goes into every Mozilla release. You can see this by looking over one of the "rough changelogs" that I create for each release (Mozilla 1.7, for example.)
What a lot of people don't know is that there's also a fair amount of (I hate this word) management that goes into pulling everything together for a release. On the Mozilla projects, much of that falls onto the shoulders of drivers@mozilla.org.
Drivers was created several years ago "to drive developers looking to help toward bugs needing assistance in a timely fashion, to moderate risk, and to aid commercial projects based on Mozilla in managing their product releases."
I've been a member of that group since its formation back in 2000. One of the areas I'm most involved with drivers is releases and in the last 4 years I've had my hand in about 65 Mozilla application suite releases, almost 20 Phoenix/Firebird/Firefox releases and about half a dozen Thunderbird releases.
On several occasions I've tried to write up a good blog post about what it takes to make our releases happen but have never finished. So this time, rather than try to put it into a post, I thought I'd just open it up to questions and see what you want to know about the process.
I'll read all of the questions here and then in a follow-up post, attempt to answer them and in doing so, hopefully give you all a better picture of our release process.
Posted by asa at January 7, 2005 12:30 PM1. How do you decide what to plus and what to minus ?
2. Is that mostly bases on what is fixable with the "limited" resources you* have or based on what you feel must/should be fixed ?
3. Why are bugs that are plussed for e.g. FF1.0, but didn't make it, automatically plussed for the next release (create the new version flag before the current releae is out) ?
This way testers don't get the idea (impression/fear) that bugs are forgotten.
*you=moz devs
Posted by: Peter van der Woude on January 7, 2005 02:09 PM1. Which is the general rule (or: the blocking=+ chances) for blocking=? bugs that...
a) have a patch that is not yet reviewed, or have a review=- and need a fixed patch?
b) don't have a patch, but the assignee promises to make one and let it review in time?
c) have no patch, and no active developer?
e.g. "we never set blocking=+ on such bugs", "only if we think we really need it", "that depends (on...)"...
2. Do the "blocking-flag guidelines" differ for alpha, beta and final releases - if so, how?
3. Would drivers try to find someone (from module owners/peers/MoFo staff/mozilla.org) to work on a bug that's been approved as a blocker, but that has no (or no longer has an) assignee? Or are bugs that are not yet being worked on a near-automatic blocking=- ?
4. During the code freeze, are there usually approved checkins that are not because of regressions and not for blockers? E.g. features that were finished last-minute and should be in alpha/beta to be mass-tested, or even go into the release because they're considered so important... If so, can you think of (current or older) examples?
5. Do you sometimes need to discuss a bug with other drivers/module owners in order to decide whether it's a blocker? If so, does it take the form of "need your advice", or is it even a consensus thing?
6. Not being a developer, do you find it sometimes difficult to estimate the risk a bug introduces?
Posted by: jens.b on January 7, 2005 03:32 PMOne and only question.
1. Why is Mozilla release schedule as muddy and delayed as Microsoft. And what are you guys doing to improve the on-time success rate? [these days, almost NO ONE believes that Firefox 1.1 is due by march end or so.] I know it is tough to predict in software etc, but, i would be glad to know, if something is done to see that we don't lapse like we often happen to do :(
Thanks Asa.
Posted by: z on January 7, 2005 10:33 PMSome bugs require more than one day work, do drivers also encourage long term fixing well in advance of a release, if so how are these bugs selected. Could you show some examples where that steering did work nicely. When does the release preparation begin relative to the closure of the three.
Usually when a release comes very close there is some hardcore reminder of bugs that don't get fixed even if they are marked as blocking and linked from tinderbox, how do you deal with them, does MoFo throw its internal resources on this?
Posted by: Bernd on January 7, 2005 11:17 PMI have great respect for the innovative style the mozilla project has been managed, which IMHO is the main reason for the success several of its products is currently enjoying. A problem area however is roadmaps.
Mozilla releases appear to be mostly time driven (release every x weeks) with the exception of major releases which follow a less predictable scheme (and are often delivered later than planned: e.g. mozilla 1.0, firefox 1.0 took the best part of a year to pin down after being announced in a roadmap)
1) do you agree?
However roadmaps are about features and technology and there seems to be a fairly consistent inability to predict/steer the delivery of features in a particular roadmap.
2) Many people are no doubt interested in mozilla 2.0 and its relation to the various mozilla projects. Could you comment on the current plans for moz 2.0 (both time and features)?
3) Is there going to be a realistic, long term technology roadmap or are you planning to stick to the proven release every x weeks and we'll see what will happen (which is perfectly fine with me).
4) Could you comment on the relation (or lack thereof) between bugzilla and the various roadmaps
Couple of questions -
1) What role do "support/issues from users" plan in determining what issues are considered blockers for the next release. The Mozillazine forums seem to be at least the most public way of gauging issues that users are having with a product like Firefox. What role do those experiences play in shaping the "critical" issues that need addressed and in the 1.1 release and beyond in order to improve the end-user experience?
2) This issue may not be completely appropriate here, but it has come up a lot in recent discussions of Firefox development with those of us on the outside looking in and it would be nice to get some comments from someone directly involved in the process. When the aviary branch landed on the trunk, there were a fair number of significant regressions and issues that appeared on the trunk builds, making them less usable for daily testing. Many of these have been there for weeks now with no apparent move towards fixing them, why is that?
Thank you Asa, your willingness to field questions from those simply trying to learn more about this process is appreciated!
Posted by: Darin Grimm on January 8, 2005 03:45 AM(The following may be more generic Ask-Asa-style questions, but presumably, the move away from Seamonkey-centric releases to a Mozilla-the-Platform-centric operation changes the release and management processes drastically, so perhaps these questions are justified here. If not, please save them for the next Ask Asa round...)
1. Beyond the "continue to maintain and release security updates for the last stable branch of the suite, 1.7.x" statement, will there be further releases of Mozilla that incorporate the latest Gecko improvements - e.g. 1.8, 1.9, 1.10(?)... What are the prerequisites for that?
2. When you posted a draft roadmap graphic, you spoke of the "Gecko 1.8" release - is this just an alias for the Seamonkey 1.8 release? What form will a "Gecko release" have - e.g. a Seamonkey build, a XULRunner version, a Gecko SDK...
Posted by: jens.b on January 8, 2005 06:48 AMThere are currently 104 bugs (Fx+Tb) that are marked fixed-aviary, but aren't fixed on trunk. Is it safe to assume that these are all 1.1 blockers?
Posted by: Doug on January 8, 2005 10:54 AMPerhaps most of the fixed-aviary bugs are actually fixed on the trunk but haven't been tested and verified? Maybe there should be a Very Special Bugday to wipe these out?
Posted by: Kevin on January 8, 2005 04:22 PM1. Is there a release process? (Evidence for a process could be a document or a web-page describing it).
2. Is the process under control? (Evidence for this could be a date or version issue control system, and approval and review system etc.)
3. Are there release critera? (Which would be listed in the process).
4. Are these criteria based on objective measurements of the software under review, or are they arbitrary? (e.g. date driven, or decided by an individual in an ad-hoc manner).
5. Are there metrics for the process?
6. Is there a process owner? (Someone who has responsibility for monitoring the process and initiating improvements).
7. Is the process effective? (Is the released sofware meeting the criteria).
8. Is the process efficient? (Does it take too much time and effort to operate).
9. Are there methods for improving the process?
10. Are the stakeholders (all those affected) in the process involved in the definition/implementation/measurement/review of the process?
....
Regds, W.
Posted by: Wally on January 10, 2005 09:06 AM