I promised I'd get to all of your questions from my making releases happen post and I've finally got a few minutes free to take a stab at it.
Some of the questions overlap and some would take a lot more time to answer than I've got for this post but I've done my best to put together replies to each of the questions. I'm put my responses together in the format of direct replies to each of the people who had questions. Maybe later I can distill this into a more condensed post about the release processes at Mozilla.
This took a good bit longer than I expected :-) (hours, not minutes) so I haven't done any serious proof reading; please excuse grammar or spelling issues. Also, this is just one man's opinion and if I've got something wrong here or misrepresented anything, pin that on me and not on email@example.com or other members of the community.
Peter van der Woulde was the first to post questions, so he gets the first set of answers.
This is probably one of the most frequently asked questions and our process isn't based on a set in stone list of rules. As I'm sure you know, there are two types of flags we evaluate and set. The first is the "blocking" bug flag and the second is the "approval" patch flag. The blocking flag is for bugs which should stop the shipping of that particular release and the approval flag is for patches which should land during the freeze for that release.
It's probably also important to realize that there are about a dozen of us on firstname.lastname@example.org and we all come with different expertise. I am particularly focused on quality and how our testing and user communities will respond to each of the product releases. Others have a focus on things like Web standards, security, freezing and maintaining APIs, or the Mozilla application platform. We all approach the plussing and minusing of bugs and patches with a somewhat different set of criteria.
There are a lot of factors that come in to play, but I think over the years all of drivers have come to settle on some basic tenants.
First, our criteria for alpha, beta, and final cycle changes are all different. We're going to have a different approach to determining whether or not a bug should block an alpha release than we would for a final release. We're also going to be a bit more relaxed about the kinds of patches we approve to land during an alpha freeze than we would for a final release freeze.
Alpha cycles can absorb more risk than beta and final cycles so we're likely to try to use email@example.com's influence to get high risk patches (including features, major code rewrites or reorganizations) landed in the alpha cycles. I'd certainly rather hold an alpha release for a day or two in order to get the added testing on a major new feature before beta and I'd be willing to approve the landing of that change during the alpha freeze when I might push out lower-impact changes to the beta. It is sometimes a bit odd that we approve higher risk changes and deny lower risk changes, but there's some sanity behind it :-)
In a beta cycle, we are working to get our features in and all of our localizable resources frozen so we chase down any last feature changes, especially those with localizable strings. Our blocking and approval flag setting during beta are heavily influenced by our desire to get those changes all settled so that our testing and localization communities have a clear idea of what we plan to ship by final and we can get good testing coverage and as many complete localizations as possible.
During the final cycle, usually on the branch, we're all about minimizing risk and preventing changes that would break localizations. In addition to those criteria, we are looking for spit and polish changes, stability improvements (any low-risk topcrash fixes) and those routine changes to our application identifiers like user agent strings.
During the alpha cycles, drivers operate more as individuals and we often make blocking and approval flags without consulting all of the other drivers on the team. During beta, we usually get more focused and try to ensure that we've got more than one person making the decision. In a final cycle, we often seek a "triple handshake" where we have at least three drivers agree to a change, to make sure that we've looked at the bug or the patch from various different perspectives before plussing or minusing it.
Lots of other factors come into play when you start talking about specific bugs and much of this is handled on a bug by bug basis. It is not at all uncommon that we make decisions outside of the criteria listed above.
It's extremely difficult for me to generalize here. In some cases it's about the available resources. In other cases, it's about the value of the change. There have been many driver decisions to not take a fix that was ready to go or to even remove a feature that was already implemented. There have been other cases where we've influenced developers, paid or unpaid, to stop work on other things and tackle something that was really important. If drivers feel it really must be fixed, we'll do what we can to make that happen. Sometimes that is a specific bug that can be tackled during a single cycle or even freeze period. Other times, with large features or architecture, it's a much longer-term effort.
Maybe I misread your question, but I assume you meant "why aren't plussed bugs that fail to make it into a release automatically plussed for the next release?" Again, this is hard to answer in the general because bugs get plussed for a lot of different reasons. Many bugs which were plussed for a release but, for whatever reason, didn't make it into the release are plussed for the next release. Some bugs which were thought to be a priority are no longer as high on the list when compared with new issues in the new release. Other times we were "abusing" the plus flag as a "we really want to get this" or "let's be sure to look at this closely before we ship" but it's not technically something we'd block the release for. It's often the case that when one of us uses a flag in this way, we comment something to that affect in the bug. The flags by themselves simply can't convey all of the purposes we've used them for so it's also important to read a bug and any comments from drivers, developers and testers in that bug.
jens.b asked quite a few questions and so I've tried to get to each one, but somewhat more briefly.
In general, jens, we really try to reserve the blocking+ flag for bugs which we would actually block the release on. If a bug has a patch, then the request should be for approval rather than blocking -- unless the problem really should block the release. Occasionally, there's a patch that we're very interested in and just to make sure it doesn't get overlooked, drivers will set a blocking flag, but the normal case is that we will often give approvals to patches on bugs that aren't necessarily release blockers so the patch flag should be used rather than the bug flag.
I certainly believe that we'd be more likely to set the blocking+ flag on a bug that had a chance of being fixed than one that didn't but the blocking flag is mostly about whether or not we think that level of problem should block the release, not whether or not we've already got a developer commitment to fix it. We often plus bugs as blockers with no developer commitment and then we go looking for help. Other times, if the bug isn't terribly important to that particular release, we'll minus it as a blocker even if there's a patch in the bug (in those cases, I'll usually say something like "not going to block for this but we'd consider an approval request for a fully reviewed and tested patch." or something like that).
Yes, they do vary some. I covered a piece of this in one of the responses above, but to try to explain it more clearly and with hypotheticals, there are lots of issues for which I'd say "if this doesn't make alpha or beta, it's not going to ship in the final" and for those bugs they might get labeled alpha or beta blockers but if they didn't make it, they wouldn't be final blockers. There are other bugs which seem bad to hand to end users, but that I believe our testing community can work around. Those might be final blockers but not alpha or beta blockers. As I mentioned above, we don't have hard and fast rules, but our general guidelines do provide the latitude we need to get alphas out with minimal developer interruption (tree closure) while maximizing new code testing, and to produce final releases of much higher quality.
I think that depends on why the bug became a blocker. There are no automatic or "near-automatic" minuses given for bugs that drivers have determined to be real blocking issues. The only semi-automatic minusing that happens usually happens when the bug flag is still in the request state, not in the already plussed state. I, for example, semi-automatically minus ancient feature requests as blockers because in most cases, an age old feature request hasn't blocked us up until now and probably shouldn't block us going forward -- that is, unless something has changed that equation.
To make a long answer somewhat shorter, yes, we do regularly go seeking help from all of our developers from module owners to paid Mozilla Foundation staff, for critical blockers.
Yes, during freezes, we often approve patches to land from bugs which were not labeled as blockers. If we determine that the risk to the release is minimal and that there is significant value to the change, we regularly approve patches to land that we would never label as blockers -- or that we had even evaluated and explicitly minused as blockers. You could probably construct a fairly simple Bugzilla query that would give you all of the cases where a patch was approved to land for a bug which was either not plussed as a blocker or even minused as a blocker. It all depends on the risk versus the reward tradeoff for the patch.
I mentioned a little bit above about this. Some folks might get the mistake impression that I'm the one making all of these decisions. Given that I'm the guy that does most of the marking in Bugzilla and that I often talk about releases as though I'm solely responsible for everything ;-) this isn't the case at all. There are about a dozen of us that drive releases and we all play a role in the decisions. Some decisions are easy for one person to make, others require more than one person, or even most of the full group.
Definitely! That's why I'm so quick to request that information from the developers who requested the approval. I'll also consult with other developers, other drivers, and get actual testing from QA. Evaluating risk was much harder for me several years ago. It's gotten a lot easier though as I get to know which areas of the code are more or less stable, which developers have reputations for breaking or never breaking things, etc. There's no doubt, however, that I'm not as capable as almost everyone else on drivers and most of our developers at judging code change impact so I am very quick to ask for that information. As a rule of thumb, if you're requesting approval for a change to land, you'll get a lot faster response from drivers (and especially me) if you include a risk assessment and any information about the testing you performed on the code you'd like to land.
z had only one question.
I'm not sure I'd agree with your characterization but maybe I can shed some light here. Yes, we are often delayed in our releases from what we publish in the developer roadmaps. First thing, though, is that the developer roadmaps, aren't necessarily commitments to consumers. They're developer roadmaps and they are designed to try to coordinate the contributions from a massive team of contributors from all across the globe. As we grow and our consumers, especially our distributors, and other large-scale customers, become more dependent on our releases, we're working to communicate our release plans better. In the end, though, we care a lot more about quality than we do about a few days or even weeks slip in our development schedule and I'm not ready to switch those priorities around. As for individual consumers waiting to download the next release, if you're looking to the developer roadmap for specific release dates, that's probably not a good idea. That page was designed as a resource for our developers and for other projects and organizations who were developing or deploying Mozilla-based applications, not as a public release date commitment.
Bernd asks a great couple of questions that I've done my best to answer honestly.
Usually when a release comes very close there is some hardcore reminder of bugs that don't get fixed even if they are marked as blocking and linked from tinderbox, how do you deal with them, does MoFo throw its internal resources on this?
Yes, we try, but this is an area where we really need to do better. First, we need to surface these problems sooner, then we need to start rounding up resources to make their solutions happen sooner.
All of drivers have other workloads and contribute time driving as only a fraction of their Mozilla workload but we do need to put more time in during the release cycle and not just in the last few days.
We're not completely falling down here, though :-) There have been quite a few large-scale changes that had driver involvement from very early in a release cycle which helped to get the issues sorted out in time for a release. Much of this used to be tracked in our branch/patch landing tool. Now that we have a better Bugzilla feature set, a lot of that has moved to bugs. A few quick examples would be things like getting the Firefox 1.0 and Seamonkey 1.7.5 Geckos synced up, getting the new Plugin scripting API landed, coordinating the code and other components for getting simultaneous localizations for Firefox 1.0 shipping in a couple dozen languages from Mozilla build machines, finding resources to fix the "mangler" crash problems, coordinating the enabling of xtf, SVG, plugin finder service, and more. I'm sure I could read back through my drivers mail and find lots more examples.
Still, we definitely need to be more proactive early in the development cycles and I'll be putting in more of my time to try to make this happen. If you've got suggestions of specific ways to improve this, please send mail to firstname.lastname@example.org. I'm always looking for help :-)
Jilles had a number of questions. I've tried to answer them but if anyone would like clarification or follow-up, please comment here and I'll try to add more in the comments.
Mozilla releases appear to be mostly time driven (release every x weeks) with the exception of major releases which follow a less predictable scheme (and are often delivered later than planned: e.g. Mozilla 1.0, Firefox 1.0 took the best part of a year to pin down after being announced in a roadmap. Do you agree?
I don't agree that we're mostly date driven. We certainly do publish milestones and shoot for hitting certain date targets. Without that, we'd have no way to coordinate the contributions from so many people and organizations. There were about 1,000 people who contributed code last year. That's a lot to coordinate and it comes from people all across the world, working on a very large codebase. Without target dates, we'd never release anything.
We've also learned over the last half decade that there are limits to how much we can change without getting the more widespread testing that we get from our alpha and beta releases. We have hundreds, sometimes thousands of people downloading and testing our nightly builds each day, but until we reach tens of thousands or hundreds of thousands of testers that we get with our milestone releases, some problems might go unseen. If we didn't have these milestone releases regularly and on roughly the cycle lengths we've been using, our code wouldn't get the depth and breadth of testing it gets in time to make any regression fixing easier to do and we wouldn't be able to make necessary improvements to new features without the widespread feedback we get from those milestone downloaders.
So, yes, we do care about getting our alpha and beta releases out on some kind of a regular schedule, but we're not doing it because we must hit certain dates. We're doing it because we've learned that we benefit from that regular testing, and talkback data.
I think I disagree with your characterization of our roadmap. It is about more than just features and technology. It's about coordinating a massive project of distributed resources. Yes, it's about planning for our projects and products which necessarily includes feature and technology planning, but it wouldn't serve us well to just have a roadmap that described a linear series of code additions.
I think I understand what you're getting at, though, that drivers haven't been responsible for describing all of the coming changes and how and when, precisely, they would land and be available in product releases. I can't argue with that. I do think that it overlooks a lot though. First, if you go back and look at the push to Mozilla 1.0, you will see that there was a lot of work put in to build the requirements list from a technology standpoint, from a product feature set and usability standpoint, and from a quality standpoint. I know because I put in a lot of hours on that effort and I think it was quite successful. We've been ramping up to get the 2.0 requirements and plan in place now. You can follow that in the newsgroups, in the various roadmaps, and at the developer wiki.
I think it's worth pointing out, though, that the Mozilla projects aren't as hierarchical and top-down as I think a lot of people expect. There are lots of feature and technology decision that are done at the Module Owner level and even some that happen at the individual developer level. A lot can and does happen without email@example.com leading or getting in the way. Drivers can and should do more to surface major changes so that they'll be visible to the broader Mozilla developer community.
We've had several major roadmaps that I think were mostly all realistic and long-term technology plans. The first one described the major changes around the movement to the new layout engine, the new networking libraries, the new xp frontend, a new build system, etc. I think we met all of those goals at the same time as "releasing every x weeks".
The second major roadmap update outlined our push through the Netscape 6 release to a new milestone schedule, a new review system, and a new project management group (firstname.lastname@example.org). For those people who weren't paying a lot or close attention back then, this was a critical time as Mozilla began to differentiate itself from Netscape and started to develop an independent technology and product plan with a focus on performance, stability and correctness rather than new features. I consider our weathering the Netscape 6 release and setting the stage for a Mozilla 1.0 release to be a great success.
The Mozilla 1.0 Manifesto was the next update and I think was one of the most specific and realistic technology roadmaps we've had. After 1.0, the next roadmap was introduced along with a new milestone plan that moved away from the release every 5 weeks to an alpha/beta/final plan. This plan had a focus on how to handle our first long-lived branch and a major push for embeddability on the trunk with a focus on footprint, memory usage, and performance. We made awesome gains in those areas during the 1.1-1.4 cycles. I'm not sure how that can be characterized as anything but a great success. It was during this time that I also made a big push to get community nominations going, which I think has been one of the most important improvements we've made in our project management process.
The next roadmap introduced the plan to move our focus to the stand alone applications, Firefox and Thunderbird. We were ambitious about the goals and the time frame. I think we delivered on these goals with our most successful product releases ever. We didn't make it on the time frame we'd hoped, but I'm hoping you'll cut us some slack given the amount of time and resources that were required to set up an entirely new organization, find the funding, find and hire the people, achieve non-profit status, and all while continuing to deliver several major updates to the legacy Mozilla suite.
It's time for a new roadmap (well, maybe a little bit past time) and Brendan, drivers, and our development leads have been working out the details since last year's Developer day where Brendan outlined the big picture (slides available online). Discussion continues on the Wiki, in drivers meetings, in meetings with module owners and even representatives from other browser makers and major Internet technology builders. This document doesn't exist yet, but much of the work that will be reflected in it has been going on for quite a while. You can see lots of great work going on in areas like XULRunner, WebForms 2.0, SVG, APNG, Canvas, NPRuntime, Cairo integration, an entirely new storage system, etc. When Brendan gets free from other commitments, he'll post the new roadmap document to the web site, but all of this work (and more) isn't happening without coordination and direction -- just because we don't have that page updated yet.
I hope this didn't come across as too defensive, and I hope that it answers some of your questions. I'm happy to try to elaborate more or answer more specific questions if you have them.
We've done quite a bit to track our roadmap plans and progress in Bugzilla. In the early roadmaps, we used dependency trees to track our blockers and I think that was quite successful. It was painful, but successful. Next I added the Bugzilla keywords for tracking nominations and approvals and with that we saw more people getting involved in the process. We also integrated the branch/patch landing tool as a half-way point between the roadmap and Bugzilla. It was extremely valuable for tracking specific steps for getting bugs or bugsets finished off to meet the goals of a milestone plan. With the advent of Bugzilla flags, I think we've taken a great leap forward in to opening up and expanding the participation of developing the roadmap and product plans to a much wider group of participants. Literally thousands of people have played a role in nominating bugs. (somewhat off-topic, but I did some queries about a month ago which suggested that nearly half of all the bugs that have ever been nominated as a release blocker by our community have been resolved as fixed.) We haven't done the best job in using the Target Milestone field (with the exception of the Thunderbird team who used it quite well in their push to 1.0) but I'm hopeful that this can be improved. Do you have suggestions for improving the relationship between Bugzilla and our other tools?
Darin Grimm asked two questions.
I think that these support groups play a couple of roles. First, there are a number of people from the groups that file, identify, or nominate bugs which impact large numbers of users. We could use more of this. If you have suggestions about how to improve this system, I'd love to hear them. The second is that I try to read as much feedback as possible in the forums, the newsgroups and blogs as so that as I'm evaluating bugs with email@example.com, I can help to represent those users and to gauge the user impact (as opposed to code or technology impact that is the focus of some of the other drivers).
Yeah, I hear you :-) A couple of things happened here. The landing was mostly successful but it was interrupted by the holidays when many at the Mozilla Foundation (including me and Tracy and Marcia and Sarah who all do regularly daily smoketesting) were taking vacations and recovering from the Firefox and Thunderbird releases. When we got back, it was time to jump right in to the 1.8a6 release work and that meant that Firefox trunk builds had to take a backseat for a couple of weeks. I hope to be able to get back focused on the state of the Firefox and Thunderbird trunk builds soon.
Thanks. I hope to be doing more of this in the future. It's somewhat time consuming and with all of my other responsibilities, it's easy to let this kind of thing slip, but I believe that it's critical for our success that we remain as transparent and engaged with our community as is possible. It's very easy to crawl into a shell and put your nose to the grindstone when pressure is high, especially around things like major releases, but in retrospect I always realize that was the time when being open and engaged with a much larger group would have been most beneficial.
Going in order of the questions, there were a couple more form jens.b.
I believe that there will continue to be trunk based releases of the Mozilla application suite because there are members of the community who want it to continue and will contribute the resources to make that happen.
Gecko is the core component shared by all of our applications and as we have more and varied products all shipping from the Mozilla codebase, we'd like to see them all ship with compatible Gecko versions. Rather than tracking all of our products releases based on "Seamonkey" branches, we're going to start looking at this as "Gecko" branches. If everything goes as it's planned today, we'll be shipping a Mozilla (Seamonkey) 1.8, a Firefox 1.1, and a Thunderbird 1.1 from the Gecko 1.8 branch.
Doug's question is about the Aviary branch landing.
We'll have to evaluate all of them. There are probably some of those which are fixed on the trunk or which never existed on the trunk and they just need to be closed. Overall, my goal would be that 1.1 doesn't regress functionality over 1.0 so if it's genuinely a bug on the trunk that was fixed for Firefox 1.0, we should get that fixed for Firefox 1.1.
The final set of questions was from Wally. I'm not sure what kinds of answers you're looking for here. It sounds to me more like a checklist from a textbook than a genuine inquiry into how we make releases happen. If I'm wrong and there is more specific information you're interested in, please let me know in the comments.
Yes, see the Mozilla Release Checklist.
Yes, see the Roadmap and Bugzilla.
Yes, see the Roadmap, the Mozilla Release Checklist, and Bugzilla.
Our criteria are based on lots of things and are rarely arbitrarily date driven or decided by an individual.
Yes. See Bugzilla.
Yes. We can always improve.
Yes. We can always improve.
As much as they're willing to be.
Well, this was fun :-) Let's do it again sometime. Maybe I can get it done in less than 5,000 words next time. But, seriously, I hope this was a valuable read for you all, especially the folks that put time into asking some great questions. If I didn't answer your question to your satisfaction, or misunderstood what you were asking, feel free to follow up in the comments here and I'll do my best to respond.
Posted by asa at January 15, 2005 07:06 PM