preed's blah-blah-blahg

With last week's release of Thunderbird 2.0.0.6, we've finished a stint of security fire drills.

My friends and family have come to learn what that means¹, but I often get asked what it feels like to be in firedrill mode, especially on a [consumer-oriented-yet-open-source] project like Firefox. I've been trying to come up with a good way to explain it, and I think I may have finally found something to help.

The idea came to me when I noticed that a scene from a particular movie kept flashing through my head, very distinctly and vividly, while working on various release activities during this last drill.

It's from a 70's movie called The China Syndrome², which I found a conveniently-You-Tubed³ clip of. You'll want to let the full clip load, since the part that flashes through my head most commonly in the last thirty seconds or so, starting at about 3:35 in.

Link to embedded clip above, since some feed readers strip out <object>s

After watching this, a friend asked "So... you're comparing firedrill-mode to a nuclear meltdown?"⁴ We shared a laugh about that, because it wasn't the analogy I was intending at all.⁵ What the scene does capture pretty perfectly is the sense of urgency and the mode the characters go into when "the alarms go off."

During a firedrill, there's bits and pieces of information coming in from all sources and angles; there's a level of coordination involved in terms of getting issues looked at and prepping for builds; there are usually always external people watching [from above] what we're doing and how we're doing it; sometimes, we have to address problems creatively, even though "the book says you can't do it"; there can often be conflicting information that makes making a decision difficult. Many of the roles, reactions, and interactions of the characters in the scene are reminiscent of how we get a firedrill out the door.

I actually hadn't noticed that until I watched the clip again, since I originally identified with the last thirty seconds⁶ and hence why it tended to flash through my head at weird hours while starting at Tinderboxen prompts.

Every component, everyone one of them, is tested, and then it's tested again. Every critical weld is radiographed. Every single thing is checked, double-checked, it's rechecked. Everything. ... In any thing that man ever does, there's some element of risk, right? That's why we have what we call "defense in depth." Now that means, backup systems to backup systems. To. Backup. Systems. ... The system works. Even with a faulty relay, even with a stuck valve, that system works.

I'm sure QA can certainly relate to the amount of testing we do to ensure we ship a quality product... and a number of those tests mesh with verification and testing we do on the release side. In our case, it's more "even with an invalid automatic update snippet, even with a bogus locale repack, we catch the error and correct them before the release."

We learn something new from every firedrill we do.

And as a result, we've got a lot of newer initiatives all over the project to make firedrills more smoother and faster, from automated regression testing in Tinderboxen to automated smoketests and BFTs to automated release deliverable verification. These are all going to improve on Firefox 2's already-pretty-awesome patch-delivery time.

The analogy I make here illustrates how stressful it can be for that time we're getting it out. All the stuff I mentioned above will reduce that stress, but I think at least some of it is self-induced⁸: we take this stuff and the way we respond to and engage in them very seriously, and I know I'm not the only one: everyone, from the developers focusing on the fix to the release managers coordinating everything to the QA engineers verifying the fix and looking for regressions, to the build engineers pushing the bits out experience this responsibility together. (In fact, as I watch this clip again, I identify with Lemmon's character at the end, but in the beginning of the clip, when he gets up from his desk as the alarms initially go off, an image of dveditz pops into my head for some reason... :-)

It's part of the commitment to security and, more generally, doing right by our users that not only defines Mozilla and our community, but that I know we all personally hold dear.

So maybe no matter how good we get at this, there will always be little alarm klaxons going off in my head when I hear the callemail come in.

But... maybe... just a little bit of that... is a healthy thing.

_______________________
¹ "Preed's gonna 'go missing' for awhile... don't ask..."
² If you haven't seen The China Syndrome, I highly recommend it... if
for no other reason than getting to see a young pre-activist Jane Fonda and 70s-haired Michael Douglas. And of course, any movie with Jack Lemmon is A++++ WOULD WATCH AGAIN.
³ Oh how I love thee, copyright infringement fair use!
⁴ I don't even want to think about the Slashdot headliner on that one...
⁵ Maybe I didn't make the connection because—not to spoil the movie if you haven't seen it—but there is no meltdown in the movie
⁶ We've striven for correctness on the common cases with the release automation, and in firedrill-mode, we can be faced with situations that aren't entirely common... so there's been some level of back-to-non-automated/manual release work. Performing certain staps manually has resulted in (somewhat familiar) typos, errors, and general suboptimality⁷ that you generally expect with a human—me—typing lots of commands in a magical, ordered incantation. When that happens, and it gets caught by QA, my reaction is... well... it's why I keep seeing this scene in my head. :-)
⁷ Is that even a word?
⁸ At least, I know for me it is... and yes, it's a bad habit. :-)

Posted by preed at 5:04 PM | Permalink | Comments (3)

Some months ago now, I was trolling through the [[Build:Farm]] for some free space to run some then-critical builds, and I happened upon a set of Tinderboxen named "Firefox-FS."

While looking at other machines, I found more of these "FS" builds. We ended up turning them off at the time, because it wasn't entirely clear what they were, and it didn't look like they were being used.

Well, that "FS" stood for "free software" and those were the builds of Firefox that were composed of completely free software (which tends to mean "no branding, no talkback, etc.")

These builds were missing for some number of releases, but now they've returned, and we've added them to our release manifest.

We will provide these free software builds moving forward for the 2.x releases.

You can find them hiding out on ftp.m.o in the contrib/free-software directories.

You might notice there aren't any Mac FS builds.

That would be bug 372859. (Now accepting patches!)

A big thanks to Coop, by the way, for reviving these builds and wrestling them to the ground for the 2.0.0.2 release. It was a lot of... not-entirely-pleasant work, shall we say, but he got it done.

Posted by preed at 12:44 AM | Permalink | Comments (2)

I'd just like to state for the record that the 1.5.0.9/2.0.0.1 release was quite possibly the suckiest release I've ever worked on.

And if any of you remember the sleeping-in-the-office-for-a-week-1.5.0.2 release, that's saying a lot.

It's also saying a lot because the entire release went relatively smoothly¹ until... well... today, when everything promptly started sucking. Hardcore.

Like most aviation accidents, it wasn't a single thing that caused so much doom and strife. It wasn't the air traffic controller who forgot to call out the traffic... or the maintenance guy who didn't tighten all the bolts... or the pilot who got distracted looking at a faulty warning light.

It was all of those things, working in concert with each other.

Similarly, the list of what-the-hell-were-we-thinkings²,³ includes things that, on their own, wouldn't have been a big deal at all. But compound them and you basically had fourteen hours straight of me banging my head against the wall and wanting to stab my face off.⁴

The sole thing that makes horrendous releases like today's at least bearable is the community's response and attitude when problems like this arise: the web team started working on the release in the morning, and remain responsive throughout the day; IT was around and helped debug deployment and database load and app cluster problems with us; the app team weren't bothered a bit by being called in the middle of the night to look at weirdness; my build colleagues pitched in to help run some extra verification while I worked on deploying the contingency plan; and QA stayed until the bitter end (past midnight!) to make sure it all worked and to sign off on it all.

I don't ever want to repeat a release day like this one.⁵

But—at the risk of sounding sappy—this pervasive and consistent attitude within the Mozilla project is what gives me the energy to face even the possibility of such beasts again.
_____________________________________
¹ At least, no bumpier than any other Moz release I've ever worked.
² Hindsight = 20/20, and all that...
³ My "WTF," while I think defensible, is particularly... WTFular.
⁴ I must admit, Winnebago Man did help at about hour ten or so.
⁵ And I hope we'll spend some time figuring looking at exactly what happened, so none of us have to.

Posted by preed at 12:10 AM | Permalink | Comments (4)

Last night, my blog became a bit of a lightning rod¹, as readers from Slashdot poured in to tell us exactly what they think of how we release software.

I just finished reviewing the hundred-some comments², and I think I've realized four things:

1. People who have managed and worked with large releases of open source software, involving millions of people, and tens-to-hundreds of partner mirrors seemed to understand the complex set of issues that accompany such a process, and were sympathetic. Those that hadn't... well... added mostly a lot of noise.
2. Most people missed the point.

   As Beltzner's comment points out, the real problem we're trying to address are news sites, articles, and the like reporting "OMGSquee Firefox 2 is released, go get it here!" when that's just flatly wrong. For sites that have editors, it's irresponsible journalism, because it's just false. And they need to know that it's both irresponsible and incorrect.

   For those that don't, users need to know the effect of all their "diggin'" and "reddit'ing," and hopefully "the community" will do the right thing.

   In both cases, many of Slashdot's core users are not what we're so worried about. When they notice others getting automatic Firefox updates that they may not be getting, they'll probably take the time to figure out why. They'll upgrade themselves, because not all 38 of their Firefox extensions didn't work with 2.0 yet. They'll find the en-US build, because they know how to chop apart URLs, and go trolling through the FTP tree for the right locale (and for Linux builds³).

   Where it especially causes problems is when geeks see it, and download it for their girlfriends' computers.

   Or boyfriends'.

   Or parents'.

   Or Liberal Arts major roommates'.

   Those people are going to start up Firefox 2, and find 404'd startup pages. The online help won't be there. Or if it is, maybe it won't be localized for their language yet.

   They may be using the wrong locale for their needs⁴. Heck, they may get an unsupported locale⁵.

   These are the people it really hurts.

   It tarnishes all the hard work that a community of hundreds put months of their time into, because some couldn't wait.

3. Yeah, this is open source software... and everything we do is in a fishbowl.

   That's the great part of open source software: you all get to see what the developers do, and what QA does, and what Build/Release does... and you get to call us stupid when we tell you the pan is hot, and you shouldn't touch it [yet]. But at the end of the day, you get heed the warnings and suggetions—or not—and make the decision about what's best for you and your needs.

   What I did find surprising was reading a comment that said "You can't appeal to people's good nature, when there are millions everywhere around the world waiting for the new version."

   Firefox and Mozilla have always relied on "people's good nature" to help each other out, do the right thing, and work together to produce great software and make the web a better place. Open source itself is founded upon this general principle.

   Firefox is unique, in that it's a very end-user facing application.

   I guess that sentiment tells me we've all got some work to do to help acclimate people to the idea of a community driven software development model, where taking down someone's Internet connection in Europe or Asia who's volunteered to host bits closer to you really isn't OK.

4. Finally, while going through the comment moderation queue, it's clear why a lot of you want Firefox 2 the second you can get it: the new inline spell-check feature for submission forms ;-)

Having said all that... Firefox 2 is now released.

And I'm off to take a nap now⁶.

___________________________
¹ Which... eh... not the first time...
² Not all of which were posted... but neither were just positive, back-patting comments posted.
³ It amuses me to no end that such a large open source site linked directly to Win32 builds
⁴ en-GB vs. en-US, for instance. Or es-AR vs. es-ES.
⁵ Where "unsupported" is defined as "won't get automatic updates, and remain secure on the web," which is like... a huge reason to use Firefox in the first place.
⁶ After having a piece of MS Cake 2006, Service Pack 2⁷ of course!
⁷ "Giving new meaning to dumping core."

Posted by preed at 2:19 PM | Permalink | Comments (3)

I mentioned this in today's Project Meeting, and now it's come up in the newsgroups.

Normally, I'd take the time to write up a more in-depth explanation, but since my schedule is... uh... "hectic" right now, I'll just say this:

1. No, we have. Not. Released. Firefox. 2. Yet.
2. When people link to bits directly on a random FTP mirror, they're doing a number of people harm including, quite possibly, themselves:
   • Digg and Reddit posts linking to direct FTP mirrors could be costing the operators of those mirrors hundreds to thousands of dollars in bandwidth bills, or may cause them to crash by linking directly to them. This could cause them to "un-volunteer" their services as a mirror, making it even harder to obtain Firefox on release days.
   • People posting direct link to FTP mirrors don't know if that mirror is a member of the Mozilla FTP Mirror Farm, or some random, unverified mirror. We work hard to verify that the mirrors in our farm are serving the same bits we released, and we cannot make the same claim about other mirrors that aren't part of our farm. When using direct FTP links to random mirrors, users run the risk of downloading bits that have not been checked to ensure they do not contain a virus or trojan.
   • "That's ok," you say: "I link directly to ftp.mozilla.org!" That can be even worse! Killing the project's FTP server does not help anyone, least of all people trying to obtain Firefox builds. And it makes for a grumpy IT group. And nobody wants grumpy IT groups. Especially a day before a release.
3. Linking directly to builds hinders our ability to remove/retract bits that we may have to remove for some reason. While this may not seem like a big deal, it becomes a problem when supporting users, one of our most important values. If, let's say, we pull a locale, due to a stop-ship bug—and yes, this is not a hypothetical—then users who've (pre-)downloaded that build will not receive valuable security updates for those builds. The counterargument to this is "Well, you should provide updates for everything you've ever offered on your FTP site." If we did this, we'd be spending valuable (and über-constrained) Build Team and QA resources generating updates and testing them for builds that weren't the final bits, and were never "released" as such.
4. Posting links before we release may point people to an incomplete FTP areas or mirrors. I haven't finished posting the source tarball, for instance. Will it happen before we release? Yes. Will there be unnecessary confusion from the open source community, wondering where this deliverable is? If you post links to an FTP site with the builds, yes.
5. Most articles have an unerring ability to link to the wrong thing. Slashdot's front page, for instance, currently links to the Windows British English build. I cringe at the thought of the community having to waste time while we're finishing things up with IRC, blog, and Bugzilla chatter asking "I got my build from Slashdot; why did you guys spell behaviour wrong?" And where are Slashdotters wanting uhh... you know... Linux builds supposed to get them? It's unclear from the article that directly links to an .exe for one [correct for one country, but mostly-wrong for everyone else] locale.
6. User experience can be degraded, leaving a bad taste in people's mouth: Firefox 2 has a number of components that use live content on websites. The whole community has been doing a lot of work to refresh, update, and translate this content, and parts of it are still coming together for the release. When you download a build, there could be various content, including certain parts of help, that are not yet ready. When you tell your friends to go download Firefox 2 before we announce it's ready, you're subjecting them to a degraded user experience, which could push them to go back to... "other browsers."

Now, before you suggest it, it's not as easy as putting in .htaccess restrictions, or setting the permissions on the files so people can't download them. The nitty-gritty details are in the newsgroups.

So please... just remember: "Preed the Release Engineer says: friends don't let friends download Firefox before it's released."

We know everyone's excited for the 2.0 release. We are too. But give us 24 hours, so we can make sure that your first experience with Firefox 2.0 is befitting of everyone's hard work on this major release.

I promise it's worth the wait.

Posted by preed at 2:27 PM | Permalink | Comments (78)

I know this is a family blog and all... but after having just pushed the button on Thunderbird 1.5.0.2/1.0.8 and Suite 1.7.13 bits, please to be forgiving me for feeling the need to quote one of my favorite movies right about now:

Bon voyage, mother!#$%er.

You. Were. Good.

I'm going to the hotel. I'm gonna take a shower. I'm gonna sleep.

For a month.

P.S. What Chris said...

Posted by preed at 10:58 AM | Permalink | Comments (2)

1.5.0.2 (and 1.0.8, if you're nostalgic like that) released today!

One of the big announcements for 1.5.0.2 is the addition of Macintosh Universal Binary builds, including all the standard locales.

A lot of people (Mark and Josh especially) worked really hard to get universal builds into 1.5.0.2. I only did a minimal amount of work to support the effort, but I just want everyone to know: the only reason I personally worked on Universal Binaries at all was to make Justin, our IT director, happy.

So, even if you don't own a MacBook (or even a Mac!) be sure to take a moment and download the Universal Binary.

Do it for Justin.

*** I must say, I get a kick when people read somewhere that the next version of Firefox is released, and I'm sitting there, staring at a bash prompt with things left to do.

14:55 <preed> I've had three friends tell me "Congrats on the release" already
14:55 <preed> I told them they have a problem with premature congratulation.

Posted by preed at 7:05 PM | Permalink

Mac universal binaries for Bon Echo Alpha 1 are finally available.

The source tarball too.

Posted by preed at 9:52 AM | Permalink

In a time warp wormhole temporal loop nexus bubble reminscent of the worst Star Trek plot-crutch, the rest of Earth caught up with /. today, when Firefox 2.0a1 (sshhhh! Don't call it that) Bon Echo Alpha 1 was actually released.

In a way, I wish the reports had been right: I've been sick for the last three days (breathing is overrated, let me tell you!), so hearing that that the release was done without me having to lift a finger would've been... nice.

Now quick: someone go publish that 1.5.0.2 and 1.0.8 are all out the door, too, so I can get some rest!

Posted by preed at 7:52 PM | Permalink

After some trials and tribulations (and lots of Tinderbox hugging), we finally have universal binaries for 1.5.0.2!

I just noticed they're being uploaded to the wrong location, so I'll have to fix that, but you can find the mixed-endianess here.

When they move, you can find the builds via the tinderbox-with-the-boring-name (and a "UniversalBinaries" tag), on the 1.8.0 Tinderbox page.

Special shout out to Mento and Josh, who helped us (me) tremendously in getting these builds available.

Truth be told, it took a little longer than it otherwise might have because...

Continue reading "Bits for every endianness and a binary for every instruction set" »

Posted by preed at 12:37 PM | Permalink

Hot on the heels of Thunderbird 1.5, and mere weeks after Firefox 1.5, Firefox 1.5.0.1rc1 was (mostly quietly) released last Friday for the world to play with over the long weekend.

(AUS2-based updates should be out this week.)

One of my first tasks was to set up the MOZILLA_1_8_0_BRANCH Tinderboxen for Linux and Win32, so while this is a release with no less than four digits in its version number (and some letters too!), it's been an important one for me personally, just to see the machinery of how a release process works at MoCo. It's been particularly valuable to see it from pretty much the beginning.

My initial impressions of it all?

It involves a lot of rsync(1)... not that there's anything wrong with that.¹

______________________
¹ As Jerry Seinfeld used to say...

Posted by preed at 11:39 AM | Permalink