preed's blah-blah-blahg

Astute IRCers may have noticed I've been on vacation. I'm now home, chillaxing¹ after having spent a few sunny days in Las Vegas.

It was my second time visiting the City of Lost Wages. I met up with a friend from the right coast and a couple more (local) friends joined us the next day.

This trip was more fun than last time, I think mostly because we did things other than gambling this trip around. We visited Hoover Dam² in addition to actually going to see a few shows.

I think my favorite was La Reve, but we also saw Blue Man Group, which was unexpectedly engaging and fun. We also randomly happened to catch Penn & Teller, an act whom it's hard to go wrong with.

Being the plane geeks that we all are, we also spent the last night at the pricey³ MGM Grand, with room overlooking KLAS's one-eights.⁴

We also sampled a bunch of buffets. The Bellagio's, while somewhat pricey, still stands as the King of Buffets. (Never eat at The Trop. You've been warned. In fact, you're better of ignoring, for the purposes of visiting Vegas, that the whole hotel exists.)

I also learned how to play blackjack in not an entirely lose-all-your-money manner. The first time I was in Vegas, I was a bit too intimidated to sit down and be required to interact with a real dealer. This trip, I learned that some dealers can be jerks, but for the most part, they're generally nice. I suppose I should've expected this, since you're basically there to give them all your money... and it would behoove them to be not-jerks.

In one of the hotels we stayed in, there was a big scary sign about removing items from the mini-bar; one of the friends who showed up the next day didn't read the sign, and started pulling a bunch of things out of the bar to inspect what type of booze-ahol they were, and then put them back. "Ooops."

At one of the friendlier blackjack tables, one of the players was asking what we all did. I mentioned I worked on Firefox, and he said he'd heard of it, but—once again—asked why he should use it.

I don't know if it was because of the free drinks or because I was up a few hundred bucks at that point, but I... remembered my friend digging through the hotel room minibar and spouted what I think would make a pretty good new slogan:

I'm sure pkim and cbeard will love this⁵

The whole table couldn't stop laughing. Hopefully that'll turn into a couple new users.

Since I did not hit the $16.8 million Vegas Bucks^TM jackpot, nor make a million playing table blackjack, I will be back, reading email and on IRC tomorrow.

__________________
¹ As the youngins these days call it
² "For all your dam needs!"
³ But actually worth it
⁴ I'm the one in the picture in the chair, with the martini, operating the radio. Obviously.
⁵ Lamentably, I am without camera. Still. So Lee L (??) provided this image, via Google Image search.

Posted by preed at 11:06 PM | Permalink | Comments (5)

Gen linked me to a recently posted article detailing an old interview from November, 2005 with Google's Eric Schmidt.

It was a great read, but Gen brought it to my attention for a specific reason¹:

Wired's Vogelstein: You've talked before about how flying your own jet helps you manage. Explain. The reason I'm intrigued by it is because the only other guy I know who flies a jet is (Oracle boss) Larry Ellison. But you can look at Larry Ellison and understand how the machismo associated with flying a jet translates into Oracle's culture. I don't get that same sense of machismo talking to you about flying.

Schmidt: It's helped me in the following sense. The difference between what I do normally every day and flying a jet is that when you are flying a jet people can actually die—right now. So while it's very hard for any actual human harm to come in this hour with you, when you fly a jet that's very possible.

And so in the jet training, and they do this whole thing called cockpit resource management, CRM, there's a whole protocol about how you handle life and death matters, what are the rules, and people pay attention and you would too obviously. I mean, it's a natural human thing. And by the way, people who don't pay attention don't pass, they literally fail them.

...

And the other rule, and this is particularly true in difficult training situations, is it's "What's next, what's next, what's next, what's next." Your eyes have to be constantly moving, and literally to the second.

And the problem, of course, is at the end of this you're very tired, but I think it's a reasonable metaphor for high tech management. The moment you let your eyes relax, there's something else coming. And by the way, it's okay for you to say you don't like that, but if you don't like it, then you can't succeed in it.

So we're always on. Everyone is on their e-mail 24 hours a day, you know, people work at midnight on a Saturday night, that kind of thing, and it's expected.

Me being... well... me, I find his analogy interesting and, of course, apt... except for one aspect, which I think is not only a false analogy, it turns out to be an extremely dangerous one.

[This is the part in the post where I'd normally go to a cut, but I think this is a relevant issue,² so I won't for this episode.]

In his answer, Schmidt accurately describes the pressures related to preparing for and executing proper Cockpit Resource Management during a flight; as he says, life is faster in a jet, but CRM is something even us lowly piston-pilots have to take into account on every flight³.

I think Schmidt does a great job of trying to convey the attention it takes to focus and fly with proper CRM; the way he describes it, it makes me believe he's referring to The Scan, which I can attest to from experience can be tiring from flying by for as little as half an hour⁴. Even flights on severe clear days, I've often come home and fallen right into bed.

Where his analogy falls apart is his conclusion: "So we're always on. Everyone is on their e-mail 24 hours a day, you know, people work at midnight on a Saturday night, that kind of thing, and it's expected."

Now, how many of you would want to get on a plane at midnight on a Saturday, flown by pilots who'd be up flying since 8 am?

Any takers?

Yeah, me either.

I obviously don't know under which rules Schmidt flies his jet, but commuter and "on-demand" operations—your standard chartered flights— are governed by part FAR Part 135, which specifically limits the amount of flight time a pilot can have in a year (1,400 hours), a quarter (500 hours), and two consecutive quarters (800 hours).

The regulations go into further detail about how much time a pilot must "rest" for before they can return to flying duty. FAR 135.267 has all the gory language for unscheduled 1-2 pilot crews.^5,6

This may seems common sensical, but what I find particularly interesting about the issue is there have been a number of recent⁷ rulings that require carriers to include "reserve duty" against the limits: "We have consistently stated that reserve duty is not rest when the reserve flight crew member must maintain accessibility (via telephone or pager/beeper) to the employer and there is a present responsibility to work."⁸

So describing our industry as "So we're always on ... and it's expected" is an awful analogy to being a (safe) pilot. If you interpret the analogy to be to "a pilot responsible for the safety of his 'airplane'⁹ and 'those on board'¹⁰, it's factually incorrect.

Another concept taught to pilots starting during the initial rating¹¹ is a concept called Aeronautical Decision Making.

It's the FAA's fancy way of describing the part of being a pilot that says "You know what? I only got three hours of sleep and I haven't had anything to eat in the last 8; maybe flying a plane isn't the best thing for me to be doing right now."¹²

Obviously, when hacking on code, the decisions (typically) aren't life-or-death. But the law of (quickly) diminishing returns still applies.

I don't buy the (implied?) supposition that "being on" at "midnight on a Saturday," and generally 24 hours a day, seven days a week, holidays included, makes you a better decision maker.

Or a better leader.¹³

I can't imagine it particularly enhances one's ability to execute.

Buuuuttt... I'm not the CEO of the most successful company this generation. I'm not a CanythingO... or, heck, even a businessperson for that matter.

I do know one thing, though: it certainly doesn't make you a better pilot.

_____________________
¹ Which I actually didn't know about Schmidt...
² Which I've pontificated on before.
³ So it turns out that you fall out the sky at the same 9.8 m/s whether you're in a Cessna or a Gulfstream...
⁴ I'm told, however, that like running, it's a skill you build a tolerance up for.
⁵ FAR 135.265 has (even stricter) dutytime/rest requirements for scheduled operations.
⁶These sorts of rules exist for air carrier operations as well, but of course they're covered by a different FAR, Section 121.471.
⁷ In the last decade.
⁸ Quoted from a 1999 Department of Transportation ruling on the matter.
⁹ "Company."
¹⁰ "Employees?" "Shareholders?"
¹¹ And reviewed heavily during most BFRs and safety seminars.
¹² A surprising number of pilots lack this little voice of preservation in their heads, and I think we all tell it to shut up from time to time, with varying implications on safety.
¹³ Confusing human beings for the baremetal in your data center seems... problematic to me.

Posted by preed at 5:13 PM | Permalink | Comments (3)

(Crossposted¹ to mozilla.dev.planning, mozilla.dev.builds, mozilla.dev.apps.camino, mozilla.dev.apps.calendar, mozilla.dev.apps.seamonkey, and mozilla.dev.apps.thunderbird).

rhelmer and I have recently been discussing the release process as it relates to tagging and retrieving source from the CVS repository for Firefox and Thunderbird releases.

The way we do it now has a lot of historical vestiges and the policy was crafted around some assumptions that aren't really valid for the project and the way Firefox and Thunderbird (and other projects in the Community) use the repository these days.

So we've come up with the following proposal we'd like feedback on.

The Proposal

When code complete is reached for the current Firefox release milestone, a release branch will be cut. The standard FIREFOX_version_RC1 and _RELEASE tags will be applied, but to a MOZILLA_datespec_RELBRANCH branch that will replace the venerable FIREFOX_version_MINIBRANCH. This _RELBRANCH tag will be applied to the same files the _RC1 and _RELEASE tags have traditionally been applied to.

If respins during the release are required, developers would check the fixes into the relevant (1.8/1.8.0) branches, as always; they could then either check the code into the _RELBRANCH themselves, or we can take care of that, whichever is easier. When the code for the candidate is ready, the next _RCn tag will be applied and the _RELEASE tag will be moved.

Rinse and repeat until the release is shipped.

Any equivalent Thunderbird release (if one is required) will come off of this branch as well. (This is why the suggested name is MOZILLA_datespec_RELBRANCH and not something like FIREFOX_version_BRANCH; products other than Firefox could come off the branch, and versions other than version could come off the branch.)

This branch will then die at the end of the release cycle.

This may be hard to visualize, so I took the liberty of adding it to our beloved branching diagram:²

The Reasoning

Currently, we apply _RCn and _RELEASE tags to the relevant development branch (i.e. MOZILLA_1_8_BRANCH) and then add _RCn tags and move the _RELEASE tag as we respin candidates. This process must be undertaken with extreme care, and while we record information when we perform the operations against the repository, because CVS does not version tag operations, that information is lost to external consumers of the source coming out of CVS (this is why we tag the _RCs individually; to track these changes).

Creating a release branch to isolate and record any changes required for a specific release is a long standing release engineering best practice. In the Mozilla Project's case, it will allow us to record the changes we make during a particular release cycle and isolate changes so that we are able to assert exactly what went into a release.

Additionally, it will make security firedrills significantly easier: the release branch can be revived at any point in time to release a fix to a particular security issue, so we can assert that a particularly release is the previous release with *only* the required security changes, an issue we've run in the past.

Finally, it will (admittedly, for me) simplify the release automation's respin support; the automation can now just track the _RELBRANCH, as opposed to attempting to fake branching by moving tags around on the regular development _BRANCHes.

The Impact

The impact to developers is quite low. In the worst case, those landing post-code complete fixes (fixes to a release candidate) will have to pull a branch to land their changes. In the best case (if a release engineer merges the change, which is TBD, but quite possible), then developers will not be impacted at all.

The impact to other projects relying on the Gecko milestone changes as well as any other consumer of Firefox-related source code should nonexistent; the _RCn and _RELEASE tags will, from an external perspective, still exist and pulling them will do the "expected thing." We have discussed, at some point, forgoing application of the _RELEASE tag until a particular _RC is declared to be the final release, but this would really only impact consumers of the source tarball who pick up the RC tarball.

In terms of the repository, it is the case that one extra tag per release will be applied to all the files; the _MINIBRANCH will be replaced by the _RELBRANCH, so for the four files that were tagged with the _MINIBRANCH, they will have the exact same number of tags.

The Upshot

We're planning on implementing this change for the upcoming Firefox 2.0.0.4 release, so please discuss and let us know if you have any questions or concerns.

______________________________
¹ Read: "spammed"
² Sorry in advance for the unwieldy size.^3
3 Click to enlarge!

Posted by preed at 5:49 PM | Permalink | Comments (6)

Late last year, as the Mozilla project began looking at the tools we'd need for the Mozilla 2 development effort, it became clear that trusty ol' CVS, while carrying the torch for so long, would not meet our requirements for ground-breaking development.

Mortal Kombat II, Version Control Edition: The Prologue

At the last Mozilla Summit, we all met in a session and were able to narrow down the choices to two contenders for consideration: Mercurial and Bazaar.

Brendan's merge requirements bring all the version control systems to the yard, and they're like "It's better than yours..."

We were able to narrow down the decision quickly because the type of development Mozilla 2 will require dictates a model that differs from CVS in ways that systems that attempt to emulate that working model, like SVN, would not work well for us. We turned our attention to the big names in open source distributed systems: Git, Mercurial, Bazaar, and Monotone.

While they've made recent progress, Git was lacking in Win32 support and it was unclear that this would ever change and if it did change, it was unclear that Git-on-Win32 would ever become something more than a second-class citizen. As good, performant Win32 (and Mac and Linux) is a hard-requirement, Git lost in early Kombat rounds. This is unfortunate because (as we would soon find out), lots of issues with the other systems did "just work" in Git.

Monotone was ruled out relatively early as well, due to the similar Win32 performance issues and not wanting to split developer resources with Monotone fix- and feature-requests.

At first, this left Mercurial standing in the ring. Ahh... at last, a simple Mozilla project decision.

But then, during the version control discussion at the Summit, Bazaar was brought up. It had a decent set of features that sounded interesting and useful to us.

We started investigating. And suddenly, there were two.

[Insert a quarter to continue...]

If that didn't cause some repository corruption, I don't know what would...

Mercurial had been a strong contender from the beginning: a distributed version control system that was fast on all platforms and, by initial accounts, handled importing the Mozilla CVS repository and with useful developer extensions, like mqueue.

But as Benjamin and I started to play with it more, we were unable to replicate an import of CVS. We both hit a number of hurdles, and it became a game of whack-a-mole: first it was encoding problems with commit messages. We'd hack around that (with a hidden environment variable setting), and then weird contortions in the CVS repository would cause breakage. "Ok," we thought; "those are branches we probably care less about; let's ignore them." Then, the parser barfed on commit messages that contained output that randomly looked like cvsps!

Mecurial was fast enough, when you had hopped over (and sometimes through) enough hurdles that you could get it to complete the requested operation.

It looked like handling larger repos wouldn't be a problem...

In parallel, we started looking thoroughly at Bazaar. As with Mercurial, we started the evaluation with an import attempt.

After working through some initial issues with the Bazaar developers, who were extremely helpful, we had an import going within about a week. In contrast to the other experiences we'd had, no matter what weirdness was in the CVS repo, Bazaar's parser and import logic just worked around it, with reasonable results.

That bad news is that the import took a long time.

And I mean a long time.

Processed 174786 patches (174786 new, 0 existing) on 743 branches (1 tags) in 2890557.7s (0.06 patch/s)

Yes, you're reading that right: on a dual-core, dual-CPU 2.8 GHz Pentium 4 with 4 gigabytes of memory, it took over a month of constant runtime to complete a trunk-only import of the CVS repository.

A month to import might be ok, if day-to-day operations were on par with Mercurial (or even CVS), but jst did some tests, and found out that, unfortunately, they weren't.

We did most of these tests with the bzr version available at the time, 0.14, and jst re-did them, multiple times upon feedback.

With some more help, we got the times on jst's test runs down to multiples of Mercurial's performance (as opposed to orders of magnitude). In the end, the third system succumbed to concerns about the first two: performance.

Astute readers of m.d.planning (or obsessive observers of the mozilla.org DNS zone file) may have noticed a new entry recently: hg.mozilla.org.

After a lively discussion, we decided to import the trunk CVS code using an epoch date (which turns out to be "22 Mar 2007 10:30 PDT", aka the MOZILLA_1_9_a3_RELEASE tag's pull date) as opposed to importing the whole history. cvs.mozilla.org will never go away, even if it's retired to the old-VCS's home and put into read-only mode, and the import, even of only the trunk, created repos with large initial pull times and space requirements.

The contents of the initial Mercurial import can be pulled from CVS by pulling the HG_REPO_INITIAL_IMPORT tag from mozilla/ in CVS (only the imported files were tagged, so client.mk-fu isn't required.

As those who spent a bunch of their time and energy involved in this decision can tell you, it wasn't an easy one. There were a lot of passionate discussions about features vs. performance tradeoffs, the priority of our requirements, and a desire need to put the rubber to the pavement on Mozilla 2 work.

It's important to realize the headline here isn't "Mozilla Project picks Mercurial for Next Generation Version Control System." It's "Mozilla Project moves to Next Generation Version Control System."

There was a lot of support in the project for both tools, and I personally know that the Bazaar developers spent a bunch of their 0.15 development time working on some of the performance issues we ran into. The great thing about these "nextgen" version control systems is that they all track the information necessary to re-create history. Because of this, switching between systems is much eaier, and in some cases, using your favorite system is possible (bzr, for instance, can pull directly from Mercurial).

Player II is up!

There's been serious talk of re-evaluating our needs and the state of these two systems in 9-12 months, after we have some concrete development, build, and tooling experience with Mozilla 2 and Mercurial. It's not a given that we'd make the same decision at that time.

One of the take-aways from this process is that all of these systems are relatively new (especially considering CVS started development in the mid 80s). None of them are perfect. They all have strengths and weaknesses and it's been interesting to see how they all solve problems, and to watch all of these systems progress in even the short amount of time we've been evaluating them.

Distributed version control is a new and strange world... but we're excited to be here.

Finally.

I'd like to give a special shout out to John Meinel of the Bazaar team for his help in this process; he spent a bunch of time working with us on import strategies, multiple, custom imports, and who loaned us their completed Bazaar import for testing purposes. We really appreciate you and your community's help.

Posted by preed at 11:55 AM | Permalink | Comments (21) | TrackBacks (1)