preed's blah-blah-blahg

Maybe I only notice these things because Gerv has so religiously run the "n00,000 Bugzilla bug"-contests...

Or maybe it was randomly running across Wil's post about all of the web team's commits the other day...

Or maybe... it's just because I'm a dork about these sorts of things...

... buuut I was amused to realize the other day that here at the nest, we're due to hit the 10k mark on both bugs (I just filed bug 9087) and commits (changeset 9200 just went in).

I'm betting we'll probably hit 10k on both before 0.6 ships... although, I find myself wondering who will make it across the milestone first. Even though Bugzilla is behind at this point, I'm betting it'll catch up to commits before we ship.

Anyone bets on when we'll hit 20k?

(I was going to ask about the next-order-of-magnitude milestone, but... a Wilson party is probably a couple years away... at least.)

Posted by preed at 11:56 AM | Permalink | Comments (1) | TrackBacks (0)

Almost four weeks ago now¹, dbaron took some time and separated out the various guts of the old Mozilla tools module, and divvied the contents of the directory² into two new Mozilla modules: a Build & Release Tools module and a Code Analysis and Debugging Tools module.

I wanted to call this change out, because I think it's a step in the right direction, for a lot of reasons; both of these new modules have seen a lot more activity in the last 2-3 years and I think it really helps to separate them out from a code management standpoint, but also from the standpoint of the way we think about these sets of tools: the guts of any organization's build/release infrastructure and analysis/debugging tools should be first-rate citizens, and this change helps to frame the way those parts of the code are thought about.

So, thank you, dbaron; it's been a long time coming, and I appreciate you taking the time to sort that goop of code all out into [more] logical units.

This obviously makes little-to-no sense.

After a lively discussion about how to best remedy that, I also wanted to note that long-time Mozilla community member Nick Thomas³ is the new module owner for the Build/Release Tools module, with Chase, Rob Helmer, and myself staying around as peers.

Coop was also added as a module peer, whose omission was, let's face it, somewhat ludicrous⁴. I'm happy to see that got corrected as well.

Congratulations, Nick; you follow a long line of Mozilla Build Engineers, and I know you're going to [continue to] hit it out of the ballpark.

Let me know where to send the bottle of scotch.

_____________________
¹ Wow... on my birthday, no less! What a present! :-)
² which had mostly become an "island of misfit code"
³ also known as the "Build Engineer Formerly Known as CF"
⁴ No one escapes Mozilla Build/Release, Coop. No one. ;-)

Posted by preed at 7:48 PM | Permalink | Comments (2) | TrackBacks (0)

There is currently an interesting discussion¹ going on right now in mozilla.dev.platform which asks a seemingly simple—but if the number of responses are any indication, actually difficult—question: "Is 'Gecko' 1.9 only Firefox?"

It's definitely required reading if you hack on anything built on the Mozilla platform that isn't Firefox.

To be honest, I find it a little surprising that others find it surprising that the answer is anything but "Yes."

There are a lot of principles discussed in the newsgroup posting, but in practice, ever since I was involved with Firefox releases, it is Firefox—not Thunderbird, not SeaMonkey, not Calendar, and not (arguably most appropriately) XULRunner— that has driven Gecko releases. The best determining factor of Gecko releases, i.e. when the Gecko platform version is bumped, was originally standardized as part of the Firefox release process, and is now codified by the release automation.²

I fully admit that I am interpreting the question from the (possibly too narrow) standpoint that I'm mostly familiar with: the Firefox release process. It is entirely possible that the triage process is different.³ But it has the benefit of being entirely devoid of a process⁴ with lots of human input and interaction which necessarily implies non-deterministic outcomes.

But the answer to "When is Gecko released?" is pretty binary. Since there are no separate XULRunner releases, just look at when the Gecko versions are bumped: for quite some time, it's been at Firefox release time.

Immaterial of how we want to do things or would like to do them in a utopian world, that's what the code in CVS does today, right now. It would seem the requirements and triage process mimic the code, namely in that Firefox drives the process.

Almost ten months ago now, it was stated pretty clearly that "XULRunner improvements should benefit a range of other applications but the focus of Mozilla employees will be on browsing." There was a lot of discussion about what the material impact of the policy statement would be, including my own two cents on the matter.

I may be misunderstanding the posit of the original question posed in the newsgroup, but it would seem that this is an concrete, identifiable example of a question directly arises from a policy of having a singular product and its release schedule drive the development of the platform.

If this isn't the outcome developers on Mozilla's non-Firefox projects expected, then it may be time to ask "where was the disconnect between what was said and what was heard?"

For me, the discussion raises some meta-questions that, to my mind, are not only more important than "Is 'Gecko' 1.9 only Firefox?" but whose answers also turn out to be more relevant than just what happens in the next N months, as lots of people work really hard to get an awesome Firefox 3.0 out the door:

• Is there going to be a separate XULRunner 1.9.0 release before or during⁵ the Firefox 3.0 release time frame? If so, how are triage decisions for that release (being) made? Does the process include stakeholders for projects that aren't-Firefox?
• (I admit that this may be a totally stupid question on my part, but it's totally unclear to me:) who is the final determiner of the answer to the above question? There's a lot of really good discussion going on in the newsgroups, but who makes the final call?

  Everyone tells me drivers@ and staff@ are "gone," and have been for quite some time. Does this mean the Firefox 3 release drivers⁶ will decide what a "Gecko 1.9" is? The Steering Committee? A vote of everyone who's posted more than five times in mozilla.dev.planning?



• If that answer is the Firefox 3 release drivers or the Steering Committee, as an open source project, is it relevant to ask if any of the lessons learned in previous days—when the Mozilla Project was largely dependent on a single company, i.e. AOL/Netscape, for engineering work—are applicable?⁷

Distinct from any discussion of the questions above, I stand ready, as I did ten months ago, to help in any way that I can. Like most people, I use and adore Firefox and agree with the argument that its Mozilla Corporation's best lever to move the (Open) Web forward.

But I also use and love Thunderbird. I have used⁹, and relied upon Calendar. And I believe that the XULRunner platform is the best lever the Mozilla Project has to future-proof against whatever develops on the Internet¹⁰, as well as provide a compelling answer to proprietary rich Internet application platforms from you-know-who.

Probably the most useful answer is the one to the question: tell me, an engineer with admittedly limited experience with the Mozilla build system, Tinderbox, and the Mozilla release process, how I can help the answer to the original question be a demonstrable, unequivocal "No."

_______________________
¹ Which was brought to my attention by a Rumbling Thunder post
² rhelmer implemented the first version; I added the respin logic
³ Although, blocking1.9-flag triage was what prompted the question originally, so who knows
⁴ Which Brendan aptly describes as "fluid"
⁵ "after Firefox" is a possibility to, but arguably of limited utility
⁶ Is this a complete/up-to-date list? A couple of blocking1.9 flags have been set by people absent from the list.
⁷ The question is posed merely by the fact that those two groups are made up of entirely⁸ Mozilla Corporation employees. I'm not implying any judgment regarding people who work on those groups; I'm merely pattern matching their relational structure to the Project from previous eras.
⁸ or nearly entirely; the FFx3 drivers list has one non-MoCo-er on it, that I can find anyway
⁹ and with the number of meetings I'm now responsible for, need to set up again
¹⁰ Who knows what we'll all be using in ten years. Gopher could make a comeback!

Posted by preed at 9:31 AM | Permalink | Comments (13) | TrackBacks (0)

For quite some time, we've been wanting to reorganize how bits and builds find their way out to the world.

It's a project we've affectionately referred to as "stage migration," and we're finally at a point where we're ready to make changes that will hopefully improve security and auditability, reliability, and expandability.

The primary goals of this move are to support:

• Ensuring that builds get scanned before they're available for download
• Audit access to the FTP tree and move to LDAP authentication for easier management
• Move to expandable storage, backed by a highly-reliable storage platform
• Combine ftp.m.o and archive.m.o, to make it easier to find builds

To address these goals, this week, the Build & IT teams will be reconfiguring the way the servers that make up ftp.m.o are set up.

The main changes that you'll notice are:

• archive.m.o and ftp.m.o will be merged into one site, which will respond to both addresses. (So, no need to switch between them if you're regression testing!)
• Directories with many builds, like firefox/nightly/, will be rearranged to allow much quicker access than the current archive.m.o. We'll be creating YYYY/MM subdirectories, and moving deliverables into those directories. For example, the builds from March of last year would be found in firefox/nightly/2006/03/, and in that directory, you would find directories with builds like 2006-03-01-04-mozilla1.8 and 2006-03-01-04-trunk. We will be doing this for all builds older than a month.
• All files will be virus scanned before becoming available. We currently virus scan all builds, but depending on a number of factors, it was possible for unscanned-builds to appear on the FTP site for a window of time; we've removed this window.
• Interactive shell accounts on the FTP farm will be replaced with sftp-only accounts.

We will be doing most of this work during an outage window starting on Thursday, 27 September at 10 pm to 2 am. The tree will be closed during this time, because we'll be teaching Tinderbox about the new FTP structure. Updates might be wonky on Friday, as well.

We will be moving the archived builds around earlier in the week, on Wednesday during the European day ("Go, go, gadget cf!" :-)

We're doing our best to minimize adverse changes but because of the above, it's possible that your account could become disabled, or data in your home directory on the current FTP machine becomes unavailable. If that happens, please email build@mozilla.org, and let us know, so we can fix it.

Posted by preed at 11:00 AM | Permalink | Comments (8)

With last week's release of Thunderbird 2.0.0.6, we've finished a stint of security fire drills.

My friends and family have come to learn what that means¹, but I often get asked what it feels like to be in firedrill mode, especially on a [consumer-oriented-yet-open-source] project like Firefox. I've been trying to come up with a good way to explain it, and I think I may have finally found something to help.

The idea came to me when I noticed that a scene from a particular movie kept flashing through my head, very distinctly and vividly, while working on various release activities during this last drill.

It's from a 70's movie called The China Syndrome², which I found a conveniently-You-Tubed³ clip of. You'll want to let the full clip load, since the part that flashes through my head most commonly in the last thirty seconds or so, starting at about 3:35 in.

After watching this, a friend asked "So... you're comparing firedrill-mode to a nuclear meltdown?"⁴ We shared a laugh about that, because it wasn't the analogy I was intending at all.⁵ What the scene does capture pretty perfectly is the sense of urgency and the mode the characters go into when "the alarms go off."

During a firedrill, there's bits and pieces of information coming in from all sources and angles; there's a level of coordination involved in terms of getting issues looked at and prepping for builds; there are usually always external people watching [from above] what we're doing and how we're doing it; sometimes, we have to address problems creatively, even though "the book says you can't do it"; there can often be conflicting information that makes making a decision difficult. Many of the roles, reactions, and interactions of the characters in the scene are reminiscent of how we get a firedrill out the door.

I actually hadn't noticed that until I watched the clip again, since I originally identified with the last thirty seconds⁶ and hence why it tended to flash through my head at weird hours while starting at Tinderboxen prompts.

Every component, everyone one of them, is tested, and then it's tested again. Every critical weld is radiographed. Every single thing is checked, double-checked, it's rechecked. Everything. ... In any thing that man ever does, there's some element of risk, right? That's why we have what we call "defense in depth." Now that means, backup systems to backup systems. To. Backup. Systems. ... The system works. Even with a faulty relay, even with a stuck valve, that system works.

I'm sure QA can certainly relate to the amount of testing we do to ensure we ship a quality product... and a number of those tests mesh with verification and testing we do on the release side. In our case, it's more "even with an invalid automatic update snippet, even with a bogus locale repack, we catch the error and correct them before the release."

We learn something new from every firedrill we do.

And as a result, we've got a lot of newer initiatives all over the project to make firedrills more smoother and faster, from automated regression testing in Tinderboxen to automated smoketests and BFTs to automated release deliverable verification. These are all going to improve on Firefox 2's already-pretty-awesome patch-delivery time.

The analogy I make here illustrates how stressful it can be for that time we're getting it out. All the stuff I mentioned above will reduce that stress, but I think at least some of it is self-induced⁸: we take this stuff and the way we respond to and engage in them very seriously, and I know I'm not the only one: everyone, from the developers focusing on the fix to the release managers coordinating everything to the QA engineers verifying the fix and looking for regressions, to the build engineers pushing the bits out experience this responsibility together. (In fact, as I watch this clip again, I identify with Lemmon's character at the end, but in the beginning of the clip, when he gets up from his desk as the alarms initially go off, an image of dveditz pops into my head for some reason... :-)

It's part of the commitment to security and, more generally, doing right by our users that not only defines Mozilla and our community, but that I know we all personally hold dear.

So maybe no matter how good we get at this, there will always be little alarm klaxons going off in my head when I hear the callemail come in.

But... maybe... just a little bit of that... is a healthy thing.

_______________________
¹ "Preed's gonna 'go missing' for awhile... don't ask..."
² If you haven't seen The China Syndrome, I highly recommend it... if
for no other reason than getting to see a young pre-activist Jane Fonda and 70s-haired Michael Douglas. And of course, any movie with Jack Lemmon is A++++ WOULD WATCH AGAIN.
³ Oh how I love thee, copyright infringement fair use!
⁴ I don't even want to think about the Slashdot headliner on that one...
⁵ Maybe I didn't make the connection because—not to spoil the movie if you haven't seen it—but there is no meltdown in the movie
⁶ We've striven for correctness on the common cases with the release automation, and in firedrill-mode, we can be faced with situations that aren't entirely common... so there's been some level of back-to-non-automated/manual release work. Performing certain staps manually has resulted in (somewhat familiar) typos, errors, and general suboptimality⁷ that you generally expect with a human—me—typing lots of commands in a magical, ordered incantation. When that happens, and it gets caught by QA, my reaction is... well... it's why I keep seeing this scene in my head. :-)
⁷ Is that even a word?
⁸ At least, I know for me it is... and yes, it's a bad habit. :-)

Posted by preed at 5:04 PM | Permalink | Comments (3)

Some people have just not learned the lesson that it's not a great idea to bet against a build engineer.

Philor is one of those people:

22:43 <preed> philor: I'll take my $5 please.
22:43 <philor> noooo! something else exists?
22:43 <preed> see the update in the bug.
22:43 <preed> now
22:44 <preed> as your bookie... I'm hoping we can solve this nice and quietly
22:44 <philor> I thought that was a sure thing, with its next-door neighbor in the waterfall hanging there just yesterday
22:44 <preed> don't make me send someone to break your legs.
22:44 <philor> preed: it's in the mail
22:44 <preed> kthx

And just look what showed up on my desk this morning:

Small, unmarked bills.

Glad we could take care of this all quiet-like...

Posted by preed at 2:50 PM | Permalink | Comments (3)

While doing a Bay Tour this weekend, I had an interesting first-time-aviation-experience (tm).

First-time-aviation-experiences (tm) tend to cause much after-the-fact pondering, because they typically turn out one of two ways:

a. Everything goes okay, you make it back to the earth, and you have a war story to tell in the hanger next time when it's IFR and rainy and everyone's having a beer instead of flying.

2. It... uhhh... doesn't go so okay... and for some value of "doesn't matter anymore," it... well... doesn't matter anymore.

I've had a couple of "first-time-aviation" experiences (mostly as a student pilot) that [I now sorta laugh at, but] at the time, they were a big deal.

While on the tour this weekend, we hit some pretty bad turbulence around Mount San Bruno, near San Francisco International. It was so bad, in fact, that we dropped about 400 feet in ten seconds¹. Arguably², I did precisely the right thing, and sacrificed altitude for attitude control of the aircraft, but at the expense of... comfortable flying.

It turned out to be a problem for one of my passengers.³

In retrospect, while the experience wasn't really all that big of a deal^4,5, it did present a new situation which I haven't had to deal with yet: "How do I get this plane on the ground, as soon as possible, in crowded airspace?" It turns out the exercise is useful for dealing with "other conditions"⁶ which pilots must be prepared at anytime to deal with.

To narrate this experience, I isolated⁷ an air traffic control clip from Saturday's flight; it's about three minutes long, and you can hear us—Cessna seven-five-one-sierra-papa—talking to San Francisco Tower and NORCAL Departure⁸:

2:40 96 kbps MP3.

The relevant portion starts about two minutes into the clip:

Me: Uhh, NORCAL, Cessna seven-five-one-sierra-papa, we need uhh... to do uhh... landing in Oakland, uhm so we're turning towards Oakland, we don't have the numbers.

NORCAL Departure: Cessna seven-five-one-sierra-papa, okay, is everything okay?

Me: Uhh, yah, we just uhh... have someone who's not feeling great, so we're gonna land in Oakland real quick before going back to Palo Alto.

NORCAL Departure: Okay, let me uhh... see if I can get direct for you. Air Canada five-eighty-one, fly heading three-four-zero, proceed direct Red Bluff when able.

Air Canada 581: Heading three-four-zero, Red Bluff when able, Air Canada five-eighty-one.

NORCAL Departure: seven-five-one-sierra-papa, you may proceed direct, altitude your discretion, contact Oakland Tower on one-one-eight-point-three, have a good day.

Me: Thanks for your help, direct Oakland, eighteen-three for the Tower, seven-five-one-sierra-papa.

What's interesting about this interaction⁹ is that the air traffic controller basically cleared us directly under Oakland Airport's departure path, so that we could get on the ground sooner (about half the time it would normally take). It required a measurable amount of extra work on her part to call Oakland Tower on the telephone and request—and receive—a clearance for us to fly directly to the airport. When we got there, we were put ahead of the line to land.

This is one element of our air traffic control system that just always amazes me and gives me warm fuzzies at the same time: you can be a two-seater 152, but the second you declare an emergency, they'll push 747s full of people out of your way if you need them to.

I've heard it happen.

In many ways, it reminds me of one of the most important aspects, in my opinion, of the Mozilla Community which is so hard to convey to others, but which makes being part of and involved in our Community so rewarding.

________________________________
¹ Do the math on the acceleration.
² Justin is the only one arguing with me... ;-)
³ It should be made clear: the passenger was apologetic to a fault which, while endearing, was unnecessary. I tried to reassure them by recounting the story of life as a student pilot, where I continuously got sick every time we tried to do "ground reference maneuvers."
⁴ The failure condition, while somewhat... uhh... disgusting... wasn't life-threatening, really.
⁵ But I'm sure it was a big deal for the passenger in the back seat
⁶ ENGINE FAILURE
⁷ Which is a fancy way of saying I removed all the other stuff so it looks like navigating San Francisco Airport's airspace is really friggin' easy.
⁸ For some reason, all my friends comment that I sound "butch"—their words, not mine— on the radio. My instructor claimed I sound like a "grizzled Eastern Airlines captain." I really don't see it.
⁹ I've italicized the relevant statements.

Posted by preed at 11:00 PM | Permalink | Comments (1)

The The Nomadic Build Engineer '07 Tour continues in the Big Apple, the always open land of real pizza, subway churro stands, and a Mass Transit System That Is Actually Usable (tm).

My NYLUG presentation went... ok, I think.

Everyone I talked to said it went great, but I'm not as sure. When I mentioned that I enjoy questions during the presentation, it seemed like we got lost on a half-hour tangent about Webkit on Windows, why Firefox doesn't work on a decade's old install of Slackware, why the "update problem" really is hard, and why we do some of the things we do (that may not always make sense to super technically-minded users). Then I had to sorta rush through the last third of the presentation to catch up.

But all-in-all, I think it was useful and informative.

(In fact, one of the NYLUG leaders randomly just told me: "I have a definitive answer about your presentation. You used the word fuck, which according to our founder, all the best presentations contain the word at least once." Well... f'ing awesome!)

The drivel presentation is available online, and there's supposedly a Podcast of it floating around (cheers to whomever recorded it, despite the fact that I'd rather ignore it exists, because hearing the sound of your own voice... ... yah).

For those in a rush, my hour-long talk can be summarized with the following slide:

The Firefox release process in six easy steps.

Obviously, this process relies heavily on the Internets' Tubes.

(Which aren't big trucks.)

Anyway, I had a lot fun, and I hope NYLUG did too. Thanks for the invite!

We stopped by Atlantic City (and got to see a major thunderstorm-induced traffic jam at KEWR!) last night, and I won $7! Which is more than I can say for Vegas.

Then, more yummy East Coast food and city-explorin'.

Posted by preed at 12:50 PM | Permalink | Comments (5)

If you've been wandering around the Mountain View office, wondering where I am, I've just finished wek one of the Nomadic Build Engineer '07 Tour.

First stop for the tour bus: Toronto!

While soaking up that world-renowned Canadian hospitality, highlights included:

I only took like three pictures in Toronto, and this was one of them. It's a statue of Winston Churchill in [[Nathan Philips Square]]; as you may be able to tell, the pigeons have a particular fondness for Churchill['s head], but frankly, haven't we all felt a little like that about Churchill['s head] at at least one point in our lives?

• A few conversations with mconnor about build-related needs for Firefox 3, so there are little-to-no surprises.
• Learning what it's like to be remote to a bunch of meetings.¹
• A meetup with Ben Hearsum, Buildbot-er extraordinaire. Chatting with him was a double-header of release automation fun-ness³ and a review of someTinderbox code. Once we got the bad taste out of our mouths, we started brainstorming requirements for a build harness that might eventually replace Tinderbox Client. Ben then filed bug 384943 today, which rhelmer, cf, and I all noticed pretty much in unison.⁴
• My first encounter with [[poutine]]; I don't think I'll ever be the same again, but my arteries thank you, Gavin and Johnath!
• My first encounter with Beer Bistro; I don't think I'll ever be the same again, but my liver thanks you, Gavin and Johnath!
• A trip to the ballet, courtesy of Lucy; they were performing Don Quixote⁵; it was a dress rehearsal, so every once in awhile the director would stop everything, give a bunch of notes, and then they'd re-do it. It was great! Like Tivo-for-ballet!
• Riding the Toronto subway.
• Walking up and down Church St. with Lucy at 2 in the morning.
• Cursing Beltzner for listening to his wiki-suggestion to stay in the downtown Days Inn, who has seemingly confused "beds" and "layers of cardboard with sheets."⁶
• A stop at Active Surplus, which is kinda like a Canadian version of Weird Stuff. Johnath has promised to build me a control panel with those old switches from the Cold War movies that have the protection cover over them to launch the missiles. "Firefox 2.0.0.5 released... now!"⁷
• While walking down the street, I asked Johnath: "Does Canada have any assholes? You're all so nice here," and of course, on cue, someone guy started laying on his horn constantly as we walked by. So I learned that, despite my earlier perceptions, Canada does indeed, have at least one asshole. But for a country of thirty million people, that's pretty good.
• Even though they are Canadian, Air Canada is still like every other airline. They just say "We're going to screw you, eh?" and thank you for the opportunity to do so when they're done.

A special shout-out to the Toronto office for taking me in, and dealing with my [now extra] screwed up sleep schedule. I had a lot of fun!⁸

The Toronto Subway was fast, clean (of course!), and easy to understand. A+++++!!!!! WOULD TOTALLY RIDE AGAIN!!!

But more than that, it's been one of the most productive trips I've been on in terms of strategic planning (hey, imagine doing that in Build-land, for once!)

Week two has me in New York City, meeting up with Ted Mezerk... Ted Mylizkriy... Ted Mcyzkiske... luser to work on Build Things for the next alpha.

Shameless self-promotion. And on my own blog, too!

Also, on Wednesday evening, the New York Linux Users' Group, one of the largest LUGs in the country is hosting me for a talk about releasing Firefox to millions of users and the unique challenges that presents how I drink scotch all day and the Mozilla community makes me look good.

The talk is Wednesday evening, but you have to RSVP by Tuesday afternoon, so if you're interested, head on over to the website.

Should be interesting (babbling), if nothing else.

14:53 <@preed> cf: so, I'm in NYC right... and I'm right next to little Italy
14:53 <@preed> so I've been enjoying the Italian food
14:54 <cf> nice
14:54 <@preed> but like... they have chianti with EVERYTHING
14:54 <@preed> "Uhm, no... I didn't order any wine with lunch... well... ok."
14:54 <@preed> "(Twist my arm.)"
14:54 <cf> twist it hard
14:55 <cf> makes afternoons kinda fun
14:55 <@preed> it sure as hell makes them go faster! ;-)

______________
¹ BRING THE MIC CLOSER! So that I might... join with it. Do it. NOW!^2
2 Sorry... sorry... awful Star Trek V reference.
³ That will likely involve some BuildBot subclassing hotness
⁴ And I don't mind telling you that it made us all a bit tingly in our special places.
⁵ Am I the only one who kept pronouncing it "Don Kwix-oat" until I was like 16?
⁶ I still <3 you, though, Beltzner; especially since you're still reviewing my presentation? Right?
⁷ Of course, QA would have the other key, and we'd have to turn them together.
⁸ So much, in fact, that I've started thinking about what it would take to work from the Toronto office for like... a year. I really like the feel of this city, and I've never done anything spontaneous like that. Of course, this is one of my [many] harebrained ideas, so it's unlikely to go anywhere... but you never know. This one may stick!

Posted by preed at 2:12 PM | Permalink | Comments (1)

I swear... the amorphous blob of people, patches, and passion that is Mozilla never stops churning.

I leave for a long-weekend-mini-vacation¹, and the world completely changes: some taggable, hour-long YouTube video has everyone talking², my desk gets moved³, a friend gets engaged, and Mozilla is killing XUL Runner.

Except... we aren't. Not really.

Looking over Planet, there's a lot of "platform" discussions going on. I've seen Mark Pilgrim's "silly season" post referenced a number of times. To be sure, it's a great post (and relevant, too), but one Mark's other posts, linked from "silly season" really resonated with me: Freedom-0.

In that post, Pilgrim writes:

WordPress is Free Software. Its rules will never change. In the event that the WordPress community disbands and development stops, a new community can form around the orphaned code. It's happened once already. In the extremely unlikely event that every single contributor (including every contributor to the original b2) agrees to relicense the code under a more restrictive license, I can still fork the current GPL-licensed code and start a new community around it. There is always a path forward. There are no dead ends.

I could go on about how I was confused, because it didn't seem like I was reading the same post everyone else was reading.

But with shaver and Benjamin's clarifications today, I don't think I need to say anymore.⁴

I will, however, say: that's what's wonderful about the freedom of open source: the fact that those interested in a project can build up a community out of interested developers, can build a project out of a CVS repository/IRC server/mailing lists, and can build a product out of a bunch of bits.

I personally have always found XULRunner to be a very interesting project. I always thought it was neat that you could download this little bundle of binary bits, write up some JS and XUL, connect it with some XPCOM components, and get an entirely new app out of it all. And all the stuff we expect out of current (non-browser) applications—HTML rendering, network connectivity, platform independent I/O, an update mechanism—were just there.

I just wanted to add: count me among the members of the Mozilla Community that are interested in XULRunner and seeing it succeed.⁶ Materially, I'd love to contribute the time to make sure that XULRunner has the build (and eventually release) resources it needs.

As shaver notes, "As XULRunner itself is about 1000 lines of code that differs from Firefox, the bulk of the work is likely to be in build, packaging, update and other "meta" areas, I suspect." I suspect he's right, most certainly because Firefox really is built on top of XULRunner, even if that separation isn't as clear as we like, and it's not packaged entirely how we'd like. It's not like we could have Firefox release, but a so-completely-broken XULRunner build, that XR was unusable.

(I was going to run out and start setting up trunk Tinderboxen for XULRunner but then I noticed "Oh right... we already have those.")

So I guess others interested in XULRunner in the XULRunner Community will have to tell me how, as an individual contributor, I can best and most effectively be of assistance.

______________________
¹ I had the best two hour massage...
² I say this not to sound dismissive; however in the interest of full disclosure, I must admit that I started drinking scotch about fifteen minutes in, and the more I drank, the more it all made sense.
³ Admittedly, to a nicer location
⁴ Except to point out that we've been faced with this [perceived?] specter before, and not only did the "nightmare scenarios" never come to pass, even if they had, it wouldn't have mattered.^5
5 Ok, I'm done. Really.
⁶ Whatever we decide that means.

Posted by preed at 10:32 PM | Permalink | Comments (2)

Gen linked me to a recently posted article detailing an old interview from November, 2005 with Google's Eric Schmidt.

It was a great read, but Gen brought it to my attention for a specific reason¹:

Wired's Vogelstein: You've talked before about how flying your own jet helps you manage. Explain. The reason I'm intrigued by it is because the only other guy I know who flies a jet is (Oracle boss) Larry Ellison. But you can look at Larry Ellison and understand how the machismo associated with flying a jet translates into Oracle's culture. I don't get that same sense of machismo talking to you about flying.

Schmidt: It's helped me in the following sense. The difference between what I do normally every day and flying a jet is that when you are flying a jet people can actually die—right now. So while it's very hard for any actual human harm to come in this hour with you, when you fly a jet that's very possible.

And so in the jet training, and they do this whole thing called cockpit resource management, CRM, there's a whole protocol about how you handle life and death matters, what are the rules, and people pay attention and you would too obviously. I mean, it's a natural human thing. And by the way, people who don't pay attention don't pass, they literally fail them.

...

And the other rule, and this is particularly true in difficult training situations, is it's "What's next, what's next, what's next, what's next." Your eyes have to be constantly moving, and literally to the second.

And the problem, of course, is at the end of this you're very tired, but I think it's a reasonable metaphor for high tech management. The moment you let your eyes relax, there's something else coming. And by the way, it's okay for you to say you don't like that, but if you don't like it, then you can't succeed in it.

So we're always on. Everyone is on their e-mail 24 hours a day, you know, people work at midnight on a Saturday night, that kind of thing, and it's expected.

Me being... well... me, I find his analogy interesting and, of course, apt... except for one aspect, which I think is not only a false analogy, it turns out to be an extremely dangerous one.

[This is the part in the post where I'd normally go to a cut, but I think this is a relevant issue,² so I won't for this episode.]

In his answer, Schmidt accurately describes the pressures related to preparing for and executing proper Cockpit Resource Management during a flight; as he says, life is faster in a jet, but CRM is something even us lowly piston-pilots have to take into account on every flight³.

I think Schmidt does a great job of trying to convey the attention it takes to focus and fly with proper CRM; the way he describes it, it makes me believe he's referring to The Scan, which I can attest to from experience can be tiring from flying by for as little as half an hour⁴. Even flights on severe clear days, I've often come home and fallen right into bed.

Where his analogy falls apart is his conclusion: "So we're always on. Everyone is on their e-mail 24 hours a day, you know, people work at midnight on a Saturday night, that kind of thing, and it's expected."

Now, how many of you would want to get on a plane at midnight on a Saturday, flown by pilots who'd be up flying since 8 am?

Any takers?

Yeah, me either.

I obviously don't know under which rules Schmidt flies his jet, but commuter and "on-demand" operations—your standard chartered flights— are governed by part FAR Part 135, which specifically limits the amount of flight time a pilot can have in a year (1,400 hours), a quarter (500 hours), and two consecutive quarters (800 hours).

The regulations go into further detail about how much time a pilot must "rest" for before they can return to flying duty. FAR 135.267 has all the gory language for unscheduled 1-2 pilot crews.^5,6

This may seems common sensical, but what I find particularly interesting about the issue is there have been a number of recent⁷ rulings that require carriers to include "reserve duty" against the limits: "We have consistently stated that reserve duty is not rest when the reserve flight crew member must maintain accessibility (via telephone or pager/beeper) to the employer and there is a present responsibility to work."⁸

So describing our industry as "So we're always on ... and it's expected" is an awful analogy to being a (safe) pilot. If you interpret the analogy to be to "a pilot responsible for the safety of his 'airplane'⁹ and 'those on board'¹⁰, it's factually incorrect.

Another concept taught to pilots starting during the initial rating¹¹ is a concept called Aeronautical Decision Making.

It's the FAA's fancy way of describing the part of being a pilot that says "You know what? I only got three hours of sleep and I haven't had anything to eat in the last 8; maybe flying a plane isn't the best thing for me to be doing right now."¹²

Obviously, when hacking on code, the decisions (typically) aren't life-or-death. But the law of (quickly) diminishing returns still applies.

I don't buy the (implied?) supposition that "being on" at "midnight on a Saturday," and generally 24 hours a day, seven days a week, holidays included, makes you a better decision maker.

Or a better leader.¹³

I can't imagine it particularly enhances one's ability to execute.

Buuuuttt... I'm not the CEO of the most successful company this generation. I'm not a CanythingO... or, heck, even a businessperson for that matter.

I do know one thing, though: it certainly doesn't make you a better pilot.

_____________________
¹ Which I actually didn't know about Schmidt...
² Which I've pontificated on before.
³ So it turns out that you fall out the sky at the same 9.8 m/s whether you're in a Cessna or a Gulfstream...
⁴ I'm told, however, that like running, it's a skill you build a tolerance up for.
⁵ FAR 135.265 has (even stricter) dutytime/rest requirements for scheduled operations.
⁶These sorts of rules exist for air carrier operations as well, but of course they're covered by a different FAR, Section 121.471.
⁷ In the last decade.
⁸ Quoted from a 1999 Department of Transportation ruling on the matter.
⁹ "Company."
¹⁰ "Employees?" "Shareholders?"
¹¹ And reviewed heavily during most BFRs and safety seminars.
¹² A surprising number of pilots lack this little voice of preservation in their heads, and I think we all tell it to shut up from time to time, with varying implications on safety.
¹³ Confusing human beings for the baremetal in your data center seems... problematic to me.

Posted by preed at 5:13 PM | Permalink | Comments (3)

I made a post last week, ostensibly about my experiences with Google Wifi. It was actually about much more.

It had an umlaut in the title, so I wasn't entirely surprised when it didn't show up on Planet.m.o, thinking maybe foreign languages caused inter-Planetary indigestion.

After some debugging, we found that Planet's parser was indeed crashing, but not on my blog.

I filed a bug to track the issue and within minutes—go go Gadget Mozilla Community—an... interesting configuration change was pointed out.

This change, which turned out to be the reason my post didn't make its way onto Planet, raises a couple of important issues which, from what I can tell, haven't really been raised or addressed heretofore:

The first is how editorial changes are made on planet.m.o, and how those changes get communicated, not only to those publishing to the feed, but to those reading it.

In my case, I was not notified that Planet's seemingly de facto (and sole) editor, Tim Rowley, had added the filter.¹ I'm not entirely sure how planet filtering works,² so adding filters to my blog without telling me effectively removed my blog from Planet's feed, because there's no way I would know which "magical keywords" I needed to use to get my content onto Planet now.

In looking at the (admittedly short) planet.m.o config history, it looks like I'm not the only one who's been silently censored: numerous people have been completely removed from Planet, without being given any notification or even a detailed explanation.³

In the bug, others have noticed community members' blogs missing from the Planet feed, which begs the question: just how many blogs were removed because a single person decided the content contained in those blogs wasn't what he (or his office⁴) was interested in reading?

It leaves me wondering "how much content, Mozilla-related, or otherwise am I missing from members of our community?"

The second obvious issue this raises is "What content is appropriate for Planet.m.o," and, slightly related, "How do we enforce that content policy?"

Obviously, this is a fairly gray line, with lots of opinions on the subject.

For myself, I write about a myriad of subjects, which tend to fall into "very Mozilla-related," "technology/web-industry related," and "not-at-all Mozilla-related."

Personally, I enjoy reading content on Planet that isn't necessarily (or maybe not directly) Mozilla-related; I see Planet as a community resource that illustrates the Community—what its working on, what its interested in and worried about, its trials and successes, how it relieves stress, what big changes are happening in people's lives that may affect the community—not merely the browser-and-related-projects and the code-that-goes-in-one-end and the bits-that-come-out-the-other.

Surely we could reduce Planet.m.o to a listing of fixed bugs and posts about what features we're working on and when releases are available. But doing so would remove the human face of our work, sucking the humanity out of the Mozilla Project. A feed that lacks these elements would serve to imply that we're all just a bunch of code-robots, hacking 24 hours a day, who don't [need to] have families or hobbies, who don't [need to] require sleep, and who don't need to be treated like we have lives outside of our work on Mozilla.

Not only would that be a tragic and depressing face to put on our project, it would be an insulting one.

I understand that not everyone may agree with me, though. For my own part, I have often written about topics not particularly Mozilla-related. I've continued to write about these because I've received numerous comments on IRC and in person saying they really liked a particular post. Sometimes it's been about why Digg and Slashdot users screw us. Sometimes it's been about flying.⁶

To be clear, I'm not saying that a more open policy of content on a space like planet.m.o does not require some consideration and respect for the forum their content is being featured in. I take special care to my non-Mozilla related posts in the "extended post" section, making it easy to skip over if the first paragraph doesn't grab your attention.⁷

In some cases, I try to use the same title, so it's really easy to skip in the feed reader. This is not about forcing people to read content they don't want to read. And you can be sure that if I'd gotten a couple of "Geez, man... stop with the stupid open letters"- or "I really hate your Gimp Art (tm)"-comments, I would've found another outlet for that particular content. But that hasn't been the case.

In fact, it's been quite the contrary.

Immaterial of the feedback from readers in the Community, I find myself experiencing a paralysis over writing anything now. It's been communicated—without being given the respect of being emailed or otherwise talked to— that I'm "allowed" to write about Mozilla, Firefox, Gecko, and Tunderbird. I find myself thinking "Well, I don't really have anything to say about those four specific topics. So I guess I won't bother writing anything," mostly for fear that this stuff really is boring and no one wants to read it.

That's a [hopefully?] unintended chilling effect of someone striving for "a lightweight process," but it nonetheless exists, and it's not very pleasant.

I'm not exactly sure how we're going to solve the problem; there are lots of possible solutions⁸, but I am sure of one thing: the silent filtering and dropping of content needs to stop.

The unilateral implementation of editorial policy that has not been discussed and is not posted anywhere needs to stop.

And the conversation about what's appropriate for our community needs to be had, not avoided.

We're the Mozilla Project; we have a history of doing better, and we can do better.

__________________________
¹ But, being among only three people being filtered, I guess I'm in good company.
² Does it filter subject titles? Tags? Posts? Blog comments? What? I actually spent some time Googling for this, and never did find a clear explanation.
³ In Pink's case, it is especially ironic that every single post he's made since he was removed for "very little Mozilla content" has been about Camino and/or Mozilla.
⁴ Pinkerton refers to a scary situation "when voices in the community can be turned off like a faucet at the whim of people at a corporation." I think this is somewhat confusing, because when we talk about "a" or "the Corporation," we're typically referring to the Mozilla Corporation. Tim Rowey works for IBM, not MoCo.^5
5 And I think it needs to be said: I have no doubt in my mind that there would be rioting in the virtual-streets if someone at MoCo were asserting this level of editorial control-without-input over a resource like Planet.
⁶ But if you read between the lines, it's actually not, which is why people said they like it; it actually applied to the Mozilla project in a usually-tangential way.
⁷ Which, of course, turns out to make it a [useful] challenge for me, the writer, to grab your attention, using only a single paragraph!
⁸ This whole problems screams out for a solution addressed by that Web 2.0-ey concept of semantics; this seems like a solved problem, what with tags and feed readers that can filter, based on tags. I would gladly tag my posts with an agreed-upon tag, so that those not wanting the strictly-not-Mozilla-stuff could filter. In fact, I already do, but "blahblahblah" isn't a very standard tag. ;-)

Posted by preed at 4:03 PM | Permalink | Comments (15)

Dear Firefox Community,

I need your help here: how do you best pimp Firefox in three minutes?

Slashdot recently posed this question in regard to Linux, and at the time, I mostly ignored it, because... well... I didn't find it particularly relevant.

But then, in a coffee shop this weekend, I was getting my morning cup o' awakeness, and the woman behind me in line started up a conversation: "Oh, I have Firefox installed! But I was going to delete it. I like your shirt though..."

I smiled and said "No! You should start using it! We just released a new version."

She asked "Well... why should I use Firefox?"

I first said "Well, because it's more secure than Internet Explorer," and then I quoted the IE was insecure for "like hundreds of days last year," and Firefox was insecure for "like a week or something."

She said "Wow, that's really cool." But then seemed underimpressed.

I then said "It's also one of those 'feel-good' things, you know; Firefox is built and supported by a community of people, and we work really hard to a build a browser with only users and their online experience in mind."

"Yeah, that's true," she said. She then said "I originally downloaded it because Safari wouldn't display something I wanted to see."

Before I could really respond, it was time to order my coffee, and then since she was behind me, we both got distracted, and never were able to really finish the conversation.

So, my question to you: if you only had two to three minutes to talk up Firefox and get someone to keep it on their machine, how would you explain why it's the best browser around to someone who doesn't care a bout (or maybe even understand) "attack surfaces" and "days of exposure," and gets "The Community" in the abstract, but... not enough to make it relevant to their personal livfe.

Which aspects of the 'fox we all know and love so well would you focus on in 150 seconds?

Aaand... go!

Sincerely,
Your Friendly Build Engineer

P.S. I've also been trying to figure out whether or not she was flirting with me, but... that's another post altogether.

Posted by preed at 12:27 AM | Permalink | Comments (9)