preed's blah-blah-blahg

For some reason¹, a friend and I had this ongoing joke about flying to beautiful, beautiful Modesto together.

This weekend, we finally got around to doing it.

It was an interesting flight.

On the way out, visibility was all over the map in the central valley, as there was a forest fire... "somewhere."² We were about four DME from the field, and I still couldn't find it visually. Our saving grace was the fact that MOD has a VOR at the field, so my friend's GPS track of the flight³ shows a fairly... obvious 70-degree turn to get into the pattern.

Modesto-the-airport was amusing, too. I think the best scene was the arrivals/departures board, which had nothing but "one" for the gate number. Dealing with "the TSA" was also... interesting.

The journey home treated us to a fortuitous landing on SJC's runway 30L.

If you've ever flown commercially into San Jose, you'll get one of the three-zeros⁴. They seldom give it to general aviation aircraft, but I guess we arrived during a slow period, and there was enough haze that giving us the runway with the full instrument approach lighting system was worth it, I guess?⁵

I didn't notice at the time, but my friend shot some video of the trip; it's a bit long, but the sunset he captured was beautiful⁶, and if you've ever wondered what it looks like landing in San Jose from the front⁷, be sure to take a gander. There's some great footage of the general area where MoCo lives, too, from about 1500 feet.

The time of night for those last couple landings⁸ and takeoffs are hands down my favorite time to fly.

Last night, it was overcast at 1900 feet in Mountain View, so I gotta enjoy it while I can.

I put it into production last week, as a replacement for gaius[-vm]. It's based on the new VC8 reference platform, and astute Tinderbox readers will notice that its cycle time is almost half that of gaius-vm!

I'll likely switch it to be the nightly Tinderbox this week after checking with Vlad and Stuart.

___________________
¹ Now long forgotten...
² And yes, that's a direct quote from the Flight Service Station: "We're still trying to figure it out."
³ I specifically took up an old school (tm) plane, because I wanted some challenge navigating on this flight... GPS makes us so lazy...
⁴ And now, since 30R is closed for some construction on the ramp, you'll always get the left.
⁵And it was, since we had a bit of trouble finding San Jose, too... of course, I could have always cheated there since they, too, have a VOR at the field.
⁶ About two minutes in.
⁷ About three minutes in.
⁸ About six minutes in.

Posted by preed at 12:13 PM | Permalink

Last night, my blog became a bit of a lightning rod¹, as readers from Slashdot poured in to tell us exactly what they think of how we release software.

I just finished reviewing the hundred-some comments², and I think I've realized four things:

1. People who have managed and worked with large releases of open source software, involving millions of people, and tens-to-hundreds of partner mirrors seemed to understand the complex set of issues that accompany such a process, and were sympathetic. Those that hadn't... well... added mostly a lot of noise.
2. Most people missed the point.

   As Beltzner's comment points out, the real problem we're trying to address are news sites, articles, and the like reporting "OMGSquee Firefox 2 is released, go get it here!" when that's just flatly wrong. For sites that have editors, it's irresponsible journalism, because it's just false. And they need to know that it's both irresponsible and incorrect.

   For those that don't, users need to know the effect of all their "diggin'" and "reddit'ing," and hopefully "the community" will do the right thing.

   In both cases, many of Slashdot's core users are not what we're so worried about. When they notice others getting automatic Firefox updates that they may not be getting, they'll probably take the time to figure out why. They'll upgrade themselves, because not all 38 of their Firefox extensions didn't work with 2.0 yet. They'll find the en-US build, because they know how to chop apart URLs, and go trolling through the FTP tree for the right locale (and for Linux builds³).

   Where it especially causes problems is when geeks see it, and download it for their girlfriends' computers.

   Or boyfriends'.

   Or parents'.

   Or Liberal Arts major roommates'.

   Those people are going to start up Firefox 2, and find 404'd startup pages. The online help won't be there. Or if it is, maybe it won't be localized for their language yet.

   They may be using the wrong locale for their needs⁴. Heck, they may get an unsupported locale⁵.

   These are the people it really hurts.

   It tarnishes all the hard work that a community of hundreds put months of their time into, because some couldn't wait.

3. Yeah, this is open source software... and everything we do is in a fishbowl.

   That's the great part of open source software: you all get to see what the developers do, and what QA does, and what Build/Release does... and you get to call us stupid when we tell you the pan is hot, and you shouldn't touch it [yet]. But at the end of the day, you get heed the warnings and suggetions—or not—and make the decision about what's best for you and your needs.

   What I did find surprising was reading a comment that said "You can't appeal to people's good nature, when there are millions everywhere around the world waiting for the new version."

   Firefox and Mozilla have always relied on "people's good nature" to help each other out, do the right thing, and work together to produce great software and make the web a better place. Open source itself is founded upon this general principle.

   Firefox is unique, in that it's a very end-user facing application.

   I guess that sentiment tells me we've all got some work to do to help acclimate people to the idea of a community driven software development model, where taking down someone's Internet connection in Europe or Asia who's volunteered to host bits closer to you really isn't OK.

4. Finally, while going through the comment moderation queue, it's clear why a lot of you want Firefox 2 the second you can get it: the new inline spell-check feature for submission forms ;-)

Having said all that... Firefox 2 is now released.

And I'm off to take a nap now⁶.

___________________________
¹ Which... eh... not the first time...
² Not all of which were posted... but neither were just positive, back-patting comments posted.
³ It amuses me to no end that such a large open source site linked directly to Win32 builds
⁴ en-GB vs. en-US, for instance. Or es-AR vs. es-ES.
⁵ Where "unsupported" is defined as "won't get automatic updates, and remain secure on the web," which is like... a huge reason to use Firefox in the first place.
⁶ After having a piece of MS Cake 2006, Service Pack 2⁷ of course!
⁷ "Giving new meaning to dumping core."

Posted by preed at 2:19 PM | Permalink | Comments (3)

I mentioned this in today's Project Meeting, and now it's come up in the newsgroups.

Normally, I'd take the time to write up a more in-depth explanation, but since my schedule is... uh... "hectic" right now, I'll just say this:

1. No, we have. Not. Released. Firefox. 2. Yet.
2. When people link to bits directly on a random FTP mirror, they're doing a number of people harm including, quite possibly, themselves:
   • Digg and Reddit posts linking to direct FTP mirrors could be costing the operators of those mirrors hundreds to thousands of dollars in bandwidth bills, or may cause them to crash by linking directly to them. This could cause them to "un-volunteer" their services as a mirror, making it even harder to obtain Firefox on release days.
   • People posting direct link to FTP mirrors don't know if that mirror is a member of the Mozilla FTP Mirror Farm, or some random, unverified mirror. We work hard to verify that the mirrors in our farm are serving the same bits we released, and we cannot make the same claim about other mirrors that aren't part of our farm. When using direct FTP links to random mirrors, users run the risk of downloading bits that have not been checked to ensure they do not contain a virus or trojan.
   • "That's ok," you say: "I link directly to ftp.mozilla.org!" That can be even worse! Killing the project's FTP server does not help anyone, least of all people trying to obtain Firefox builds. And it makes for a grumpy IT group. And nobody wants grumpy IT groups. Especially a day before a release.
3. Linking directly to builds hinders our ability to remove/retract bits that we may have to remove for some reason. While this may not seem like a big deal, it becomes a problem when supporting users, one of our most important values. If, let's say, we pull a locale, due to a stop-ship bug—and yes, this is not a hypothetical—then users who've (pre-)downloaded that build will not receive valuable security updates for those builds. The counterargument to this is "Well, you should provide updates for everything you've ever offered on your FTP site." If we did this, we'd be spending valuable (and über-constrained) Build Team and QA resources generating updates and testing them for builds that weren't the final bits, and were never "released" as such.
4. Posting links before we release may point people to an incomplete FTP areas or mirrors. I haven't finished posting the source tarball, for instance. Will it happen before we release? Yes. Will there be unnecessary confusion from the open source community, wondering where this deliverable is? If you post links to an FTP site with the builds, yes.
5. Most articles have an unerring ability to link to the wrong thing. Slashdot's front page, for instance, currently links to the Windows British English build. I cringe at the thought of the community having to waste time while we're finishing things up with IRC, blog, and Bugzilla chatter asking "I got my build from Slashdot; why did you guys spell behaviour wrong?" And where are Slashdotters wanting uhh... you know... Linux builds supposed to get them? It's unclear from the article that directly links to an .exe for one [correct for one country, but mostly-wrong for everyone else] locale.
6. User experience can be degraded, leaving a bad taste in people's mouth: Firefox 2 has a number of components that use live content on websites. The whole community has been doing a lot of work to refresh, update, and translate this content, and parts of it are still coming together for the release. When you download a build, there could be various content, including certain parts of help, that are not yet ready. When you tell your friends to go download Firefox 2 before we announce it's ready, you're subjecting them to a degraded user experience, which could push them to go back to... "other browsers."

Now, before you suggest it, it's not as easy as putting in .htaccess restrictions, or setting the permissions on the files so people can't download them. The nitty-gritty details are in the newsgroups.

So please... just remember: "Preed the Release Engineer says: friends don't let friends download Firefox before it's released."

We know everyone's excited for the 2.0 release. We are too. But give us 24 hours, so we can make sure that your first experience with Firefox 2.0 is befitting of everyone's hard work on this major release.

I promise it's worth the wait.

Posted by preed at 2:27 PM | Permalink | Comments (78)

Thunderbird continues to amaze me.¹

One feature that I hadn't started using until a couple of weeks ago was Thunderbird's built-in spam detection. I mostly didn't use it because it looked like it never worked. It didn't mark anything as spam, and I thought "Well, this is a crappy filter."

It turns out that I was just a crappy user.

You have to train it to recognize the spam by tagging spam messages as junk. Then it learns from this, and actually starts detecting the spam. Who knew?

(Yes, yes... this may seem obvious to some, but... try not to laugh. I come from a spamassassin/mutt-based world, where most of the spam detection heuristics used are either network-based or based in rules that ship with the software. I didn't expect that I'd have to actively train Thunderbird, because I was expecting a spamassassin-like experience. I also haven't seen any obvious documentation on this aspect of Thunderbird, a problem I know Sheppy is hard at work fixing, which is awesome.)

Now that I've trained it, and setup a filter to auto-move spam to junk, my inbox is much cleaner. And it seemingly hasn't [incorrectly] grabbed any ham yet.

In talking with Scott about it, he said optimally, you should mark about 20-50 messages as ham, in addition to marking spam the filter doesn't catch. This helps build its ruleset so that it understands what kind of ham your inbox gets. This feature isn't as "discoverable" as marking messages as spam; you have to go to Message -> Mark -> As Not Junk (aka Shift+J).

The other cool "Wow, it just works" Thunderbird feature I ran into that really impressed me was a message that had "^2" in it. Thunderbird (correctly) rendered this in the message as a superscript "²".

Today, I bit the bullet and begrudgingly moved work-related blog-reading over to [the new incarnation of] Google Reader. I had been using Thunderbird for this task, but I have an installation of Thunderbird on my work laptop and work desktop, and I found that because it has no way² of syncing the read-lists of blog posts I've read, I either a) don't use it, or b) have to wade through a bunch of posts I've already read.

We'll see how it goes with Google Reader, but I'd rather use Thunderbird. I'm not a huge fan of not-knowing what data Google Reader is collecting about what I may (or may not) be reading.

____________________________________
¹ To be clear, in the good way...
² That I know of?

Posted by preed at 5:44 PM | Permalink | Comments (10)

Last time I did 2.0 release candidate localization builds, I ran into a problem with the tagging and checkout.

For various reasons, we only tag the locales that we ship for a particular release with the _RELEASE tag. During the build, client.mk checks out every locale specified in all-locales. This has worked beautifully for the 1.5.0.x maintenance series, since the locales we ship aren't shifting a lot, there aren't many, if any, new locales on that branch, and if you check out a directory that isn't tagged, you get... nothing. Exactly what we want, right?

Well, we repeated this process for the 2.0 RCs, and suddenly, the builds started failing with "cvs [checkout aborted]: no such tag ". This was certainly a surprise; "I had just created the tag," I kept thinking to myself.

"Am I going insane?"

I started debugging it, and was only able to reproduce it once originally. Once I got a checkout going, it seemed to work repeatably, and since I was busy with 2.0 RC 2, I didn't investigate more.

Well, it happened again with today's l10n builds. Originally, rhelmer and I thought it may have been a problem of using the wrong CVSROOT, since cvs-mirror.m.o only gets updated every few minutes, and I had just created the tag. This didn't make a huge amount of sense, as the command is run with -d, to specify the CVSROOT directly. A peek at the source confirms that -d takes precedence over $CVSROOT and CVS/Root. Then I thought maybe it was a compatibility problem. It turns out that we use CVS 1.11.2 on the client side to create release tags; maybe this is so old—it was released in 2003—that it was hiccuping with something server side?

After trying to reproduce this problem for rhelmer, I was only able to reproduce it once before it worked. Again. Something must be modifying the state server-side.

After some experimentation and more source reading, it turns out that an "optimization" introduced in the CVS 1.11 line, so-called "val-tags", is responsible for the bug.

In a nutshell, when using val-tags, CVS searches for the existence of a tag by 1) looking into val-tags, and then b) looking at the RCS files themselves. This normally isn't a problem, except in the case where an untagged directory is requested before a tagged directory. In the case of RC2, l10n/af was not part of the release, and therefore untagged, but l10n/ar was tagged. They were checked out in that [alphabetical] order.

Running "cvs co l10n/af l10n/ar" will repeatably produce the (incorrect) "invalid tag" error until you run "cvs co l10n/ar" (or some other checkout for which the tag does exist first. This adds the tag to the "val-tags" file, and after that point, CVS will check the val-tags file first, to see the that tag does indeed exist, and then traverse all of the directories you've listed, instead of the first one.

All the gory, buggy details are in tag.c's tag_check_valid(), which, based upon my very cursory reading of the source code, still exists in CVS 1.11.22.

While reading through the source, I was surprised at the number of comments that... didn't inspire confidence:

/* FIXME: This routine doesn't seem to do any locking whatsoever
(and it is called from places which don't have locks in place).
If two processes try to write val-tags at the same time, it would
seem like we are in trouble.  */

/* FIXME: should check errors somehow (add dbm_error to myndbm.c?).*/

/* warm fuzzies */
if (!really_quiet)

I think the moral of today's story is: open source is cool because you can look at the source... but if you ever do, there's a very real chance you could become very depressed.

Or scared.

Or both^1,2.

I only remembered this because I was walking around Google's campus and saw a sign noting it.

It always seems to sneak up on me every year⁵.

___________________________
¹ Which makes me wonder how much of our source has "gems" like that...

² I'm sure at least someone out there is asking "So, Bigshot where's the patch then?" Well, I spent a some time looking at this... and decided that I couldn't spend anymore time wrapping my brain around a function (start_recursion) that calls yet another long function (do_recursion), especially when it seems that most of the CVS devs don't either^3
3update.c says /* FIXME-twp: the arguments to start_recursion make me dizzy. This function call was copied from the update_fileproc call that follows it; someone should make sure that I did it right. */
⁴ Read "completely"
⁵ It's much like MMLRD, but on a yearly scale...

Posted by preed at 1:09 AM | Permalink

Despite the fact that I have a PowerBook [for work], and use a PowerBook [for work], I'm no fan of Apple.

I've got a lot of complaints, really... but I think they all boil down to disagreement with one of the company's values: aesthetics above... well... anything and everything else.

This is a constant pain whenever I['m forced to] use Apple's products.

The current middle-finger-from-Cupertino's-general-direction is the inability to purchase Xserves. At all.

We need more Xserves. This isn't a huge secret. We're pretty short on Mac-compute resources in the build farm and since Apple is the only platform that can read and write all of the various package formats we ship¹, we've sacrificed a Mac to use as a console for various build-related activities, in addition to the nightly and release builds we need to get for various products. (Ironically, this has impacted the Mac browser product, Camino, the most, as they've been stuck using older, less capable, unsupported build machines for months now, because... we can't get new Xserves).

I'd like to get more Xserves. I had IT check into buying more PPC Xserves, but he can't find any distributor that is selling them anymore. Why?

Well, since Stevey-boy announced Intel-based Xserves, everyone's waiting for those... and in fact, you can't even seem to get them from Apple directly anymore.

So basically... if you depend on Macs, need more of them, and want to give Apple money... well... screw you. Just wait an indeterminate amount of time until Apple is good and ready to ship you their Rev A hardware².

(And since Apple's reputation for Rev A hardware is so wonderful, I can't tell you how excited I am to put these immediately into production. We might even get to be one of the lucky ones that finds our colo burnt to the ground because a power supply blew up. But hey, at least it's got brushed metal and pretty blinky lights in the rack.)

I just don't understand how people can be so beholden, philosophically, religiously, heck even economically, to a company that so consistently thinks that its users', developers', and partners' time is worthless, and that it's completely reasonable to stop supporting certain hardware [platforms] and software because... well... the new rev is prettier. And faster. And certainly hotter!³

This is but one of the reasons I don't like Macs.

It's the main reason I detest being required to use Macs (in a business capacity).⁴

No clue why that could be.

____________________
¹ And before you chime in that this is proof how wonderful Macs are, this problem is entirely created by Apple, because they refuse to release documentation on how to read/write DMGs that include their special installer format
² Yes, we've ordered Intel Xserves. The current estimate for when they'll ship is—I kid you not—"sometime in October." Probably. Maybe.
³ I mean that in a degrees-Celsius sort of way, not a Steve Job's-stylin' sort of way.
⁴ And actually, in this regard, Apple loses to Microsoft. Sure, Microsoft sucks in lots of ways, but at least Microsoft never deprecates anything.

Posted by preed at 8:48 PM | Permalink | Comments (6)

While sitting here, waiting for the Firefox 2.0 RC 2 candidates to finish compiling (go Win32 SpeedRacer!!), I started thinking about that post I wrote (has it already been?!) two weeks ago.

After re-reading it, I fired up Google to scour the blogosphere to see if anyone else is writing about this stuff. "Build/Release Engineering blog" seemed like a good place to start.

The picking were pretty slim, with the first twenty or so results all posts referring to build/release engineers, job pages (including a cache of MoCo's old Careers page), and feed links to... well... my blog.

That's... uhh... real useful there.

Next I tried "build engineering blog," which pointed me at Make Magazine's blog and a bunch of Google API documentation? Hrm... alright.

"Release engineering blog" doesn't get us any closer, providing a weird combination of content from "build engineering blog" and random posts complaining about release dates of various products slipping.

Changing my search trajectory, I tried "scm blog." I guess the business majors' "SCM" is more popular than mine.

All in all, I managed to find two lighthouses in the Supply Chain Management-dominated sea.

The most engaging aspect of his essays is how they prompt me to consider subjects I normally wouldn't. Sure, a lot of the Win32-specific stuff he discusses every so often isn't directly applicable to my daily hacking-life, but most of it, including his treatises on scheduling and the other "soft stuff," apply in a "build land." Admittedly, the wavelength is often... a few degrees out of phase, so it does require some mental modulation.

But that probably isn't the most valuable aspect of his writings: I don't always agree with him.

Sometimes, he writes about Microsoft with a nostalgic flare that would choke a anti-trust judge. And I can't say that I'm impressed with his constant reference to Netscape's "huge mistake.¹"

But whether or not you agree, this valus is in pushing the discourse forward. Making others form their own opinions, translating to their own sandboxes as necessary, should never be underestimated.

And honestly, I've never seen a man—to say nothing of a software engineer—squee so much about his office space [re]design.

I just love it.

I asked a friend what he thought of when I said it, and he said "Oh, that's like 'poor little boy' in Spanish."

Urban Dictionary is less charitable, but as for the subject matter discussed herein, probably more accurate.

What I have yet to figure out is whether it refers to the author... or the reader(s?).

I guess you'll have to tell me.

And while you're at it, feel free to pass along any Google-fu on finding more lighthouses to be looking out for.

_______________________
¹ I go back and forth on that one; the truth is probably somewhere in between...

Posted by preed at 11:50 AM | Permalink