preed's blah-blah-blahg

When I joined Mozilla Corporation, you either ran your own blog¹, had a Mozillazine blog, or didn't involve yourself in such frivolities at all.³

An X-Ray of Major Software Releases

And all was good with the world. For a time.

But times change, and at some point in the last four years, it became cool⁴ to run your own blog.

So now, years later, since all the cool kids are doing it, I finally set up my own blog.

If the name—The Sober Build Engineer—doesn't immediately make sense, maybe the introduction will help.

In an attempt to not spam Planet Mozilla, I won't be syndicating everything; but if you find my ramblings amusing⁵, the raw RSS feed is always available.

I'd like to thank MattyT, Mozillazine, and OSUOSL for hosting "my Mozilla blog" for the last few years.

And I hope to see y'all on the flip side⁶.

Oh... and of course, the first round is on me...

__________
¹ Not as likely, since it wasn't as "cool" back then^2
2 Yes, I'm that old....
³ Until the "blogosphere" became a "big deal"
⁴ Dare I say chic?
⁵ Or otherwise...
⁶ Bug 532342 tracks movin' me over

Posted by preed at 11:45 PM | Permalink

The Mozilla Project has a history of producing useful software and tools in the pursuit of its main mission that are not directly in the path of its mission.

MozillaBuild is one of these.

If you're trying to build something that's in any way similar to or remotely expects a Unix-ishy environment, and you don't want to deal with Cygwin's eccentricities¹, MozillaBuild not only provides such an environment, but packages a host of useful goodies along with it.

In short, MozillaBuild provides enough "It Just Works (tm)"-ed-ness for the whole family! It's gotten to the point where I often forget that I actually still am in Windows when using it².

Recently, though, a jarring reminder that I was, indeed, still in Win32-land occurred as I went to install some Perl³ modules a particular tool we use pretty heavily here at Songbird depended on.

I went to run the command on our Win32 release build machine, and was met with a missing-modules error. "No matter, that's what CPAN's for!" I say. Weeelll... not so fast.

It turns out MozillaBuild's Perl installation didn't get the memo that it's in a Win32 environment, so when you try to run CPAN, you'll get all sorts of errors.

Apparently, I'm not the only one to have had this problem.

Since I really needed to get this tool working on Win32⁴, I did some couples counseling with MozillaBuild⁵ and CPAN, and came up with a fix:

1. Configure CPAN.
   • Log in as Administrator
   • Run perl -MCPAN -e 'shell'.
   • When prompted whether you're "ready for manual configuration," say "no"
   • Exit the shell
   • In an MSYS terminal, manually edit /usr/lib/perl5/5.6.1/CPAN/Config.pm; change the following variables:
     1. build_dir: [Anything that doesn't contain a space; I ended up using:] /d/.cpan/build
     2. cpan_home: /d/.cpan
     3. ftp: /bin/false
     4. keep_source_where: /d/.cpan/sources
     5. urllist: q[http://mirrors1.kernel.org/pub/CPAN], q[http://mirrors2.kernel.org/pub/CPAN], q[http://ftp.osuosl.org/pub/CPAN/]]
   • Run perl -c on /usr/lib/perl5/5.6.1/CPAN/Config.pm to make sure it compiles and you didn't typo anything in the changes you
   • Apply the following patch to /usr/lib/perl5/5.6.1/CPAN.pm.

And you're done!⁶

You should now be able to run commands like:

perl -MCPAN -e 'shell'

install Digest::Perl::MD5

force install URI

One (rather large) caveat to this process is many Perl modules have C extensions that get built at installation time. Most of these assume Unix, and therefore that gcc is around.

If you don't have a full MSYS environment, installing them will fail, unless you can turn these bindings off. (Even in the case of some modules (URI, for instance), they will refuse to install because of failed tests; you can get around this by forcing an installation.)

It's not perfect, but if your requirement is pure Perl modules, as mine was, it's a workable solution.⁷

__________________________

¹ And less-than-spectacular performance
² as much as that's possible to do; I usually remember around the point I want to copy/paste some text around...
³ Yes, yes, I know; Python forever... but every build infrastructure I've ever worked with has perl in it... somewhere
⁴ Otherwise, our Lord and Savior, AUTOMATION, couldn't proceed
⁵ I experienced this issue with MozillaBuild 1.3, but I checked the latest MozillaBuild and the Perl installation there doesn't seem to have changed, so this process+patch should still be applicable
⁶ What this process is generally doing is: removing paths with spaces in them, since %20's in Unix paths are a rarity; skipping manual CPAN configuration which will get tripped up on a bunch of things and generally try to install the world for you; always cause ftp commands (which use ftp.exe) to fail, since ftp.exe != the venerable unix ftp, and the CPAN module will get very confused trying to talk to it; and finally, forcing CPAN to always use wget, which MozillaBuild includes, but not use redirection to save files, since MSYS seems to (unhelpfully, I might add) attempt to translate line endings for you.
⁷ Until you just rewrite your can't-live-without-it tool in Python...

Posted by preed at 1:33 PM | Permalink | Comments (4) | TrackBacks (0)

The definition of "latest and greatest version" just changed...

I heard some some rumbling that the latest version of Firefox, version 3.5¹, hit the web today.

This is most certainly a big accomplishment and everyone involved has many reasons to be proud of this release. But that's not what I wanted to talk about².

I wanted to step back for a moment and call out a group of really important people that are often overlooked in all the excitement during a major release: the sustaining engineering team³.

They're the group of developers, bug triagers, QA engineers, build engineers, ops engineers, web developers, and project managers who have kept the 300 million and change Firefox 3.0.x⁴ users safe, secure, and stable for over a year now.

Sustaining engineering has never been particularly sexy work.

Often times, the pressures imposed by consumers of "sustaining releases" make it particularly grueling work: risk assessment becomes a large—and often difficult—part of the job. Mistakes can be very costly and affect higher numbers of users. Such users are less tolerant of sometimes-necessary changes. And in the case of many software products, Firefox included, when a security vulnerability is involved, all of these decisions and work needs to be done on a very tight schedule.

These realities are doubly true for an open source project, where the "release early, release often"-mentality⁵ often leaves those who toil away on sustaining efforts appearing in relative obscurity. And since open source capital is about visibility, less visibility can translate to less understanding of the actual work being done and the value these teams and individuals bring not only to the community, but to the end users.

So to those who've done this difficult, thankless work, quietly, consistently, and proudly: you have my thanks. And even though they may not know about you, you have the thanks of every Firefox user who is able to easily click "Check for Updates" to get today's New Hotness (tm), because your work has kept them safe and secure on the web for the past year.

If history is any guide, these tireless souls are already getting ready to do it all over again and take stewardship of 3.5.x after a couple of releases or so.

So take a moment to raise a (virtual) glass to them⁶... and then get back to enjoying the awesomeness that is Shiretoko.

____________________
¹ Back in my day, it was called "Firefox 3.next." P.S. Get off my lawn.
² If you do want to read more about the 3.5 release, try here, here, here, here, here, here, here, here, here, here, here, here,here, here, here,
here, here, here, here, here, here, here, here, here, here, or here.
³ I don't know what Mozilla Corporation calls this team, exactly; unlike some other organization, I don't think (structurally) they split them out; but the function is the same
⁴ And you have to admit... it was a pretty good vintage of Firefox
⁵ Which, don't get me wrong, is critical for any open source project's success
⁶ Or, if you're in a physical localized spot next to them, a real one...

Posted by preed at 3:53 PM | Permalink | Comments (3) | TrackBacks (0)

It's like a Turbo button... only faster!11

That's not to say that us build engineers are sitting in a room, typing make && cp over and over again and staring at the screen until it finishes.²

On the contrary, improvements to our build and release infrastructure are made constantly.

But they often involve making our lives easier: a script that handles errors more appropriately, or hooking up two pieces of the puzzle together, so we can go to bed at 2 am, instead of 4 am. These are not readily visible to the end-user or our customers, the developers³.

So, given that, you'll have to forgive me for being a bit squee⁴ about the following:

svn up -r 13127 | grep ^Updated &&
time make -f songbird.mk > /dev/null 2>&1 &&
make -f songbird.mk clobber > /dev/null 2>&1 &&
svn up -r 13128 | grep ^Updated &&
time make -f songbird.mk > /dev/null 2>&1

real 2m40.991s
user 2m22.129s
sys 0m17.253s

Yes, that really is a 29% improvement in wall-clock build time.

On repeated trials, it drops to about 26%. I'm guessing this has to do with caching the source files.

And these results are from Linux; since the core problem⁶ had to do with excessive use of $(shell ...), and therefore fork(), the results should be even slightly better on Win32.⁷ I didn't test Mac, but one of our Mac gurus twittered the Turbo before I mentioned it to anyone.

Lessons learned:

1. make's $(shell ...) construct is very useful. It's also extremely dangerous, in part because it's so convenient, easy to use, and... useful. If you can do anything else to avoid it, do that.
2. If you're assigning the results of a $(shell ...) call into a variable, you probably want :=, not =. And if you really want =, see LL#1.
3. Win32 MSYS fork()ing performance has a notoriously bad reputation... but it's not like fork(2) is free on Linux or Mac. I'm surprised at the savings on Linux and the small delta between Linux and Win32.
4. Running make SHELL="/bin/sh +x"⁸ on your project might prove very interesting to you. Are you really executing that much extra fluff in a subshell? Yes. Yes you are.⁹
5. make has a surprising number of useful functions; if you're spending any time doing build system stuff, read up on them. I myself was guilty of doing ifeq (exists,$(shell test -e $(FILE) && echo exists) before I learned about $(wildcard ...). The former spawns three processes. The latter: zero.

This is just one of the few makeovers¹⁰ that I'm working on; there's something in the works for the installer, and I'm working on some pretty fundamental changes to the build system itself.

(I call that project Build SystemNG, or BSNG for short.)

I'd be lying if I said I didn't enjoy working on something that people notice... either because it's noticeably faster or completely broken.

One of the two.

__________________________________
¹the famed "rebuild the airplane while it's in flight"-problem, which I've decided is probably the most asinine analogy I've ever heard. So I'm going to stop using it.
² As some might have you believe
³ Except, maybe, if you're paying attention to the amount of scotch consumed
⁴ I keep thinking everyone knows what this means, but I keep running into people who don't. Even people who I would think would... don't. So here's a convenia-link⁵ for it to Urban Dictionary, in case you don't.
⁵ Like a perma-link, only less permanent, and more convenient
⁶ Bug 15529 has all the details
⁷ I only had a VM to test those; I don't have the numbers in front of me, but as I remember, they were somewhere between 30 and 32%.
⁸ A benchmarking hint from Mecklenberg's O'Reilly GMake book, which I recommend
⁹Over half a million of them, actually... which is why tips-and-tricks are always helpful: everyone wants faster builds.
¹⁰ Hahahahhaa... make over? gmake? Get it?
¹¹ Photo courtesy D.G.S..

Posted by preed at 1:24 AM | Permalink | Comments (2) | TrackBacks (0)