June 30, 2009

Firefox 3.0.11 (still) released!


The definition of "latest and greatest version" just changed...

I heard some some rumbling that the latest version of Firefox, version 3.51, hit the web today.

This is most certainly a big accomplishment and everyone involved has many reasons to be proud of this release. But that's not what I wanted to talk about2.

I wanted to step back for a moment and call out a group of really important people that are often overlooked in all the excitement during a major release: the sustaining engineering team3.

They're the group of developers, bug triagers, QA engineers, build engineers, ops engineers, web developers, and project managers who have kept the 300 million and change Firefox 3.0.x4 users safe, secure, and stable for over a year now.

Sustaining engineering has never been particularly sexy work.

Often times, the pressures imposed by consumers of "sustaining releases" make it particularly grueling work: risk assessment becomes a large—and often difficult—part of the job. Mistakes can be very costly and affect higher numbers of users. Such users are less tolerant of sometimes-necessary changes. And in the case of many software products, Firefox included, when a security vulnerability is involved, all of these decisions and work needs to be done on a very tight schedule.

These realities are doubly true for an open source project, where the "release early, release often"-mentality5 often leaves those who toil away on sustaining efforts appearing in relative obscurity. And since open source capital is about visibility, less visibility can translate to less understanding of the actual work being done and the value these teams and individuals bring not only to the community, but to the end users.

So to those who've done this difficult, thankless work, quietly, consistently, and proudly: you have my thanks. And even though they may not know about you, you have the thanks of every Firefox user who is able to easily click "Check for Updates" to get today's New Hotness (tm), because your work has kept them safe and secure on the web for the past year.

If history is any guide, these tireless souls are already getting ready to do it all over again and take stewardship of 3.5.x after a couple of releases or so.

So take a moment to raise a (virtual) glass to them6... and then get back to enjoying the awesomeness that is Shiretoko.

____________________
1 Back in my day, it was called "Firefox 3.next." P.S. Get off my lawn.
2 If you do want to read more about the 3.5 release, try here, here, here, here, here, here, here, here, here, here, here, here,here, here, here,
here, here, here, here, here, here, here, here, here, here, or here.
3 I don't know what Mozilla Corporation calls this team, exactly; unlike some other organization, I don't think (structurally) they split them out; but the function is the same
4 And you have to admit... it was a pretty good vintage of Firefox
5 Which, don't get me wrong, is critical for any open source project's success
6 Or, if you're in a physical localized spot next to them, a real one...

Posted by preed at 3:53 PM | | Comments (3) | TrackBacks (0)

March 23, 2009

Build System Improvements By the Wall Clock Numbers


It's like a Turbo button...
only faster!11
In between the constant grind of getting releases out the door, it's often a struggle to find time to make improvements that are imminently noticeable.1

That's not to say that us build engineers are sitting in a room, typing make && cp over and over again and staring at the screen until it finishes.2

On the contrary, improvements to our build and release infrastructure are made constantly.

But they often involve making our lives easier: a script that handles errors more appropriately, or hooking up two pieces of the puzzle together, so we can go to bed at 2 am, instead of 4 am. These are not readily visible to the end-user or our customers, the developers3.

So, given that, you'll have to forgive me for being a bit squee4 about the following:

svn up -r 13127 | grep ^Updated &&
time make -f songbird.mk > /dev/null 2>&1 &&
make -f songbird.mk clobber > /dev/null 2>&1 &&
svn up -r 13128 | grep ^Updated &&
time make -f songbird.mk > /dev/null 2>&1

Updated to revision 13127.

real 3m45.008s
user 3m7.301s
sys 0m26.064s
Updated to revision 13128.

real 2m40.991s
user 2m22.129s
sys 0m17.253s

Yes, that really is a 29% improvement in wall-clock build time.

On repeated trials, it drops to about 26%. I'm guessing this has to do with caching the source files.

And these results are from Linux; since the core problem6 had to do with excessive use of $(shell ...), and therefore fork(), the results should be even slightly better on Win32.7 I didn't test Mac, but one of our Mac gurus twittered the Turbo before I mentioned it to anyone.

Lessons learned:

  1. make's $(shell ...) construct is very useful. It's also extremely dangerous, in part because it's so convenient, easy to use, and... useful. If you can do anything else to avoid it, do that.
  2. If you're assigning the results of a $(shell ...) call into a variable, you probably want :=, not =. And if you really want =, see LL#1.
  3. Win32 MSYS fork()ing performance has a notoriously bad reputation... but it's not like fork(2) is free on Linux or Mac. I'm surprised at the savings on Linux and the small delta between Linux and Win32.
  4. Running make SHELL="/bin/sh +x"8 on your project might prove very interesting to you. Are you really executing that much extra fluff in a subshell? Yes. Yes you are.9
  5. make has a surprising number of useful functions; if you're spending any time doing build system stuff, read up on them. I myself was guilty of doing ifeq (exists,$(shell test -e $(FILE) && echo exists) before I learned about $(wildcard ...). The former spawns three processes. The latter: zero.

This is just one of the few makeovers10 that I'm working on; there's something in the works for the installer, and I'm working on some pretty fundamental changes to the build system itself.

(I call that project Build SystemNG, or BSNG for short.)

I'd be lying if I said I didn't enjoy working on something that people notice... either because it's noticeably faster or completely broken.

One of the two.

__________________________________
1the famed "rebuild the airplane while it's in flight"-problem, which I've decided is probably the most asinine analogy I've ever heard. So I'm going to stop using it.
2 As some might have you believe
3 Except, maybe, if you're paying attention to the amount of scotch consumed
4 I keep thinking everyone knows what this means, but I keep running into people who don't. Even people who I would think would... don't. So here's a convenia-link5 for it to Urban Dictionary, in case you don't.
5 Like a perma-link, only less permanent, and more convenient
6 Bug 15529 has all the details
7 I only had a VM to test those; I don't have the numbers in front of me, but as I remember, they were somewhere between 30 and 32%.
8 A benchmarking hint from Mecklenberg's O'Reilly GMake book, which I recommend
9Over half a million of them, actually... which is why tips-and-tricks are always helpful: everyone wants faster builds.
10 Hahahahhaa... make over? gmake? Get it?
11 Photo courtesy D.G.S..

Posted by preed at 1:24 AM | | Comments (2) | TrackBacks (0)

February 18, 2009

Back the change out... and nobody gets hurt


Remember what Preed The Build Engineer says:
"Bad Checkins Hold Everyone Hostage!"
This Valentine's Day1, the boyfriend and I bucked convention and took the opportunity to do something we had both randomly expressed interest in some weeks earlier, but neither of us had done before: learn how to shoot a gun.

We found a shooting range2, which offered a package deal that included a twenty minute how-to course4 followed by an hour on the range, to practice shooting a .22 caliber pistol at various things5.

(Interesting sociological note 1: a shooting range is a fascinating place to people watch while you're waiting for your class to start.)

(Interesting sociological note 2: apparently, shooting with your significant other on Valentine's Day is more common than one might think: we were joined by no less than six couples making use of an assortment of firearms.)

This particular range happened to be near KSFO, so afterward, we spent a few minutes watching planes land on the southeast plan, followed by a video game-related Fry's run and (altogether too much, but oh-so-tasty) thai food.

All in all, it was a pretty awesome way to spend a day with my Valentine.

***
When I got back to the office this week, I brought one of the targets in to show off my mad, GTA-honed6 aiming skillz.7

After joking around for a bit, it became clear that this particular target could be used as a visual aid in imparting a much more important lesson.

Because as we all should know: the sooner a sad, frowny build is fixed, the sooner we can douer it.8,10

_____________________
1 Or, depending on status, location, local restriction, and phase of moon, "Singles Awareness Day"
2 Ironically3 in the South Bay; south city to be exact
3 Or not?
4 Apparently, actually discharging a firearm pales in comparison to the moral, circumstantial, and legal issues often associated with doing so?
5 Ok, just one thing.
6 Interesting sociological (or maybe merely generational) note 3: of the five people in the class, more people had played GTA than had shot a gun
7 Why yes, that is a bullet hole in the center of his forehead; there's also one in his left eye and three centered squarely on the chest!
8 Where "douer it" is defined as "release the product"9
9 A complex, three step process that involves four steps
10 Is this a sufficient number of footnotes, Mr. Dolske?

Posted by preed at 10:33 PM | | Comments (4) | TrackBacks (0)

January 16, 2009

Sieben Dinge

Ack! I got tagged!

Planet has been awash in this meme the last few days, so I'll try to make mine short. Or interesting. Well... one of those, at least.

In roughly chronological order:

  1. I was born without a hip socket, a congenital birth defect that wasn't diagnosed until I was 17 months old and everyone started wondering why I hadn't started walking.

    They fixed it by breaking the hip bone—no high-impact plastics or metal here!— and letting the socket create itself as it healed. The body cast I was in for four months was a "minor" side-effect. Good times!

    I do have a scar; my mother was relieved that it started out about an inch long, but it grew along with my leg.

    No, you can't see it.


  2. As a junior and senior in high school, I was a reporter and columnist for the local newspaper.

    I wrote about a number of things which made me a class and high school administration favorite, including articles on unconstitutional searches, district-wide attendance rules which were not in accordance with state education law, and asking my girlfiend-of-the-time to prom.

    (That last one got the most reader feedback of all of my columns.)

  3. I spent the last three years of college on disciplinary probation.

    To make a long story short: When asked for it, I gave some not-so-glowing feedback on a new set of "responsible use" policies the campus IT department was drafting. Unrelated to this, I ran a port scan (woo nmap!) from my dorm to locate a company server I was working on at the time (I was scanning for an open ssh port).

    The IT department used this event to put my tits in the proverbial ringer (in the parlance of our times). And unrelated to that, the experience uncovered a lot of illegal behavior by the University's Judicial Affairs.

    I was eventually "convicted" in a campus kangaroo court where the "judge" conveniently turned out to be the same professor who had written the exact policy I had been charged with breaking.

    There's an [embarrassingly] dated website chronicling the whole event, including MP3s from the all judicial hearings (holy crap did they hate when I posted those!) I was also on the news a couple of times.

    I still get emails every so often from current students who find the website and want to tell me about how they're being railroaded by Judicial Affairs.

  4. The same year I was "convicted," I wrote a web application that over 40% of the 15,000+ students at my university ended up using to get slots in perpetually impacted courses.

    The system was called CRASH: Cal-Poly Robot-Assisted Scheduling Helper.

    My bachelor's thesis covered the history of the application. The university ran the service using my code for a couple quarters, the [awful, but functional] code was then open sourced, and about a year after I left, the University implemented the features directly within their registration system, due to outcry from students and professors.

  5. Like a few others apparently, I am a licensed pilot: airplane single engine land (ASEL) type rating with a high performance endorsement (but sadly, no instrument rating... yet).

    A lot of people know that; what you may not know is that I got interested in it because I originally wanted to be an air-traffic controller. I didn't go that route because I became interested in it during my last year in college and couldn't justify wasting a perfectly fine computer science degree to go push tin.

    I also couldn't justify the federal government dictating where I'd be living based upon its opaque staffing needs.

  6. I got started in Mozilla (and coincidentally my career path) in a pretty random way: I was 17 and perusing photos posted from the first Mozilla party and I kept seeing a person in these photos that looked interesting.

    I randomly started conversing with them via email. As we chatted more about what I was doing—sysadmin at an ISP and a contractor—they offered me an internship at Netscape over that summer. It was on the client product engineering build team.

    I learned a ton of lessons while I was there about life, love, and the pursuit of a repeatable, sustainable build and release engineering process. I know a lot of software engineers find it perverse, but I feel in love.

    Searching Bonsai with a regular expression of "preed" is imminently embarrassing.

  7. I have a guilty little pleasure that when people hear it, they tend to think "Wow... this guy is super nerdy and quite possibly somewhat insane..."

    Mixing—ha ha—two skills in life I wish I had, but don't, I blend electronica music air traffic control to make... interesting music.

    I enjoy it for a bunch of reasons, not the least of which that as a pilot, it's good practice to listen to (and decode) that stuff. Electronica tends to ebb and flow, so it lends itself well to what you find in air traffic control transmissions as well.

    I also find it great music when I need to multitask and do a lot of [typically build-related] work simultaneously: I find it easy to work to, I find myself working faster, and it puts me in the right mindset to complete tasks succinctly and correctly, lest I smack packets carrying precious bits into each other.

    You may think it's weird, but I have a few fans on the Internet who've discovered it and pester me for more mixes; one of them is a Mozilla Corp. QA'er.

    If you're curious, listening to the first ten minutes of this will give you a pretty good idea.


Since this has been bouncing around PMO for awhile now, I'm going to point the finger at other people, many of whom work on Mozilla-related apps and technology, but some of whom who don't and are just awesome people:

  1. John Gaunt: He used to have my job, but he got promoted to Bugzilla Janitor and Programmer Lie Detector
  2. Ali Rayl: I can't seem to find her blog, so maybe she'll tweet seven things we don't know; but she spends her days making sure the Bird doesn't suck and letting us know when it does...
  3. Nick Kreeger - Just call him "DJ Fail"
  4. Rob Lord: Chief Birder; beyond his eclectic decorating tastes (which I'm more-than-slightly jealous of), I know I know 7 things about him, but I also know there are way more than 7 things I don't know about him.
  5. Adam Fritzler: Adam does a lot of things (great photography among them) but will never be able to escape what he did first (if you're chatting on AIM using anything but AOL's client right now, you owe him).
  6. James Castañeda: James taught me that urban planning is a lot like programming, except it uses heavier things (like steel and concrete) and "refactoring" is slightly more "involved."
  7. Mark Chen: Mark (via the FDA) [among other things] makes sure your spinach isn't covered in poo. (He's also a first-class recycler!)

Posted by preed at 3:27 AM | | Comments (3) | TrackBacks (0)