preed's blah-blah-blahg

Probably one of the most common operations that build/release automation systems do is running other simple commands¹, and making sure that they succeeded. Sometimes, you end up wanting to parse the output of the command you ran in your native language. Maybe you want to log it to a file, too.

At first, you might think this is an easy problem: most scripting language give you a multitude of ways to do accomplish this seemingly trivial and common operation.

Except... they mostly leave something out. A lot of the most commonly-used methods for this—backticks and system()—really mess it up, save for the trivialist, "No I don't actually care if you ran the command I told you to run"-cases.

It turns out that if you want to call lots of commands in robust ways, it's a somewhat complicated process²

I'm generally defining "robust" as

• Handles command arguments safely and securely
• you can pass it a timeout value, and it'll do the right thing
• if some external force kills your subprocess, it'll do the right thing, and communicate as much information as possible about what happened, and
• you can easily get access to standard out and standard error (SEPARATELY³), and manipulate them to your liking
• Has other niceties, like letting you know how long the program took to execute and changing the working directory

None of Perl's⁴ built-in constructs do this for you, which is why there's a beautiful⁵ 263 lines of Perl I wrote almost two years ago to do this correctly. And it took about nine months to get all the corner cases correct and all of the bugs ironed out.⁶

Here at the Nest, we mostly use Python for build stuff, so when the need for "running a subprocess and collecting its output" came up, as it invariably does, I thought to myself "Well, with the Python book coming in at almost 1,600 pages, there's gotta be something that wraps all the ugly, operating system-level (and -specific!) functions to do this!"

It turns out there is: the subprocess module!

At first, I was ecstatic, but as I played around with it more, it turns out that it's completely and utterly useless for arbitrary data output sizes.

And I don't mean "arbitrary" in the "gigabytes of output"-sense, but "arbitrary" in the "whatever the size of your kernel's pipe() buffer"-sense.

Oh, and no... there isn't a desire to make the library actually... y'know... useful.

Oh Lazy Web⁷ please tell me I'm missing something very obvious, and that there is a Python module out there for me.

I have high hopes that this is, indeed, 2008, and I don't have to implement this myself.

Again.⁸

'Cause I don't wanna.

I really don't wanna.

________________________________
¹ Like "make" and "cp"
² No pun inten... oh, sure, whatever. Sure. Pun intended.
³ Or not.
⁴ Let us say...
⁵ If I do say so myself...
⁶ Well... not exactly, 'cause I just found a minor bug while perusing it now...
⁷ Dear Lazy Web: I cannot link to you because you are not defined on Wikipedia or Urban Dictionary. Plz to be fixing so I can has. <3 preed
⁸ For the fourth time in the third language...

Posted by preed at 5:51 PM | Permalink | Comments (11) | TrackBacks (0)

I think it is pretty well known that I am a sheep, and have a natural tendency to follow the crowd.

At home:

[preed@underworld ~]$ uname -a && history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
Linux underworld 2.6.19.7 #14 SMP Mon Nov 26 12:18:50 PST 2007 i686 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux
3870 mtail¹
1110 mutt
965 j²
717 imtail³
472 ps
374 fg
313 mocp
142 screen
139 cd
112 mail⁴

For comparison's sake, at work:

preed@preed-desktop:~$ uname -a && history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
Linux preed-desktop 2.6.22-14-generic #1 SMP Tue Feb 12 02:46:46 UTC 2008 x86_64 GNU/Linux
1043 svn
861 dir⁶
836 cd
267 more⁷
229 ssh
221 ls
200 vi
136 grep
134 fg
119 ps

It would seem I pretty much am only using my workstation for reading email and listening to music. At least my work history has some semblance of related productivity tools.

Apparently, I also use a ton of bash aliases.

Huh.

I think I tend to run "mtail" and "ps" as nervous ticks when I'm thinking about something...

__________________________
¹ mtail is a bash alias to tail -n 12 ~/.fetchmail.log ~/.procmail.log
² j is a bash alias to jobs
³ imtail is a bash alias to parse-timpslog -d ~/timpslog -a, which gives me a log from the AIM proxy I use
⁴ Not what you think; mail is aliased to mtail^1,5
5 First recursive footnote! Booyah!!
⁶ dir is a bash alias to ls -laih --color; yah, yah... I know.
⁷ more is a bash alias to less -X -I

Posted by preed at 9:23 AM | Permalink | Comments (1) | TrackBacks (0)

This comes a day late, but certainly not a dollar short...

Continue reading "Should've Been Born A Couple Days Earlier" »

Posted by preed at 9:46 PM | Permalink | Comments (3) | TrackBacks (0)

It occurred to me this weekend that I haven't written about going aloft in quite some time¹, which is especially egregious given that I'm a birder these days.

Multi-hundred dollar bread-bowl of clam chowder. And Mexican Coke.

The weather turned out to be beautiful!

I didn't have the time to really do any flight planning, so I ended up asking a close friend to head down to San Luis Obispo, the town of my alma mater, for a $600 bowl of clam chowder which, since I've flown the route more-fingers-than-I-have times, requires all of five minutes of flight planning³.

We ended up going down the coastal route instead of the inner-Salinas Valley route⁴, which provided great beach views for almost ninety minutes. Plus, I was able to enjoy the fifty-plus miles of visibility on the flight back by having my friend fly us home⁵.

My friend saw fit to capture the flight with his new camera, documenting both his expert pre-flight suggestion and a secret routine all pilots conduct before every flight, but that is an entirely unmentionable ritual of the industry.

The entire Flicker⁶ set here.

______________________________
¹ a couple of posts excluded from the count
² which is no one else in the known Government Holiday-world (tm) seems to get off, but on the other hand, it's smack dab in the middle of MLK-day and Memorial Day, so it turns out to be a better distribution of holiday-ness.
³ I mean, what's easier than KNUQ VPLEX SNS SARDO—you want to be sure to avoid restricted-2504—PRB direct?!!
⁴ KNUQ VPLEX KMRY BSR MQO direct, roughly...
⁵ I was somewhat mean; I turned the GPS off, and said "if we get to the Oregon or Nevada borders, I'll let ya know we've gone too far."
⁶ Non-web two-point-oh spelling

Posted by preed at 8:33 PM | Permalink | Comments (2) | TrackBacks (0)

One of the important roles of a build team is to help define, create, support, and use an official build environment.

In the case of a company working with a product that is open sourced, this can turn out to be a... colorful, shall we say, task, since it can often involve community members trying to build on "exotic"¹ platforms like AIX and BeOS.

MoCo uses virtualization in the build farm (and on the IT side, too) for operating systems work as guests², the result of a decision that made a lot of sense back when I was trying to get various important build systems-that-had-no-defined-environment off of shaky, failing hardware.³

As machines have gotten faster (and these days, as they have more cores), virtualization provides an easy, reasonably performant way to create, isolate, and distribute official build environments.⁴

Therefore, it wasn't a surprise to find VMs in production roles for build-related tasks here at the Nest.

One of the common complaints about virtualization is that a completely unmodified operating system image run under reasonable (or high) workloads tend to lose time.

Virtualization may seem somewhat magical, but there's a pretty simple reason for why this happens.⁵ In the good ol' days, VMs tended to only lose time, but now with the 2.6 Linux kernel trying to be smart⁶, VMs can gain time, too.

In our case, a whole lotta time.

The originally reported symptom of the issue was build deliverables with timestamps far in the future. After reading up on the current fixes to the time-syncing problem (which VMware tends to rev as operating systems change), I tried a number of the standard fixes, all of which had worked for me in the past. None of them worked.

We were still gaining two minutes for every ten minutes of wall clock time. That works out to almost one-and-a-quarter days per week the VM leapt into the future.

Something was obviously, seriously wrong with this machine. I've been working with VMs for quite a few years now, and I'd never seen clock skew this bad.

This was a 64-bit VM, and it turns out that the kernel options you pass to fix time skew on 64-bit Linux VMs differs from the options for 32-bit. But even the "correct" options didn't fix it.

After much Googling and forum trolling, someone suggested turning off the CPU frequency scaling daemon in the guest OS. This, of course, makes perfect sense, but the guest VM didn't have it on.

Then it hit me: we're using hosted virtualization. Originally, it didn't even dawn on me to poke around the host OS's settings, since MoCo uses the enterprise version of VMware⁷, and I was used to tackling VM problems from that angle. ESX runs VMware's own proprietary (vm)kernel⁸, which I'm sure turns all the whizbang features on modern CPUs that would confuse the hypervisor off.

Five minutes of poking around, and I found the cause of the problem: powernowd. It was running on the host OS and "helpfully" scaling the CPU for us.

To understand why this totally screwed us, an analogy may be in order: imagine, if you will, that you must drive a car at thirty mph down a street or it'll blow up. "Easy," you think to yourself. But imagine that someone else is erratically controlling the gas pedal and you have to manage your speed by applying opposite braking power. This is the original virtualization timing problem.

Now imagine that the car—an eight cylinder gas guzzler—randomly decides how many cylinders to fire at any given time. It might decide to use just two, or it might decide to start using all eight.

Now what do you think your chances of maintaining a steady thirty mph would be?

A quick apt-get remove, and the next day, the builds were not... "from the future," and the clock was still synced to the host (courtesy of vmware-guestd).

Sure, we're using a few more electrons... but at least their deaths are for a good cause: keeping space-time from ripping apart.

In the virtual machine's universe, at least.

____________________
¹ These days, anyway...
² [Insert complaint to Steve Jobs about lacking MacOSX Virtualization here.]
³ P2V to the rescue!
⁴ It also turns out to be cheaper and faster to have a SAN-backed, VMotion-capable build infrastructure than triply-redundant Mac Minis doing the same thing. I have to imagine it's certainly more reliable, and less of a physical-management headache.
⁵One of the core techniques involved in virtualization is playing with clock interrupts; this means you don't get a constant stream of clock interrupts, but rather (especially under load), you may get one or two interrupts followed by none for an interval, followed by 3 or 4 to catch you up. Of course, if you lose these interrupts over time, you'll lose time.
⁶ The 2.6 kernels contain a few algorithms to attempt to "recover" lost clock interrupts on physical hardware. This doesn't work in a VM, but it can cause the kernel to count interrupt that never existed, thus time moves quicker.
⁷ MoCo originally used hosted virtualization—GSX, to be specific, the product on which VMware Server is sorta-kinda based—as well; I whined (sorry Justin) until we tried out the server product.
⁸ Which, popular to contrary belief, is not a set of patches against RHEL; it merely uses RHEL to boot, and I've heard they can boot the vmkernel on its own in the labs in Palo Alto.

Posted by preed at 11:46 PM | Permalink | Comments (3) | TrackBacks (0)

Today was my first day at Songbird¹ as the new Manager of the Build/Release Team.

San Francisco International Airport's Runway two-eight-right Category III ILS Approach Procedure

Contrary to (popular?) irc.m.o legend, I didn't die or get run over by a compiler.

I was, however, spending some quality (and deserved, I think) time on my couch with various DVD box sets and, post-Christmas, my new PS3.²

I must admit, even though I spent most of today setting up my workstation and email and wikis and all that (important) stuff, I'm worried I may have looked like I was a little bit out of it: I was so excited about today, I only got about ninety minutes of actual, useful, real sleep before I had to get up at 6 am.

(Oh, and now I'm finding out all the various ins and outs of CalTrain schedules. I think I need some more practice, as I totally got on the wrong train going home today. But hey, I say we all need a tour of the Silicon Valley corridor from time to time to keep our appreciation fresh.)

I'm really looking forward to the new opportunities and challenges working in an environment like Songbird presents, but I think I'm most stoked about:

• Working on a project and product related to Mozilla and Mozilla's myriad technology.
• Once again being able to work on an open source project. I've always believed it's a privilege to be able to make a living working on open source. Plus, I think as software engineers and hackers, working in that context keeps us honest about our code... or at the very least humble. ;-)
• Working on a project whose focus is on open media on the web, which I admit I'm just beginning to really feel like I grok. As an admitted die-hard Linux user who now-has-a-commute and is getting a shiny-new-iPod-for-said-commute, having an open application to get various media from various sources onto that iPod, on an open operating system, is... really important to me.

Admittedly, this is just from my sense from a single day, one with way-not-enough-sleep. :-)

Anyway, I'll be hanging out on IRC—even moreso now³—so if you need to find me... just holler, and you'll get me.⁴

Still.

______________________
¹ Or Pioneers of the Inevitable—a name my mother absolutely loved when she heard it—if you prefer...
² Xbox-ers and Wii-ers: no making fun. I actually like it! And now that "Bluray has won," I'm even more pleased with it.
³ Which admittedly wasn't a ton the last couple months 'cause... well... come on... West Wing. And Arrested Development!
⁴ Or... Reed... one of us...

Posted by preed at 10:14 PM | Permalink | Comments (8) | TrackBacks (0)

Monday, October 22^nd, will be my last day at the Mozilla Corporation.

The San Francisco QUIET TWO Departure

Except, even that's not quite true, since I'm planning to continue in my role as the release module owner and am hoping to play an active part in its development.

So that doesn't work.

...

Well... I'll keep thinking about what should come after that sentence.

Until then, it definitely needs to be said that it's been a pleasure and a privilege working with everyone at the Corporation.

I'll still be hanging out on IRC, if you need to find me... just holler, and you'll get me.¹

______________________
¹ Or Reed... one of us...^2
2 You had to know I was gonna do some footnotes here...

Posted by preed at 11:51 AM | Permalink | Comments (8)

It occurred to me the other day that I haven't blogged about my IFR training in forevar.

I think that's more a function of I-stopped-doing-IFR-training-because-I-only-have-so-much-flying-monies- and-the-summer-weather-has-been-just-too-damn-beautiful-to-be-spending-it- under-a-training-hood than... y'know, any other factor.

In late August, I flew myself to Portland's¹ Hillsboro airport.

It's a six hour flight with a fuel stop, and cost over $1,000². Sure Alaska airlines would've gotten me there in ninety minutes at a quarter of the cost, but it wouldn't have been nearly as fun.

I wouldn't trade the experience for anything.

I learned more about myself, as a pilot, a software engineer, and an individual, than I have in... quite awhile. Of course, being aloft at 10,000 feet, flying along the Northern Californian coast, and then inland across western Oregon's beautiful, seemingly endless deep-green forests has a tendency to... be conducive to that kind of contemplative thought.

I highly recommend it if you've never had the opportunity. ;-)

I flew up (and back) alone³, but since it was my first flight out of both the state and beyond the boundaries of a Californian ARTCC—Helloooo Seattle Center!—my friends demanded that I have a camera (video and still), a voice recorder to grab all the interesting interaction with air traffic control, and that I file⁴ flight plans, so they could "fly along." I got some incredible pictures and some great footage.

I finally sat down this weekend to digitize some of the video.

This was on the way up to Portland, landing at KCEC, my fuel-stop airport. The video camera was placed on the dash of the plane, not held, so the camera depicts accurate movement of the aircraft itself.

Oh, and for those that don't readily do the conversion from knots to mph⁵, the (cross)winds were eighteen mph, gusting to about thirty mph⁶.

Every time I watch this video, I have to remind myself that the pilot is, in fact, me.

Sure, it's not my prettiest approach ever⁷... and I've certainly had smoother landings... but when push came to shove, it's a pretty concrete reminder that I do, in fact, possess the ability to handle the situation to a successful outcome⁸.

Of course, any situation can become overwhelming... in which cases, material like this video is useful in helping to figure out where the lines between a "fix it" and a go-around are.

Sometimes, in the heat of the moment, when the approach gets bumpy, it's easy to forget that one possesses those skills... and those options.

But there's nothing better than a beautiful, long flight with only you, your chariotplane, and the endless blue sky... to remind you that you do... and if you use them, it'll all be fine.

_______________________________
¹ Oregon, not Maine
² Yet more proof that you don't take up flying because it's faster or cheaper
³ Which means I was stuck doing my own drink and snack service...
⁴ and close
⁵ Like *cough* me...
⁶ Astute viewers will be probably be able to see the points at which I caught one of those nice, thirty mph gusts...
⁷Especially in retrospect.
⁸A testament to my instructor, to be sure.

Posted by preed at 6:33 PM | Permalink | Comments (1)

With last week's release of Thunderbird 2.0.0.6, we've finished a stint of security fire drills.

My friends and family have come to learn what that means¹, but I often get asked what it feels like to be in firedrill mode, especially on a [consumer-oriented-yet-open-source] project like Firefox. I've been trying to come up with a good way to explain it, and I think I may have finally found something to help.

The idea came to me when I noticed that a scene from a particular movie kept flashing through my head, very distinctly and vividly, while working on various release activities during this last drill.

It's from a 70's movie called The China Syndrome², which I found a conveniently-You-Tubed³ clip of. You'll want to let the full clip load, since the part that flashes through my head most commonly in the last thirty seconds or so, starting at about 3:35 in.

After watching this, a friend asked "So... you're comparing firedrill-mode to a nuclear meltdown?"⁴ We shared a laugh about that, because it wasn't the analogy I was intending at all.⁵ What the scene does capture pretty perfectly is the sense of urgency and the mode the characters go into when "the alarms go off."

During a firedrill, there's bits and pieces of information coming in from all sources and angles; there's a level of coordination involved in terms of getting issues looked at and prepping for builds; there are usually always external people watching [from above] what we're doing and how we're doing it; sometimes, we have to address problems creatively, even though "the book says you can't do it"; there can often be conflicting information that makes making a decision difficult. Many of the roles, reactions, and interactions of the characters in the scene are reminiscent of how we get a firedrill out the door.

I actually hadn't noticed that until I watched the clip again, since I originally identified with the last thirty seconds⁶ and hence why it tended to flash through my head at weird hours while starting at Tinderboxen prompts.

Every component, everyone one of them, is tested, and then it's tested again. Every critical weld is radiographed. Every single thing is checked, double-checked, it's rechecked. Everything. ... In any thing that man ever does, there's some element of risk, right? That's why we have what we call "defense in depth." Now that means, backup systems to backup systems. To. Backup. Systems. ... The system works. Even with a faulty relay, even with a stuck valve, that system works.

I'm sure QA can certainly relate to the amount of testing we do to ensure we ship a quality product... and a number of those tests mesh with verification and testing we do on the release side. In our case, it's more "even with an invalid automatic update snippet, even with a bogus locale repack, we catch the error and correct them before the release."

We learn something new from every firedrill we do.

And as a result, we've got a lot of newer initiatives all over the project to make firedrills more smoother and faster, from automated regression testing in Tinderboxen to automated smoketests and BFTs to automated release deliverable verification. These are all going to improve on Firefox 2's already-pretty-awesome patch-delivery time.

The analogy I make here illustrates how stressful it can be for that time we're getting it out. All the stuff I mentioned above will reduce that stress, but I think at least some of it is self-induced⁸: we take this stuff and the way we respond to and engage in them very seriously, and I know I'm not the only one: everyone, from the developers focusing on the fix to the release managers coordinating everything to the QA engineers verifying the fix and looking for regressions, to the build engineers pushing the bits out experience this responsibility together. (In fact, as I watch this clip again, I identify with Lemmon's character at the end, but in the beginning of the clip, when he gets up from his desk as the alarms initially go off, an image of dveditz pops into my head for some reason... :-)

It's part of the commitment to security and, more generally, doing right by our users that not only defines Mozilla and our community, but that I know we all personally hold dear.

So maybe no matter how good we get at this, there will always be little alarm klaxons going off in my head when I hear the callemail come in.

But... maybe... just a little bit of that... is a healthy thing.

_______________________
¹ "Preed's gonna 'go missing' for awhile... don't ask..."
² If you haven't seen The China Syndrome, I highly recommend it... if
for no other reason than getting to see a young pre-activist Jane Fonda and 70s-haired Michael Douglas. And of course, any movie with Jack Lemmon is A++++ WOULD WATCH AGAIN.
³ Oh how I love thee, copyright infringement fair use!
⁴ I don't even want to think about the Slashdot headliner on that one...
⁵ Maybe I didn't make the connection because—not to spoil the movie if you haven't seen it—but there is no meltdown in the movie
⁶ We've striven for correctness on the common cases with the release automation, and in firedrill-mode, we can be faced with situations that aren't entirely common... so there's been some level of back-to-non-automated/manual release work. Performing certain staps manually has resulted in (somewhat familiar) typos, errors, and general suboptimality⁷ that you generally expect with a human—me—typing lots of commands in a magical, ordered incantation. When that happens, and it gets caught by QA, my reaction is... well... it's why I keep seeing this scene in my head. :-)
⁷ Is that even a word?
⁸ At least, I know for me it is... and yes, it's a bad habit. :-)

Posted by preed at 5:04 PM | Permalink | Comments (3)

After having been conned convinced to buy an iPhone by ss on opening night, I finally bit the bullet and activated it a couple of days later (using a Mac... sigh), and have been using it for a couple of weeks now.

Admittedly, I've had more than a few "iPhone moments." I think my favorite was when I was heading home from an early-evening appointment and a hankering for Chinese food snuck up (you know how these things are). There's a place I always go to, so I Google-directioned them on my iPhone; the entry included their phone number. I called, placed my order, and by the time I got over there, which was on my way home anyway, my food was ready to go.

Welcome to iLife 2.0!

With that...

iPros

[Continued...]

Continue reading "iInitial iPhone iImpressions" »

Posted by preed at 2:12 AM | Permalink | Comments (6)

While doing a Bay Tour this weekend, I had an interesting first-time-aviation-experience (tm).

First-time-aviation-experiences (tm) tend to cause much after-the-fact pondering, because they typically turn out one of two ways:

a. Everything goes okay, you make it back to the earth, and you have a war story to tell in the hanger next time when it's IFR and rainy and everyone's having a beer instead of flying.

2. It... uhhh... doesn't go so okay... and for some value of "doesn't matter anymore," it... well... doesn't matter anymore.

I've had a couple of "first-time-aviation" experiences (mostly as a student pilot) that [I now sorta laugh at, but] at the time, they were a big deal.

While on the tour this weekend, we hit some pretty bad turbulence around Mount San Bruno, near San Francisco International. It was so bad, in fact, that we dropped about 400 feet in ten seconds¹. Arguably², I did precisely the right thing, and sacrificed altitude for attitude control of the aircraft, but at the expense of... comfortable flying.

It turned out to be a problem for one of my passengers.³

In retrospect, while the experience wasn't really all that big of a deal^4,5, it did present a new situation which I haven't had to deal with yet: "How do I get this plane on the ground, as soon as possible, in crowded airspace?" It turns out the exercise is useful for dealing with "other conditions"⁶ which pilots must be prepared at anytime to deal with.

To narrate this experience, I isolated⁷ an air traffic control clip from Saturday's flight; it's about three minutes long, and you can hear us—Cessna seven-five-one-sierra-papa—talking to San Francisco Tower and NORCAL Departure⁸:

2:40 96 kbps MP3.

The relevant portion starts about two minutes into the clip:

Me: Uhh, NORCAL, Cessna seven-five-one-sierra-papa, we need uhh... to do uhh... landing in Oakland, uhm so we're turning towards Oakland, we don't have the numbers.

NORCAL Departure: Cessna seven-five-one-sierra-papa, okay, is everything okay?

Me: Uhh, yah, we just uhh... have someone who's not feeling great, so we're gonna land in Oakland real quick before going back to Palo Alto.

NORCAL Departure: Okay, let me uhh... see if I can get direct for you. Air Canada five-eighty-one, fly heading three-four-zero, proceed direct Red Bluff when able.

Air Canada 581: Heading three-four-zero, Red Bluff when able, Air Canada five-eighty-one.

NORCAL Departure: seven-five-one-sierra-papa, you may proceed direct, altitude your discretion, contact Oakland Tower on one-one-eight-point-three, have a good day.

Me: Thanks for your help, direct Oakland, eighteen-three for the Tower, seven-five-one-sierra-papa.

What's interesting about this interaction⁹ is that the air traffic controller basically cleared us directly under Oakland Airport's departure path, so that we could get on the ground sooner (about half the time it would normally take). It required a measurable amount of extra work on her part to call Oakland Tower on the telephone and request—and receive—a clearance for us to fly directly to the airport. When we got there, we were put ahead of the line to land.

This is one element of our air traffic control system that just always amazes me and gives me warm fuzzies at the same time: you can be a two-seater 152, but the second you declare an emergency, they'll push 747s full of people out of your way if you need them to.

I've heard it happen.

In many ways, it reminds me of one of the most important aspects, in my opinion, of the Mozilla Community which is so hard to convey to others, but which makes being part of and involved in our Community so rewarding.

________________________________
¹ Do the math on the acceleration.
² Justin is the only one arguing with me... ;-)
³ It should be made clear: the passenger was apologetic to a fault which, while endearing, was unnecessary. I tried to reassure them by recounting the story of life as a student pilot, where I continuously got sick every time we tried to do "ground reference maneuvers."
⁴ The failure condition, while somewhat... uhh... disgusting... wasn't life-threatening, really.
⁵ But I'm sure it was a big deal for the passenger in the back seat
⁶ ENGINE FAILURE
⁷ Which is a fancy way of saying I removed all the other stuff so it looks like navigating San Francisco Airport's airspace is really friggin' easy.
⁸ For some reason, all my friends comment that I sound "butch"—their words, not mine— on the radio. My instructor claimed I sound like a "grizzled Eastern Airlines captain." I really don't see it.
⁹ I've italicized the relevant statements.

Posted by preed at 11:00 PM | Permalink | Comments (1)

The punchline: my shiny new iPhone is currently sitting on my kitchen table, resembling something of an iBrick.

The joke:

[Continued.]

Continue reading "Too old for [i]Phone two-oh?" »

Posted by preed at 4:19 PM | Permalink | Comments (7)

So... a few weeks ago, I finally broke down and got a Facebook account.

After having spent some time filling in all the little "tidbits about me,"¹ and more or less trying to interact with... "it" over the past couple of weeks, I've decided I don't really grok Facebook.

The first thing I noticed about Facebook is that it is very... SYN-ACK heavy.

[Continued...]

Continue reading "Too old for web two-oh?" »

Posted by preed at 8:56 PM | Permalink | Comments (8)

With a time-warpish flight, the Nomadic Build Engineer '07 Tour has come to an end.

I think United is playing games with the flight times on this flight: they claim it's about a five and a half hour flight on the plane, it's published as a five hour, forty-five minute flight, we left JFK almost fifty minutes late, and got into San Francisco a half-hour early.

Yah, I don't get it either.

I flew United home, so of course I listened to channel 9 (the ATC audio channel) and got to enjoy the beauty of the nation's aviation transportation infrastructure again.

No matter how many times I fly commercially, I'm always amazed that "RBV J230 LARRI J230 AIR J80 EMPTY SPI J80 HLC HGO J197 DVC ILC RUMPS OAL MOD3" gets turned into:

We got to see a bunch of big planes skid the turn to one-three-left today; they tend to keep their speed up, because they all have traffic on their tails, so it's very interesting to see them come out of the turn, drop everything in, and plant it on the runway.

Especially when the plane is something like a 747 or A340.

A 20 knot crosswind kept things interesting.

I was gonna grab some of the clips from air traffic control that kept things... bearable (but probably only for me... since I was seemingly the only one sitting in my seat, giggling) while we sat and waited for fifty minutes, but it turns out that JFK's ATC Internet feed is multiplexed, so... the joke about the pilot asking "A few whats between friends?" isn't as funny when the recording missed the tower controller saying "What's a few digs between friends?"

Lamentably, it also only grabbed half of the conversation about the pilots "turning into pumpkins" in a half hour.

And Tony: I promise next time, we'll let you make omelets.

Posted by preed at 12:14 AM | Permalink | Comments (3)

So, we can't comment our code the way we see fit anymore?

Didn't we already go through this once?

I think the most logically inconsistent part of the bug is the whole "You can't say certain 'bad words,' but we'll still help you spell them"-thing.

This seems like a losing proposition. As censorship of any form typically turns out to be, if you integrate over a large enough time interval.

I might just have to add an $equalAmount++ of "that word" to my next patcher2 patch.

Posted by preed at 8:31 AM | Permalink | Comments (1)

On Memorial Day, The ReedBot^TM and I went on a flight down to Oceano Airport for lunch.

Section 8.1.2.b, paragraphs 2, 3, 4, and 5 of the Airmen's Information Manual say, in part:

Either an upper respiratory infection, such as a cold or sore throat ... can produce enough congestion around the eustachian tube to make [pressure] equalization difficult. Consequently, the difference in pressure between the middle ear and aircraft cabin can build up to a level that will hold the eustachian tube closed, making equalization difficult if not impossible. The problem is commonly referred to as an "ear block." An ear block produces severe ear pain and loss of hearing that can last from several hours to several days. Rupture of the ear drum can occur in flight or after landing. Fluid can accumulate in the middle ear and become infected. An ear block is prevented by not flying with an upper respiratory infection or nasal allergic condition. ... If an ear block does not clear shortly after landing, a physician should be consulted.

Having just recovered from a nasty cold the Sunday morning before, that was (what some would call) the first mistake of the flight.

(Some might call spending two hours stuck in an airplane next to Reed the first mistake, but I assure you, he was a model passenger, and I'm proud to report I won't be invoking the "interference with a flight crew felony" clause—which applies to general aviation aircraft as well, thank god—the airlines always cite.)

Since I'm going on Day Five of not being able to hear out of my right ear, it may have turned out to be the worst mistake of the flight... but arguably, I didn't know it at the time.

Continue reading "Head in the Clouds, Golf (Now, with 100% more Mozilla Intern (tm)!)" »

Posted by preed at 11:31 PM | Permalink | Comments (4)

You ever have one of those weeks where it feels like... like... you did a bunch of... stuff... but you didn't really get anything actually done?

Yeah, that was me this week.

I got assigned "light and airy" bugs like "Nightly updates broken - all apps, branches, platforms."

I also spent a lot of the week trying to turn:

Exhibit A

into...

Exhibit B

Now, if you don't know what those pictures mean, that's OK... because after having worked for most of a day fixing it, I really don't understand how it works, what I did to fix it, how I would fix it again... or what the pictures actually mean.

All I know is there's a bunch of red in the first picture... and there's a bunch of green in the second picture. And green is... good.

I think.

(Right?)

Anyway, in the end, that thing they say about it not being the destination, but the journey... just might be true.

Posted by preed at 7:35 PM | Permalink | Comments (1)

I know I have a bit of a... reputation for being up at weird hours. But lately, I've been falling into a work pattern that... has some good elements to it, but probably isn't sustainable.

The main cause of this has been cf.¹

Since he's working across the pond, he tends to be up and responding to email/IRC around 1 or 2 am and falling offline around noon to 1 pm PDT. (Astute readers will notice that this is an 11 hour day... which... now I feel really bad about some of the meetings we ask him to be at.)

He's been a really good sport about this and we got spoiled because he was here for a few weeks just a couple of weeks ago², so we typically just expect that he'll be able to be in meetings (because he was in them while he was here... and now he's "just a phone call away!")

I've always been a late(r) riser, so I've noticed that if I don't touch bases with Nick until I'm booted up, then he'd be parking his brain's drive heads for the day. I've found that to really be effective in helping him with things and doing reviews, I tend to need to stay up until 2 or 3 in the morning, when it's his morning, and try to be around earlier in the PDT day, to catch him at the end of his day.

On certain days (that tend to be coincident with the days I set up my coffeemaker's timer), this turns out to be workable; I'll go to bed around 4 and get up around 8 to the smell of fresh brewed coffee. It affords me the opportunity to catch up on IRC, email, and the fires that flared up overnight, and deal with them before coming into the office and trying to get something done.

But if I don't set up the coffeemaker, I find myself getting up at 10 or sometimes even 11, and not getting into the office until noon or sometimes even 1.

I don't like this for a number of reasons.³ The largest one is that it tends to fragment my work day too much, I think.

Anyway, this is relevant because the weekly build team meetings have traditionally been at 2 pm Pacific... and this turns out to be suboptimal, but workable for East Coast peeps. For Nick, it's positively hell.⁵

We've been trying to find a new time for it, but we've run into conflicts with other "random" meetings. ;-)

It looks like we've found a place that may work, but it's a 10 am meeting... and I've always hated any meetings before 11 am (what with getting up, getting into the office, and getting caffeinated, I tend to not be very useful before 11 am anyway).

The upshot is that I think I'm probably going to have to find a way to start shifting my days earlier on principle. I think this is going to be difficult for me because I've always been a night owl⁶, and going to bed at, say 10 or 11 pm, goes against at least fifteen years of going to bed at 2 am. Or later.

I think one of the hardest things, though, about going to bed before midnight is that it would make me feel like... too much of an adult. "Hey, I can still stay up until 3 am, and get up for a 7 am class, and be useful!!"

Except I can't.

Anyway, in the next couple of months, it'll be interesting to see how the meeting schedules shift... I'll also be curious to see if the rest of the build team begins to move "earlier" in the [Pacific Time] Day, since we've got Coop and Nick tugging that way for useful collaboration.

It's not like this is a new problem for the Mozilla Project, but I'll be curious to see how we, as a team of people who work closely together, end up solving it for ourselves.

And now, a friend from New York just IM'd me... so it's probably really time to get to bed now, so I can hide for an hour or two from a westward-marching sun.

___________________________
¹ I'm sorry Nick! I still love you though!
² Or so it feels like...
³ Not the least of which being it makes me feel like⁴ a huge deadbeat loser.
⁴ [other people are thinking I'm]
⁵ More than a build meeting already is, since he's currently working on the 2004/150twelve release.
⁶ And I've recently gotten back into hacking by candlelight... which I used to do all the time in college, but had forgotten how... awesome that is.

Posted by preed at 1:45 AM | Permalink | Comments (5)

Joel—yes, that Joel—made a couple of blog posts recently reminding people that today is Mother's day.

(Tangential aside: Hi Mom! I didn't turn out to be a famous movie star... or a wünderkind athlete... or a renowned writer... or a Nobel prize-winning scientist... but this blog [seemingly?] [and inexplicably?] gets syndicated all over the place, so if you Google for "preed" and "mother's day love" in like... a couple of weeks, this post should show up... and since I can't dedicate anything to you, like those other professions tend to lend themselves to, hopefully seeing those Google search results will remind you of how much I love you, not just today, but everyday!)

What I thought particularly interesting about both of his reminders was

a. He's offering a free day usage of his company's Copilot software today¹, so us tech-savvy children can help our Mothers update their vulnerability pockmarked Windows PCs.

and b. in both posts, he mentions installing Firefox on her computer, so she can have a cleaner, faster, more customizable, and (most importantly) safer experience on the Web.

I happen to be in Colorado right now, so I can accomplish those things with my Mom at the kitchen table²... but if you don't have the luxury of being within fifty feet of your Mother today, here's a chance to tidy up her PC a bit and make sure she stays safe on the Web.

So if you haven't already, take a few minutes to introduce your Mom to Firefox. She'll thank you for it, even if she doesn't entirely understand why.³

(My mom notes every time I see her that every few weeks, she's pleasantly reminded of me when she gets the "It's time to update Firefox" popup. For most of our users, it's an annoying⁴ distraction to using the web, but for her, it's like her son⁵ is giving her a hug from three states away.)

________________________
¹ And on Father's Day, too
² Actually, since Mom already uses Firefox (of course!), the big task for today is teaching her how to use flickr, since she got a digital camera for Christmas, but all the pictures she tells me about having supposedly taken have seemingly gone into a black hole!
³ Explaining Firefox extensions to my mother took awhile, but at the end of the conversation, she was very appreciative of shaver, morgamic, and everyone else's work on AMO3!
⁴ but necessary
⁵ Even though rhelmer and cf push releases as often as I do...^6
6 rhelmer and cf: please promise to be nice while giving my mother AUS2-hugs

Posted by preed at 11:05 PM | Permalink

Astute IRCers may have noticed I've been on vacation. I'm now home, chillaxing¹ after having spent a few sunny days in Las Vegas.

It was my second time visiting the City of Lost Wages. I met up with a friend from the right coast and a couple more (local) friends joined us the next day.

This trip was more fun than last time, I think mostly because we did things other than gambling this trip around. We visited Hoover Dam² in addition to actually going to see a few shows.

I think my favorite was La Reve, but we also saw Blue Man Group, which was unexpectedly engaging and fun. We also randomly happened to catch Penn & Teller, an act whom it's hard to go wrong with.

Being the plane geeks that we all are, we also spent the last night at the pricey³ MGM Grand, with room overlooking KLAS's one-eights.⁴

We also sampled a bunch of buffets. The Bellagio's, while somewhat pricey, still stands as the King of Buffets. (Never eat at The Trop. You've been warned. In fact, you're better of ignoring, for the purposes of visiting Vegas, that the whole hotel exists.)

I also learned how to play blackjack in not an entirely lose-all-your-money manner. The first time I was in Vegas, I was a bit too intimidated to sit down and be required to interact with a real dealer. This trip, I learned that some dealers can be jerks, but for the most part, they're generally nice. I suppose I should've expected this, since you're basically there to give them all your money... and it would behoove them to be not-jerks.

In one of the hotels we stayed in, there was a big scary sign about removing items from the mini-bar; one of the friends who showed up the next day didn't read the sign, and started pulling a bunch of things out of the bar to inspect what type of booze-ahol they were, and then put them back. "Ooops."

At one of the friendlier blackjack tables, one of the players was asking what we all did. I mentioned I worked on Firefox, and he said he'd heard of it, but—once again—asked why he should use it.

I don't know if it was because of the free drinks or because I was up a few hundred bucks at that point, but I... remembered my friend digging through the hotel room minibar and spouted what I think would make a pretty good new slogan:

I'm sure pkim and cbeard will love this⁵

The whole table couldn't stop laughing. Hopefully that'll turn into a couple new users.

Since I did not hit the $16.8 million Vegas Bucks^TM jackpot, nor make a million playing table blackjack, I will be back, reading email and on IRC tomorrow.

__________________
¹ As the youngins these days call it
² "For all your dam needs!"
³ But actually worth it
⁴ I'm the one in the picture in the chair, with the martini, operating the radio. Obviously.
⁵ Lamentably, I am without camera. Still. So Lee L (??) provided this image, via Google Image search.

Posted by preed at 11:06 PM | Permalink | Comments (5)

I was looking for a specific air traffic control clip (more on that later), but I randomly found this clip of KJFK Ground Control.

It's kinda long (12 minutes), but it's got tons of great lines; highly amusing.

Upon hearing it, a friend said the New York accent makes it all the better.

There are a couple of points in the clip where I just expected him to blurt out: "I can't f@#*(! make my mind work...," ala Winnebago Man.

"We'll visit you in the duty free..."¹

Definitely recommended.

________________________
¹ Makes you wonder: can you bribe ground controllers the same way you can bribe build engineers?

Posted by preed at 4:54 PM | Permalink | Comments (4)

Over a month ago now, I blogged about the... "hilarity" that ensued when I moved into a (pretty nice) new apartment that... had no Internet (or cable, for that matter) connection to speak of.

In the ensuing month+, I've learned the following things:

• My apartment turned out to be the only apartment in the entire F#!(*!!9#% complex that didn't have a cable connection. Go figure.
• Google Wifi is... ok. But you can't really rely on it too much. I think it only worked a few times because there were low clouds, or something, because it never really worked again after I blogged about it.

  On the other hand, I never got an external antenna, and I don't assume it's supposed to work from your bedroom.

• Home owner's associations move slowly.
• Contractors that are "the cheapest bid" move even slower.
• Drilling holes in concrete to run eensy-weensy wires through is fun!

  Oh wait, no... what's the complete opposite of fun? Oh yeah... loud.

• When you have no TV and no Internets, your DVD collection becomes relatively... important. And DVDs you had forgotten you owned become surprisingly interesting again!
• Despite this, you can only watch so much [[QaF]] before... you just can't really watch it anymore.

  I never thought I'd get there, but... here we are.

• At some point, you start to care less about the "why" to the whole question about "wires being cut to your apartment in some painting/remodeling accident" and you start to care more about the "What the H&#@$! are you going to do about it in a not-six-month-timeframe?"-question
• Planning a flight without innerwebs is hard. And has not-so-great results (in terms of less-pilot-stress; in terms of artistic beauty, the results were spectacular...)
• When the innerwebs return to you after being gone for a month,
  
  • The Flickrsphere is brighter.
  
  
  • The You-Tubes are tubier.
  
  
  • The Blah-go-sphere is emo-er.
  
  
  • The bugs are bugier.
  
  
  • The torrents seem torrent-ier.

  It's like I'm a teenager again, running pine on an OSF/1 box for the first time.

  And everything is new.

There's probably something to be said for the fact that I haven't really felt as "moved in" to this new place until this evening, when I could actually write email(s) from my couch. There's probably something... pathological about that.

But for now, I'm going to bask in the niceness of it.

More as I catch up.

On everything.

(Interesting factoid: it literally took my computer at home some 30+ minutes to process my entire mailspool with spamassassin; and my computer is a 3 GHz Pentium 4, so it's not like it's slow. I also found about 10 emails that I hadn't seen among all the spam. That's... wow... in a depressing sort of way.)

Posted by preed at 12:42 AM | Permalink

A lot of interesting things happen to you as a pilot at the 90-day mark of being stuck on the ground, both from a government/regulatory standpoint and a club standpoint.

The weather here has been pretty not-VFR for the last couple of weeks, and I've been getting antsy about the fact that I may be getting close to hitting the 90-day mark. I pulled my logbook the other day to find out when I last went up, and it was January 24th, for a Bay Tour (with mento, actually!)

I'm not even close to hitting the 90 day mark, but I've been on the ground so long, it feels like it's been 90 days.

Yeah, I think it's pretty clear I'm hopeless.

Fortunately, looks like I'll be going up today, which is good, because I need to remind myself how to land.

(Surprising factoid: it turns out that finding yourself 1000 feet above a runway is a pretty good motivator to remembering [or figuring it out again], because... you're coming down one way or another, eventually.)

That's right, a cat.

Continue reading "Head in the Clouds, Delta" »

Posted by preed at 12:07 PM | Permalink

Having spent most of my weekend moving¹, I haven't had much time to miss the fact that I don't yet have Innerwebs at the New Hotness.²

Being the geek that I am, I of course called Comcast to set up a deactivate/reactivate service appointment, and they promptly showed up on moving day, but then after about a rousing twenty-minute round of "coax cable hunt," followed by bothering my [new] downstairs neighbor to find out why the cable goes into her balcony, but does not pop out of the floor of mine, I get some sordid tale involving remodeling-gone-wrong-three-years-ago-and-the-cable-was-broken-after-that-
and-you'll-have-to-drill-through-concrete-and-that-involves-talking-to-the-homeowner's-
association-but-they-only-meet-once-every-three-decades.

Or something like that.

Not such a huge deal since I've been busy packing, unpacking, and repacking. But since I decided to take it easy tonight, I really started jonesing for the Series of Tubes.

On a lark, I decided to see if there were any open Whee-fee access points. I'm still in Mountain View, so I wasn't too surprised when my laptop associated with an access point called "GoogleWiFi."

I had never used it before, but knowing Google and their penchant for recording-every-single-thing-any-user-ever-searches-for, I wasn't very surprised when they wanted me to log in with my Gmail account before I could do anything.

In some sense, while it kinda grates on me, I admit that I can't complain about it too much. I'll also openly say: I'm pretty impressed with the quality of the service³; and they don't seem to filter out things like outbound ssh and such.⁴

(It also makes me wonder how many Mountain View-ites are completely basking in Google-backed electromagnetic radiation, since I have no idea where the Google access point is, but I'm writing this from my bed in my new apartment, and I'm getting an average of 44 msec ping times to the default gateway with about four percent packet loss.)

But none of that is actually what I was thinking about when I started writing this.

No, what prompted this particular episode of blahginess was the blatant [and somewhat unsettling] realization of how much I happen to rely on a particular entity (to say nothing of that entity being a privately traded corporation) to provide me with a chunk of my daily Internet experience.

This realization took form after I clicked on someone's Picassa web album, linked to from a blog post indexed by Google Reader, all served up by [some AP claiming to be] GoogleWiFi.

As I traced back my steps of my quick checkin this evening, I've spent most of it on Google properties: Gmail, Reader, Google News.

And I don't know how I feel about that.⁵

Either way, it now makes the prospect of those saying there will be a "shadow GoogleNet" that we'll all be using in 5ish years less... insane-sounding.

But, until the home owner's association can get its butt in gear to get stuff fixed⁶, I'm pleasantly surprised such a service exists and actually works... and will gladly use it to let you all know that I'm somewhat troubled by the thoughts I'm having while mulling over its privacy- and future-of-the-Internet-implications as I do so.

____________________
¹ One almost-down, five to go
² As rhelmer would tell you, this has been my new catch phrase over the last week. "New Vista-Signing Hotness." "New BuildBot Hotness." Etc. Yes, It's pretty 1995-ish.
³ Which is to say it actually works, and is usable
⁴ Which, if I really cared about the privacy implications, I would shunt all of my web traffic through an ssh proxy. I won't, however, be using this particular link to be doing any Interwebs banking.
⁵ Of course, it can't be very great and/or squee-u-lar, since I took the time to write this...
⁶ That's not me being purposefully vague; I don't think anyone really knows what the problem is, and thus any solutions are still scary and mysterious.

Posted by preed at 11:39 PM | Permalink | Comments (5)

My IFR clearance—a concept which I've been meaning to devote an entire bloggity-blog post to—was issued by air traffic control as follows:

Cessna three-two-three-romeo-foxtrot is cleared to the Napa County Airport via: on takeoff, right turn, heading zero-six-zero within one nautical mile of the airport, radar vectors IMPLY intersection, Victor one-oh-seven, Oakland, Victor one-ninety-five, CROIT intersection, Victor one-oh-eight, Scaggs Island, direct. Climb and maintain three thousand; expect five thousand, five minutes after departure. Departure frequency is one-two-one-point-three, squawk zero-three-two-three.

It took me about ten minutes to set up the departure.

Everything went mostly fine... until my vacuum pump failed.

Then it got interesting.

Continue reading "Head in the Clouds, Charlie" »

Posted by preed at 11:25 PM | Permalink | Comments (2)

The FAA, in its infinite wisdom—and, for once, I'm actually not being sarcastic while referring to the concepts of "the FAA" and "wisdom"—allows you to record up to twenty hours of your required forty hours of in-IFR-conditions training to be in a sim. This tends to help out, since flying a sim is around $50/hour (depending on the sim) and flying a real plane is $130+/hour (depending on the plane).

As such, I've been putting some amount of sim time in lately, and I think I've begun to discover an interesting secret of IFR flying.

Its application(s) to Real Life (tm), if any, are left up as an exercise to the reader.

Continue reading "Head in the Clouds, Bravo" »

Posted by preed at 3:01 AM | Permalink

Before I present my open letter, some context is necessary.

From #foxymonkies:

00:23 <@Ryan> @preed-scotch: Drinking at the office? :0
00:24 <@preed-scotch> Ryan: hell yeah.
00:24 <@preed-scotch> I'm stuck here until 4 am.
00:24 <@preed-scotch> What would *you* be doing if you wer stuck here until 4 am.
00:24 <@Ryan> ouch
00:24 <@preed-scotch> so
00:24 <@preed-scotch> I've got me some Aqua Teen Hunger Force DVDs
00:24 <@preed-scotch> and three bottles of scotch
00:24 <@preed-scotch> and three hours to kill.
00:25 <@preed-scotch> time to get tore up.
00:25 <@preed-scotch> followed immediately by signing Win32 builds.

Dear Cygwin,

Please kindly FOaD.

Love,
preed

Posted by preed at 12:40 AM | Permalink | Comments (6)

I had a couple of people ask me about a recent post.

Their inquiry basically amounted to "Woah... are you ok?"

Truth be told, I was appreciating the picture on a "sometimes you can't quite see where you're going, but if you have the right tools, you'll probably be OK"-level, not an apocalyptic "OMG We're all going to die!"-level.

Plus, as I said, I really enjoy the composition.

I didn't do any such thing for my private pilot's license, probably because I had so many things going on in my life when I started, and I dragged it out for so long... but I thought I'd try and write a bit about my experience getting my IFR ticket...

(I'll always make these as mostly-extended posts, since they're not particularly Mozilla-specific.)

Continue reading "Head in the Clouds, Alpha" »

Posted by preed at 1:09 AM | Permalink | Comments (2)

Sometimes, you see a random photo whose composition and perspective all just click... and all you can think to yourself is "Yeah... that's my life right now... that's me."¹

It's one of life's better [surprise] pleasures.

_____________________
¹ Photo courtesy of my friend Mid,² who invited me to go plane watching today, but I declined. Maybe that was a bad call.
² The official photographer on all my flights...

Posted by preed at 1:29 AM | Permalink

Dear Hawthorne Automated Flight Service Station:

I'm sorry.

No, seriously, I really do apologize.

Continue reading "An Open Letter, Alpha" »

Posted by preed at 10:16 PM | Permalink | Comments (2)

Yesterday, I talked a bit about my 2007 thus far.

Schrep and I were discussing New Years Resolutions yesterday, and