" /> preed's blah-blah-blahg: December 2006 Archives

« November 2006 | Main | January 2007 »

December 20, 2006

Compiling in a Winter Wonderland

I've got a secret to confess: I've been working remotely from my childhood home in Colorado this week. I flew out last Sunday.

That whole plan turned out to be really fortuitous, considering this afternoon's Doppler radar:

Me in the center, with all the badness heading in the wrong direction...

This storm, easily in the top ten Colorado blizzards in recorded history the local news networks tell me, has shut down I-25 statewide, shutdown I-70 up into the high country, and completely shutdown Denver International, the "all weather" airport:

07:07 <@preed> KDEN 201441Z 35025G36KT 1/4SM R35L/1800V2000FT +SN BLSN FZFG VV002
07:07 <@preed> that's awesome.
07:07 <@preed> winds 25 gusting 36 knots, one quarter mile visibility, heavy snow, freezing fog, vertical visibility of 200 feet.
07:07 <robcee> I'll take your word for it. I haven't progressed to the ability to parse METAR data in my head
07:08 <robcee> ah, thank you for the translation
07:08 <robcee> man, that's nasty
07:08 <@preed> I'd have to look up what BLSN is
07:08 <robcee> blowing snow?
07:08 <@preed> probably "bitch, loser there's friggen SNOW OUTSIDE!"
07:08 <@preed> or that.

Anyway, the release is done, the holidays are near, and I'm totally excited about the prospects of a snowed-in white Christmas, which I haven't experienced in at least five years and certainly don't get to see in California.

So here's wishing you a happy holiday season—whatever that may mean to you—from OMGSOMUCHSNOW Colorado:

P.S. if you don't hear from me for more than 48 hours on IRC, I'm either on vacation, or dead. Either way, call the Colorado National Guard. Or rhelmer.

One of the two.

You want a release? Well here's your #@($! release!

I'd just like to state for the record that the 1.5.0.9/2.0.0.1 release was quite possibly the suckiest release I've ever worked on.

And if any of you remember the sleeping-in-the-office-for-a-week-1.5.0.2 release, that's saying a lot.

It's also saying a lot because the entire release went relatively smoothly1 until... well... today, when everything promptly started sucking. Hardcore.

Like most aviation accidents, it wasn't a single thing that caused so much doom and strife. It wasn't the air traffic controller who forgot to call out the traffic... or the maintenance guy who didn't tighten all the bolts... or the pilot who got distracted looking at a faulty warning light.

It was all of those things, working in concert with each other.

Similarly, the list of what-the-hell-were-we-thinkings2,3 includes things that, on their own, wouldn't have been a big deal at all. But compound them and you basically had fourteen hours straight of me banging my head against the wall and wanting to stab my face off.4

The sole thing that makes horrendous releases like today's at least bearable is the community's response and attitude when problems like this arise: the web team started working on the release in the morning, and remain responsive throughout the day; IT was around and helped debug deployment and database load and app cluster problems with us; the app team weren't bothered a bit by being called in the middle of the night to look at weirdness; my build colleagues pitched in to help run some extra verification while I worked on deploying the contingency plan; and QA stayed until the bitter end (past midnight!) to make sure it all worked and to sign off on it all.

I don't ever want to repeat a release day like this one.5

But—at the risk of sounding sappy—this pervasive and consistent attitude within the Mozilla project is what gives me the energy to face even the possibility of such beasts again.
_____________________________________
1 At least, no bumpier than any other Moz release I've ever worked.
2 Hindsight = 20/20, and all that...
3 My "WTF," while I think defensible, is particularly... WTFular.
4 I must admit, Winnebago Man did help at about hour ten or so.
5 And I hope we'll spend some time figuring looking at exactly what happened, so none of us have to.

December 11, 2006

AfterBurner

The New York Times Sunday Magazine had an interesting article last week on some new research and a book about burnout. (For those into the Podcast-o-sphere, NPR covered this as well.)

It mostly focuses on data from Wall Street brokers, a cutthroat existence to be sure, but it also has interesting perspectives from social workers and teachers.

A few particularly thought provoking passages stood out to me:

If one of the surest recipes for burnout, as Michael Leiter has said, is the sensation of inefficiency—particularly if we're still expending energy and seeing little in return—then there may be something about the modern office that conspires to burn us out. In 2005, a psychiatrist at King's College London did a study in which one group was asked to take an IQ test while doing nothing, and a second group to take an IQ test while distracted by e-mails and ringing telephones. The uninterrupted group did better by an average of ten points, which wasn't much of a surprise. What was a surprise is that the e-mailers also did worse, by an average of six points, than a group in a similar study that had been tested while stoned.

That's right. Stoned. Those people were literally burned out, and they did better.

I've been trying to pay attention to this more recently, and I've noticed that if my day starts with a number of fires, I can easily lose the rest of the day to "pacing on my desktop," that is switching between Thunderbird, IRC, Bugzilla, and AIM, all in a very tight poll()ing loop, just waiting for one of them to generate an interrupt.

If you get stuck into this mode, it's easy to waste twenty or thirty minutes in a tight loop, just switching between windows. One recent afternoon, I got into almost this trancelike state, where I was just switching between those applications, so quickly that I wouldn't have even picked up any state changes had they occured. Fortunately, I noticed it pretty quickly, but before I generally started paying attention to this tendency, it was easy to let this cycle pockmark my afternoon to the point where I didn't have any significantly contiguous blocks of time to use to accomplish any notably involved (and by implication, forward-thinking) tasks.

I've attempted recently to tackle this problem by closing Thunderbird entirely and, stealing a page from sspitzer, changing my nick on IRC to preedMsgMe (which seems to cause some confusion; but I interpret it as "I'm not going to pay attention to any channels, so if you want to talk to me, /msg me privately.)
***
"The worst case of Wall Street burnout I know is of this guy who wound up driving a cab," Cass tells me.

A cab?

He shrugs. "There's a control factor in driving a cab. You go from point A to point B. On Wall Street, you start your day with no idea how it's going to end."

My fantasized mode of transportation is different, but the sentiment is the same. Every time people have asked if I want to become a commercial pilot, my answer is the same: "No, that would just suck all the fun out of it."

But I've found myself thinking about it on occasion... like... today for instance:

17:06 <marcia> next week on "As the Update Churns"
17:07 <marcia> preed annouces he is leaving the build team to become a Fighter Pilot
...
17:09 <preed> marcia: I actually thought about that last week
17:10 <preed> maybe not a fighter pilot, though
...
17:17 <Pike> preed: now that you're panic-proof, you want to become a commercial pilot? bah, we shouldn't train you that well
17:17 <preed> I'm trying to figure out which is more stressful
17:18 <preed>this, or flying a hundreds of irritable people around in a tin can.
17:18 <Pike> 300 passengers vs 30 million users
17:18 <Pike> luckily, the 30 million don't die if you [fargle] up :-)
17:19 <preed> Pike: and to think I wanted to be an air traffic controller
***
One has to wonder whether the developments of a high-speed world haven't made burnout worse. First, the obvious: With the advent of e-mail, cell phones, laptops, BlackBerrys (or "CrackBerrys"—the argot here seems extremely apt), and other bits of high-speed doodadry, it has become virtually impossible, in senses both literal and metaphorical, to unplug from our jobs. As Schaufeli, the Dutch researcher, notes, one of the strongest predictors of burnout isn't just work overload but "work-home interference"—a sociologist's way of saying we're receiving phone calls from Tokyo during dinner and replying to clients on our BlackBerrys while making our children brush their teeth.

See also.

This type of "interference" seems to be one of the best kept secret neuroses of the project.

I'm as guilty of it as anyone: whenever on vacation, we're still "available via cell" and responding to email." Build requests are often made and responded to after 10 pm PST. We all talk about being walking zombies because we were up until 3 am last night (or "didn't bother sleeping" on that flight back from Europe).

I think part of my own tendency to do this stems from the dedication most of us feel toward The Project (tm) and The Community (tm) and The Cause (tm). That feeling is useful, because it creates an community full of engaged participants who care about their work, the sense of fraternity, and are here by genuine desire.

Never [forcing ourselves to] unplug, has made me wonder about the failure modes of our all-too-human bodies and psyches.

I'm betting they're not pretty.

December 8, 2006

Uno.

I noticed yesterday that it's been one year this week since I started working at MoCo.

For some reason, it feels like I've been here longer. Much longer.

Maybe it's partially due to the fact that joining MoCo was actually a return to the project for me. I interned on Netscape's Build Team a few months after the lizard was freed, and did some Bugzilla work throughout college. An old friend I hadn't talked to in awhile mused at the circular pattern of life: "when we were hanging out those first few years of college, you were keeping watchful eye over the Tinderboxen... and you now you are again."

So it goes.

I spent some time thinking about what it's been like to be here a year, and what I've learned. I came up with the following jumble...

  1. I've been to my fair share of "company parties," but nothing even comes close to touching a Mozilla-party.
  2. "Have you filed a bug on that yet?" and "Now accepting patches"1 are two of the most useful phrases. Evar.
  3. You can get yourself into trouble on IRC.
  4. It is often difficult to figure out who to pay attention to in the community, and the proper weight to attribute to their statements. One will make this mistake frequently and egregiously in the beginning, but receive plenty of feedback. It is a self-correcting problem if you pay attention when it happens, and you'll soon come up with a trusty metric.
  5. Alcohol2—in measured amounts—will help you solve just about any [build] problem.
  6. People are perfectly willing and happy to give you alcohol—in measured amounts—to get you to solve... just about any [build] problem.
  7. When in doubt, blame morgamic.
  8. After blaming morgamic, remember: Carebears hugs solve everything.
  9. If one of the Mikes suggest doing something, it's probably a good idea. If two of the Mikes suggest doing something, you should probably just do it. Probably.
  10. Tinderbox is a pit of black pit of despair that will suck your soul away. But it's our lil' black pit of despair that will suck your soul away.
  11. We have a lot of code like Tinderbox.
  12. A lot of that same code is some of the most robust code in the project. Maybe in the world.3
  13. Don't respond to IRC requests off-hours unless you're willing to respond to an IRC request every single day at that same time. While admittedly frustrating to others, it staves off burnout.
  14. Letting sspitzer talk you into eating anything is just a recipe for a bad scene.
  15. Respect WONTFIX.
  16. Never underestimate the ability of a single community member to cause a respin of an entire release4.
  17. Similarly, never underestimate the ability of a single community member to completely fix an entire module/subsystem.7
  18. Talk and design documents are cheap; checked-in code with bugs filed for improvements is the gold standard. This is not the case at all software companies.
  19. We don't always agree with each other about how to get where we're going... but we all seem to share a sense of generally what the destination is. That unity of vision is remarkable within a project and company. In that regard, the Mozilla project is much like a family.
  20. You never know when some random blog entry that was mostly a rant to begin with is going to get put on Slashdot's front page.
  21. To this day, I still get a kick out of the creator of JavaScript making me a martini... they're really good martinis, too!
  22. In such a consensus-driven organization, sometimes, you have to start a discussion with "Yes, it sucks, but we need to leave this meeting with a) a decision and as a result of a), b) some unhappy people." That's really hard to do. It's also sometimes really necessary.

There's probably some others I forgot.

Or maybe blocked out.

As I look back on the past year, it's been one of the most difficult of my career. I've had to do a few things that have grated on my professional aesthetics; I've spent more than a few early mornings fighting with builds and (in internet time) ancient infrastructure, and been stressed out on a host of occasions.10

But I've also gained new perspective on developing and releasing software on a scale not seen by many projects or developers, how to prioritize the myriad requests that get thrown at us every day, how to figure out what people are really asking for, when to do what people want, and when to push back.

I've got more to learn about all of those, but I wanted to thank you all for those experiences.

I'm a better software engineer... and a better person for it.

Now then... who needs a tinderbox kicked?

__________________________
1 Both courtesy of Shaver.
2 In my case, scotch.
3 But still... not Tinderbox.
4 Totally not referring to anyone5 or any particular release6
5 Jonas.
6 1.5.0.3.
7 Totally not referring to anyone8 or any particular module9
8 cf.
9 Nightly updates.
10 With all the inherent interpersonal pleasantness that goes along with that.

December 2, 2006

Tids and Bits

Finished up some fixes to my NOAA Metar Microsummary tonight, including:

  • Fixing the wind parsing code to handle AUTOmated and CORRected METAR reports
  • Munging the visibility parser to handle fractional visiblities (i.e. "1/2SM") correctly
  • Parsing (and removing) variable wind direction reports (i.e. "12315KT 300V340 4SM")
  • Reformatting the reporting of visibility to be "vN" over "NSM", saving valuable toolbar space

New version is available here.

Sadly, the ugly webpage remains.

Myk was kind enough to review my microsummary and noticed that the auto-installing code on the webpage had a bug in the Javascript, so if you couldn't get it to work before, click the link above and give it a try that way, if you're interested.

This new version is much better anyway.

***
Speaking of microsummaries, Shawn Wilsher commented
Here I was hoping that NOAA had microsummaries. Sadly, the METAR reports are a bit much for me.

I heard that sentiment a few times, actually.

(Possibly?) unsurprisingly, much of the National Weather Service's live ground weather data comes from these encoded METAR reports. Since airports have to do them, it's an easy way to get predictable weather data every hour from all over the country, observed by trained weather spotters (aka "air traffic controllers.")

Point is: we all use this data all the time; most of us are just unaccustomed to seeing it in this format.

The good news is the munging the microsummary does to save space on your toolbar actually makes it much easier to read; I replied to Shawn with a crash course on METAR reading.

If all you want to know is "How windy is it, how cloudy is it, and is the visibility obstructed by anything like haze or rain showers or ice pellets or volcanic ash," this gets you like... 100% of the way there.

And let's face it: who wouldn't want to know when their visibility is going to be restricted by volcanic ash.

***
I've had no less than two people comment on the fact that in my VCS Shootout post, the last picture looks like Mercurial is getting pummeled by ClearCase.

I don't know how people are thinking that, since there are clearly numerous little red spiky... thingies being shot at ClearCase... who is on the floor... getting hit by them.

Besides, as I mentioned, this was merely a screenshot from that round of fighting. We all know Mercurial could kick ClearCase's ass any day of the week. I honestly don't even know why we even bothered with that matchup.

(And of course you missed it Shaver; we all knew that round was such a joke, so you left to get us all beer. Remember?!)

December 1, 2006

"And that includes me..."

I've been suffering from a bunch of thinkos1 today, and I was trying to explain to rhelmer what that was all about.

I eventually just broke down and just showed him the "Winnebago Man" video2, which my friends and I have used for some time now to explain when we're having a... thinko-filled day.3

Watching it with Rob and Jay, I mused:"Is this the ray of sunshine build and QA can be like on release days?"

Either way, I certainly feel about bugs on code freeze day the way this guy feels about flies.

___________________
1 Kind of like typos, only with... thinking.
2 Caution: not work or little-ears safe.
3 "God, I can't f$#@!ng make my mind work..."