preed's blah-blah-blahg

Benjamin recently wrote about the current state of our effort to try to import our CVS repository to... something from this century.

His conclusion is spot on, although I think it... minimizes the head-banging he and I have been going through for a couple of weeks. My original characterization didn't turn out to be far from the truth, it seems... except, it's me and my quad-core-P4-with-4-gigs-of-RAM sitting there, bloodied and bruised on the floor, not ClearCase. ;-)

I was somewhat surprised by the number of responses to Benjamin's post that seemingly amounted to "Can't you just use Subversion? Subversion works. And if you want distributed, use SVK."

Well, the first issue with that is cvs2svn¹ doesn't seem to import the Mozilla CVS tree anymore: it's hitting the error that Hg tends to hit², and while completely dying is arguably more correct, bzr and cvs2svn 1.3.0's approach—annotating and ignoring the error, so the import can actually continue—is much more satisfying.

The second issue is that the march towards a distributed version control system really isn't about a distributed version control system; it's about using a tool that support merging algorithms that weren't invented in the 80s, back when you never did branches anyway, because it was annoyingly difficult with the tools of the time.

During the original discussion, the main issue that limited Subversion's advancement in the race was that it didn't support any better merging functionality or techniques than its predecessor. It requires external tools to record which merges had been performed and the actual algorithms used are the old ones we all love and/or hate.

Now don't get me wrong: I use Subversion for all my personal stuff and I like it. I think it's a great improvement over CVS (which I used for years and imported from) and in many (most?) cases, I would recommend it.

But when you're going to be doing the kind of "agile"³, disruptive, reconstructive work that Mozilla 2.0 requires, at a minimum, you need a tool that makes branching and merging easy. SVN does work for me (and lots of other people and projects) because I'm not faced with, for example, renaming nsIFrame::GetPresContext, a task where a branch makes a lot of sense, and I'm going to be doing hundreds of renames.

I contend that it's not so much that we require (or necessarily even want) a "distributed" version control system. In fact, as a counter example, Perforce is a [closed source] centralized VCS that has a lot of great features, including merging primitives that are awesome. Accurev is another (although, I've never personally used it.)

We just happen to be focused on "distributed" VCSs because those are the only open source offerings that have merging facilities that handle complicated situations and get the merging stuff right. This is likely because a distributed version control system isn't worth anything if you can't merge your work back in easily and [more importantly] reliably.

I'll concede, of course, that once you have things like offline diff/commit and easy patch sharing among peers, all built-in-and-tracked-by the VCS, that's (possibly addictive) icing on the cake.

But it's not about "distributed" part. It's about the capable-merging part.⁴

Breaking code apart is easy. Putting it back together is hard.

We want and need a tool that intrinsically expects, is designed to handle, and expertly supports the latter.

_____________________
¹ As of 1.5.0
² Which amounts to deleting files which don't exist on branches [possibly yet] that they're being deleted from.
³ I hate using that [buzz] word.
⁴ Coincidentally, Joel recently blogged about version control systems and large teams, and it seems the Windows team uses a model very similar to that of the 2.6 kernel developers, and possibly similar to what we'll end up using. It seems that easy branching (which is easy) and easy merging (which is hard) is the only real way to scale a development project into the thousands.

Posted by preed at 11:17 PM | Permalink | Comments (7)

Sometimes, you see a random photo whose composition and perspective all just click... and all you can think to yourself is "Yeah... that's my life right now... that's me."¹

It's one of life's better [surprise] pleasures.

_____________________
¹ Photo courtesy of my friend Mid,² who invited me to go plane watching today, but I declined. Maybe that was a bad call.
² The official photographer on all my flights...

Posted by preed at 1:29 AM | Permalink

Dear Hawthorne Automated Flight Service Station:

I'm sorry.

No, seriously, I really do apologize.

For whatever reason, it seems that when I land at a destination in your jurisdiction, I forget to close my flight plan.

I don't know why that is, exactly... it's on my checklist.

And the good folks at Oakland Flight Service always remind me to do so when I open with them.

And I never seem to forget to do it when I'm in Oakland or Rancho Murrieta Flight Services' jurisdiction.

See? I even double-underlined it after I forgot the first time...

Maybe it's just because I'm so squee-u-lar whenever I land so far away from home that I forget to give you guys a call until it's too late.

You always give the lecture about having started the search and rescue process. The one saving grace is I tend to land at towered airports, so this only involves a call to the control tower, who (I can only presume) ruffle through their stacks of paper, find that I did indeed land, and then tell you to call the search off for the stupid-pilot-who-forgot-to-close-his-flightplan.

"Only."

You also always warn me that if it had been an non-towered airport, the police would have been called. That's a scary prospect. I always mumble some useless excuse and thank you for your services.

Then—and I'm not kidding; you can ask my passengers—I tend to feel bad about it for the next couple of hours.

You and the control tower have better things to do than "searching" for me. This is probably the single best reason I can think of to get my IFR rating: so I can fly places IFR; then not closing my flight plan will be someone else's problem!

Anyway, I wanted to let you know that I really do feel bad about it, and I'm really trying to find ways to make sure in the future to remember to give you guys a call right when I touch down.

Me, filing my flight plan with one of your friendly (and patient!) briefers...

Keep up the good work. You really do provide top notch service, even for forgetful pilots like me.

Sincerely,
preed
[Pilot certificate number redacted]

Posted by preed at 10:16 PM | Permalink | Comments (2)

Last week, the Firefox trunk build needed to be clobbered due to a checkin that broke depend builds.

dbaron emailed us, requesting the clobber, but then he must've remembered me mentioning adding CLOBBER file support to the Tinderbox client.

The idea was that developers shouldn't have to find someone in #build to clobber builds for them; as long as the person requesting the clobber and reason can be tracked, developers should be able to clobber their own builds.

Implementation of the idea original stalled, though, because CLOBBER file support relies on a) all the configs being public and 2) all the Tinderboxen paying attention to those public configs. That's been done for awhile now (thanks again Coop), but we never had a chance to finish testing the clobber file support, so I had never announced it.

Well, dbaron tried it out and—much to my surprise—it actually worked!

I also tested on the 1.8 branch later that day and it worked there, too.

So, if you need to clobber a build for a checkin, here's how you do it:

1. After your code goes in...
2. Checkout mozilla/tools/tinderbox-configs; you probably want mozilla/tools/tinderbox-configs/[product]/[platform], i.e. mozilla/tools/tinderbox-configs/firefox/win32. If you want to clobber the 1.8 or 1.8.0 branch(es), check out those branches (-rMOZILLA_1_8_BRANCH).
3. If there's not a CLOBBER file in that directory (case matters), cvs add one. Edit the file with the reason (and hopefully a bug number?) for the clobber request.
4. Check it back in; include the reason in the commit message (yes, this is duplicated from the step above...)
5. Annotate the build on Tinderbox to let others know it will (should be ;-) a clobber build, and thus will take longer to complete.
6. If you're feeling benevolent, please stop by #build to let us know you requested a clobber, so we know to expect it
7. Next build should be a clobber build; if it's not, please stop by #build to let us know, so we can track down why it didn't work

Obviously, clobber builds take longer than depend builds, so please only use this functionality only when necessary (there's a code change that requires it; nightlies are broken for some obvious reason). I'm pretty confident that we all know valid reasons for requesting clobbers. And please coordinate clobbering on IRC, if you can.

While it took us awhile to get here, this is the type of functionality I was talking about working more on this year to give developers the tools they need to be more productive/"agile," but in a framework that tracks the information we need to track.

I'm hoping this is just the beginning.

Posted by preed at 10:08 AM | Permalink | Comments (3)

Yesterday, I talked a bit about my 2007 thus far.

Schrep and I were discussing New Years Resolutions yesterday, and he said he didn't believe in New Years resolutions: "It's never too early or too late to make a change in your life."

I'd never thought about resolutions that way, but that's a really good point.

As I mentioned, I'm more a fan of birthday resolutions; birthdays are like our own personal New Years and I tend to be more introspective about my life around my birthday anyway. New Years is mostly arbitrary timing in relation to our individual lives, so it's never really done it for me.

I guess the New Years resolution thing is helpful because it throws the possibilities of change to make our lives better right in our faces. New Year does provide a bit of synchronicity since many are simultaneously thinking about changes in their past tense and what the future tense could, and should be.

Whatever you think of resolutions, I spent some time thinking about what mine might be:

Mozilla-related goals

1. By year's end, the automated release harness will be able to basically perform an entire release, in one¹ step. Extra points if signing is automated as well.
2. Create a clear separation between the gloop that we call "Tinderbox", which actually is a mess of Tinderbox interface code (which we may not care about in the long term), our build harness (which we probably care a lot about, but needs some serious love), and the build system, which I hear is in line for some serious love.)
3. Work on improving the way I personally communicate regarding build projects we're working on that could make everyone's lives better. This is something that's only recently come up because we've been so bogged down, we haven't been able to focus on changes that are sweeping enough to make a difference; that's starting to change, and so changing the way we communicate these developments is important (more on that later this weekend...)
4. Work with an eye and a nod towards more self-service services for developers; there are lots of things that we, as build engineers, need to track to ensure a successful release. That doesn't mean we have to perform or even be involved in everything to track it. It does mean we need to create systems were people can work within a framework that works for them, but captures the information we need.
5. Participate in, and hold, more postmortems, where it makes sense to have them.
6. In numerous forums, I've heard the sentiment that more could be happening in certain contexts if we had more build/release resources (not to be confused with build/release people); I'd like to work with an eye towards making it so I don't hear that as often. ²
7. Become larger than #bs and crazycapsguy.³

Personal goals

1. Stay off thedailywtf.com.⁴
2. Get my instrument rating this year.⁵
3. Move. I'm still in my just-out-of-college apartment, and... it feels like a just-out-of-college apartment.⁶
4. Average one "notable" (defined as requiring-at-least-two-weekdays-off) vacation per quarter.
5. Take more pictures of myself. Or have more taken of me. Either way.⁷
6. Execute—as pilot-in-command—a cross-country flight that's actually across the country. I'll loosely define this as across ARTCC boundaries (but Los Angeles Center doesn't count), and likely involving another state; I'm thinking PDX or PHX.

In general, I think my resolutions boil down to having—not trying, but having—a better year than the last three⁸; I'm going to be my-favorite-number years old this year, so I think that means this year is basically required to not-suck.

Despite the first few days thus far of 2007, I have this weird feeling that it's looking quite possible.

________________________
¹ At least, from a build engineers' perspective
² This is, admittedly, a pretty fluffy goal; I'm not quite sure how this one will turn out.
³ Would this require SPEWING BS IN CAPS?
⁴ Which at first glance, might seem easy, but if we're refactoring Tinderbox client code...
⁵ And I mean this year, not mumble-mumble years, like my private took.
⁶ Imagine that!
⁷ P.S. Flickrsphere!!!
⁸ Combined, if possible!

Posted by preed at 5:58 PM | Permalink | Comments (4)

mrz caught me in the hallway today to let me know about a late Christmas present for those doing checkouts on the other side of the globe: cvs-mirror.nl.mozilla.com!

This CVS mirror is not in production yet, but he told me about it so that he could get some help testing the mirror integrity/process.

So, if you've got a few extra seconds, you can help by doing a checkout of either a) your favorite tag or b) a specific date spec (-D) from cvs.m.o (or cvs-mirror.m.o) in one directory, and then pointing your CVSROOT at cvs-mirror.nl.mozilla.com, and performing the same checkout into a different directory, and then diff -ring the contents. The process for logging into the mirror should be identical to the current process. (If it is, that's a bug).

You should only see differences in the CVS/Root files.

You can even go hogwild and check out different branches and make sure they're correct!

If you're on this side of the planet, and feel like helping out, you can perform the same test using cvs-mirror.sj.mozilla.com, which will someday grow up to become cvs-mirror.mozilla.org.

I want to stress that neither of these mirrors are in production yet, but you can help make sure they're production ready by helping mrz and I out.

Posted by preed at 2:49 PM | Permalink | Comments (4)

It's not even the first week of 2007, and it's already been a harrowing... journey:

• Due to The Blizzard^TM, Service Pack 2, I moved my flight from Friday the 29th to New Years Eve, arriving into SFO at 8 pm PST; this would've been fine, except...
• The plane, out of LAX, was three hours delayed, due to a passenger freaking out while the plane was in line to take off, necessitating their deplaning.¹ Don't even get me started on that whole thing...²
• Upon arriving in SFO and waiting for my luggage-which-never-came, someone saunters by and says "Oh yeah, check in that pile of luggage over there," as I was filling out a lost baggage claim. Sure enough, in a stunning display over overcompetence, United had put my luggage on the earlier (on time) flight to SFO... the one the agent said I couldn't get on because my luggage would never get transferred from the late flight to that one in time.
• After the lateness and the luggage-waiting, I rang in the New Year while traveling down highway 101 at 80 mph. Welcome, 2007.
• Arrive home. "Screw it, I'm going to bed."
• Next morning, I found that whenever I tried to do anything I/O intensive on my workstation at home, it either froze up or immediately rebooted. It was like a fun little "Let's make a Deal!"-game... but that's not all!
• Upon rebooting, the ethernet card only sometimes honored me with its presence on the PCI bus. Replacing the card with a spare (of a different brand) I had lying around did not help.
• Since my workstation had crashed once while I was away³ and I kinda care about the data on that machine, I decided to not screw around and replace the whole damn thing.
• Got a barebones box from Fry's; wasted a day finding out that its power supply and/or mobo was dead. "Your best buys... are always at..."
• Finally got a machine up and running last night, and starting transferring everything over; didn't notice that I hadn't compiled the specific chipset support into the Linux kernel to handle IDE DMA... so my 140 gigabyte music collection was puttering along at a stab-me-in-the-face 3 megabyes per second, as opposed to a I-won't-be-dead-before-it-finishes 170 megabyes per second.

On the upside, I took this opportunity to make my machine complete RAIDified, so crappy hard drives should no longer make me stabby-stabby, and I also used the opportunity to get all paranoid⁴ and encrypt my /home partition. I must admit, with the 2.6 kernel this was intriguingly easy to do on top of RAID. (Some people actually run encryption on top of LVM on top of RAID, but those layers are all in software, and... I just don't know if I trust my read()s and write()s going through oh-so-many layers before hitting hard magnetic filings.)

I don't really do New Years Resolutions⁵, but I'm throwing down the gauntlet now: 2007: Stop. Sucking. Now.⁶

I want this year to rock... and I actually think this one has quite the potential to do so.

More on that tomorrow, though.

_________________________
¹ And, because "That's how the terrorists would do it," their luggage, too... which, since it's not, ya know, organized in any particular way, means tearing the baggage compartment apart to find it.
² This passenger freaking out caused about 20 people to miss their connection in SFO, including (I overhead) people going to Seattle, Honolulu, and Sydney. Man, I was pissed, but if I had been going any of those places, I would've lost it right there on the concourse in Denver.
³ And I had to have a non-Linux using friend reboot it; "Type Slash-eee-tee-see-slash-init... no, INIT... eye-enn-eye-tee... INIT... no, not slash, then a dot... dot-dee... slash..." But he was a good sport, and we got it rebooted.
⁴ Maybe I've just been watching a little too much Law and Order lately...
⁵ I much prefer Personal New Years, aka birthday resolutions...
⁶ "I don't want any more BS during the year... from anyone... that includes me!

Posted by preed at 6:56 PM | Permalink