« March 2006 | Main | May 2006 »
I know this is a family blog and all... but after having just pushed the button on Thunderbird 1.5.0.2/1.0.8 and Suite 1.7.13 bits, please to be forgiving me for feeling the need to quote one of my favorite movies right about now:
Bon voyage, mother!#$%er.You. Were. Good.
I'm going to the hotel. I'm gonna take a shower. I'm gonna sleep.
For a month.
P.S. What Chris said...
One of the surprisingly common reports we get in #build are missing build reports: "The builds in latest-foobranch are five days old! Double-ewe-tee-eff!?"
Like a partner in a dysfunctional marriage, Tinderbox is an enabler of this bad behavior because after a certain amount of time, it just drops builds that haven't reported in without letting anyone1 know, so often times, by the time we receive a complaint about missing builds, they're not a day or two old, they're five or six days old.
This is unacceptable, and the Mozilla Community deserves better.
Having said that, with everything that the release team is typically doing during any given cycle, we don't have the bandwidth to sit there and monitor tinderboxen to make sure that every single one is building what it's supposed to. Often, this list changes so quickly that the documetnation about what each is supposed to be building isn't even correct.
Rob and Dave have been working on fixing this, though, with—drumroll, please—automation.
Now, Nagios monitors the contents of ftp.m.o, and we get an email whenever builds in the latest-* directories for relevant branches are more than a day old. And we continue to get this email every few hours until it gets fixed.
This should help to cut down on having to let us know that builds haven't shown up for five (or even more) days. It's always been a reporting problem, as we've been typically able to respond to Tinderbox machine issues within 24 hours.
The lesson here is twofold:
The next time you notice that the nightly build you were expecting to exist actually does... be sure to think of Rob and Dave.
___________________
1 Who can do anything about it... yes, these abondonings are announced in IRC, but in ways that seldom get noticed.
2 This, of course, isn't anything new... but it's nice to be in a place where we have the bandwidth to really start working on automation projects3
3 And we're working on even more projects I haven't had time to blah-g about...
1.5.0.2 (and 1.0.8, if you're nostalgic like that) released today!
One of the big announcements for 1.5.0.2 is the addition of Macintosh Universal Binary builds, including all the standard locales.
A lot of people (Mark and Josh especially) worked really hard to get universal builds into 1.5.0.2. I only did a minimal amount of work to support the effort, but I just want everyone to know: the only reason I personally worked on Universal Binaries at all was to make Justin, our IT director, happy.
So, even if you don't own a MacBook (or even a Mac!) be sure to take a moment and download the Universal Binary.
Do it for Justin.
14:55 <preed> I've had three friends tell me "Congrats on the release" already
14:55 <preed> I told them they have a problem with premature congratulation.
From the "Happy Easter"-department, LJ shmivejournal answers the age old question: what happens if I make a cake with... Cadburry Eggs?
Deliciousness!
I haven't had a chance to blog about some of the (interesting and helpful, I think) changes that have been going on in the Mozilla build/release world... been too busy pulling all-nighters trying to get the 1.5.0.x releases out the door.
One of the cooler (and almost-complete) changes going on with the Build Farm has been the result of a particular bug that started out small and simple and then kind of... became a catch all bug for awhile.
That main issue that bug tracks is fixing a tinderbox problem that had been so longstanding, that the wiki docs actually codified steps to work around it. The bug was...
...when you tried to stop tinderbox, it wouldn't actually kill any of the sub-processes under it. So the builds would continue unless you also issued a killall make gcc perl cvs right after stopping tinderbox.
After more or less solving that issue (CVS is still ill-behaved, and will ignore being told to shut down; Mark Mentovai, who helped with these patches has already filed a rainy day bug for that), I figured while I had my hand in that part of tinderbox's guts, why not work on a problem that I've been told we've been wanting to solve for a long time: versioned configurations.
There are two config files that Tinderbox really cares about; mozconfig and tinder-config.pl. These files can change for a variety of reasons, often at critical points during releases. Even in the short time I had been here, I had noticed myself logging into busted tinderboxen to investigate why a build had not produced the expected result, and then trying to remember if I had made that mozconfig change and forgot, or someone else had.
I added a --config-cvsup-dir option can be used for all sorts of things, but in our case, will be used to pull those two configuration files from CVS for each build.
Finally, the last longtime complaint, which I shared after trying to deploy this change on the relevant tinderboxen, was updating the tinderbox code itself. So there's now code to autocvs up the directories containing the tinderbox code after each set of builds runs, so each tinderbox always gets the latest changes. This solves the problem I found where a tinderbox would have components that were months or even years old (surprisingly, when updating these tinderboxen, we did so very carefully so we could revert them back, but all of them updated fine, and continued to produce builds).
All of this auto-updating and versioned config goodness is a blessing for those of us responsible for managing the 40+ machine build farm, but it's also a curse. A checkin that causes tinderbox to die for some reason will now mirror itself to all of the tinderboxen, which would all promptly die. (Rob and I have discussed making them update to a _STABLE tag, but we didn't implement it because no one but us really checks into tinderbox1 anymore, and it hasn't been a problem... yet).
Here's fair warning, though, that if you do work on the tinderbox code, your code will get mirrored to the build farm. So definitely perl -c it!
This initiative is almost complete. In fact, Rob is finishing up the machines that don't currently have it. Thus far, it's been a god send. We've been able to make mozconfig changes much easier (without even logging into build machines anymore!) and now all the machines get Tinderbox code updates on the fly, so we don't run into the problem of a machine with a months' old copy of Tinderbox.
As we start to get ahead of the constant releases curve, this is just the first of many automation changes we've talked about making to make all our lives easier, which translates to getting builds out the door faster, with higher confidence, and more consistency, which eventually translates to smoother releases.
Stay tuned!
I'm really excited to announce that we can now finally stop snickering at the term Build "Team."
Join me in welcoming the newest addition to the Build crew: Rob Helmer!
He's done a lot of open source work, including work on revision control system visualization tools and a Ruby-based pluggable webserver module.
We're really excited to have him on board, so if you see him on IRC—rhelmer—please say give him a big Mozilla Project welcome.
And be sure to stress how much fun he's going to have. ;-)
Welcome, Rob!