preed's blah-blah-blahg

pacifica-vm, which most of you probably know as the Firefox 2 Windows nightly build machine, has had an interesting week.

Last week, I took the machine down to back up its virtual disk image, increase the amount of RAM available to it, and, in an attempt to decrease the cycle-time, added a VMware virtual-CPU to the VM, increasing it to two.

This didn't really have the intended effect. Cycle times for both nightly and depend builds went up by about 15%.

Thinking that maybe the build system wasn't making as efficient usage of its shiny new virtual CPU as it could, I upped make's -j value from 3 to 4. This reduced cycle times... to what they were before the memory/CPU "upgrade," but also had the useful side-effect that make would hang every few builds, including most notably on nightly builds. (That is, incidentally, why nightly builds on Tuesday, Wednesday, and Thursday were all late; make kept hanging overnight with -j4.)

Finally, last night, I removed the second VCPU, but kept the extra memory and higher -j values.

That change not only made the machine start reliably producing nightlies again (or, at least, make stopped hanging), but it took the cycle time down to 40 minutes for a depend cycle, and 2ish hours for a nightly build. (Interestingly enough, that full build value seems to fluctuate anywhere from about 90 minutes to just over a couple of hours; I think this is because the trunk build machine and the 1.8 build machine are on the same VM box, and they're both starting their nightlies at the same time, which slows both of them down a bit.)

So, to recap here:

• Linking Firefox, especially on Win32, takes memory. A lot of memory. In the couple of trials I paid attention to, it took around 700 megs. Seeing as the VM had 700 megs, a large part of the problem seemed to be the machine descending into swapping thrash-hell when trying to do the final link.
• Win32 SMP = Teh suck. I had actually learned this from previous experiences in previous lives, but... a reminder is always good.
• When you actually pay attention to VMs, and spend some time "tuning" them—which in this case, amounted to creating a better match between the memory profile for the machine's task and the virtual hardware—VMs don't perform all that badly, relatively to physical hardware. gaius-vm, for instance, has horrible cycle times compared to gaius, but it's not "just because it's a -vm." It's because no one's paid enough attention to it after migrating it to tune it. (No, I haven't taken what I've learned here and applied to gaius.)
• Once again, sometimes... VMware['s performance] continues to confuse the hell out of me.

Posted by preed at 8:13 PM | Permalink | Comments (3)

Mid said: "You should post the translated version of that Quote of the Day; it's much funnier."

So, I present... the TQOTD:

More seriously, if the spirit of Free, it is râler by saying that Firefox is not Libre because of the trade mark, I am not sure that that is a spirit which attracts me.

Posted by preed at 2:34 AM | Permalink

If you ever hear me complaining about crosswind landings, just remind me that I really don't have all that bad...

On a return from KSBP on a recent flight, my pax got a great picture of what it's like to operate in a clear night sky in the Bay.

I have one other seat available. Anyone else game?*

* Extra headset procurement, weather, and RC2 release schedules permitting.

Posted by preed at 8:08 AM | Permalink | Comments (1)

"I'm fluent in JavaScript as well as Klingon!"

Posted by preed at 1:33 AM | Permalink | Comments (1)

I presented my overview of Mozilla's Build System to Seneca's Topics in Open Source course this morning:

13:33 <vlad> how'd preed's talk go?
13:34 <Moe> vlad: he had an awesome sport jersey
13:34 <Moe> with a lizard
13:34 <Moe> thats all i took from the lecture
13:34 <Moe> oh and of course there was that whole build thing

Glad to know I made a positive pedagogical impression.

I ended up mentioning this during the presentation because I think it's important to point out to newcomers who may be working hard to integrate themselves into the community that even the more... seasoned among us still can have huge grey-to-black areas in our knowledge about parts of the project. That's part of the project's charm—its deep breadth of things to work on—and its curse.

Seneca, through this course and others they're working on developing, is trying to bring more open source material into the classroom. This has a number of benefits for both students in the course and the open source projects they choose to work with.

The benefit to the body of open source code is obvious. The benefit for students is more subtle, a point I had spent some time trying to convince professors at my alma mater of: quite simply, open source provides real world conditions that the classroom cannot.

When students go out into the workplace, the chances that their first job will entail designing and implementing software systems from scratch is very small. Rather, the "fresh meat" typically get all the bugs that no one else had time (or wanted to) fix. Most software jobs out of the gate are maintenance jobs.

Contributing, as part of a course, to an open source projects mimics this real-world condition in a way that is difficult, if possible at all, to re-create in a classroom setting. The one idea I had advocated was a sufficiently complex software system that was passed off from software development class to software development class, where each quarter, features were added and bugs were fixed, but... this implementation of that idea is better. Much better.

It's heartening to see a college program that really understands this and is trying to directly address it.

I told Dave he has a good paper covering the academic aspects of using open source as a model for teaching waiting for the Journal of Higher Education.

I asked in IRC how they were being graded on the assignment. The answer is a writeup and screenshot of their locally-built browser.

I started going through a few of the writeups, and found it very interesting. The lacking specificity of the assignment allowed prompted each student to think about it differently.

Some students grabbed the trunk from CVS and built that. Others used the source tarballs from a 1.5.0.7 release. All three major platforms were represented.

I think there's a lot of feedback in those write-ups, detailing all the ways that we could make life easier for new contributors. And what great design for an assignment.

I had a lot of fun today.

Seneca students: I hope you do continue to wander into #build throughout and after the course! Thanks for having me today.

Posted by preed at 11:09 PM | Permalink

Tonight, with help from morgamic and sspitzer, we've published Firefox's first ever "major update."

This type of update is intended to pull people from (for instance) 1.5.0.7 to 2.0. We won't be immediately publishing any major updates (including that particular update path). It includes the ability to display EULAs and allow users to ignore the major update, including forever (so they can stay on 1.5.0.x, if they wanted).

We'll be doing a couple of tests:

• We've published an update for build 2006091813, win32/en-US only; most people have already updated to today's nightly, but if you'd like to try it out, you can download that build, run it, and check for updates. You should get offered a major update to "2.0mt1" ("mt" stands for "major [update] testing").


• Within the next couple of days (hopefully tomorrow afternoon), we'll run a clobber build on the 1.8 branch, for all platforms/en-US, and publish an update to those builds that is a major update (will likely offer itself as "2.0mt2").

If you'd like to help test out the major update functionality, Seth, morgamic and I, along with millions of users, and at least a couple of pirates, would much appreciate it!

Posted by preed at 11:33 PM | Permalink | Comments (8)

I'm about to board the Air Canada red-eye¹ to present to Seneca's Open Source course on the wonderful and intriguing world of Mozilla builds!

The TSA was intent on taking my deodorant, so if something smells amiss... remember, it was all to save freedom.

_____________
¹ Yes, yes, perfect timing... I know...

Posted by preed at 4:32 AM | Permalink

In case you've only ever read this "blahg" via its RSS feed, the tagline reads "What does a Build Engineer do all day, anyway?"

I've been thinking a lot lately about the answer to that question.

When I think about what my answer actually is, day-in-day-out—doing builds, triaging broken builds, restarting automatic builds, respinning builds, making sure there's enough space for builds, signing builds, staging builds, getting new places for people to get builds, fighting fires on the machines the builds are created on, making sure the builds... well... you get the idea—it all seems somewhat... repetitious.

(And the more I look back at that, actually somewhat depressing.)

But even though we, Rob, Coop, TR, and I, do work on those things, I'm pretty sure we all think about other things, especially the fourth or fifth time in a day you've logged into the same Tinderbox to kick it.

Part of this recent introspection about what we work on, what we want to work on, and what we should work on was prompted by a question Schrep posed to me a week or so ago: we had just finished a company-wide discussion of goals and missions and he asked off the cuff what the Build Team's manifesto would be, if we had one.

Something along the lines of "What should we be thinking about in the day-to-day that prompts us to work on things that have perceptible, positive impact, not only for ourselves, but for everyone else?"

There are the obvious ones: doing builds, triaging broken builds, restarting... well... you know... but isn't there more than that?

Should there be more than that?

If you've ever talked to Rhelmer or I right after a release, our reaction to the mere mention of the word "build" implies that there should be.

In trying to tackle the answer, the only thing I could think of was make a list. As I sat there, trying to brainstorm what the list should include, I found myself jotting down the things I think about all day (and sometimes worry about at night). Some of them had that dirty, five-letter b-word... but a surprising (to me) number had nothing to do with that.

As I sit here, counting the thirty-plus items on my laundry list, I wonder if anyone would be surprised that someone thinks about these aspects of a software project that stare back at me. Does anyone really care what the number of old builds we store says about us?

Or why bumping versions is a big deal?

Or what the deal with centralized build data stores is?

Ok, so maybe people wouldn't be surprised.

But, still I wonder. I've been in situations where I've been asked "Hey, can you guys do X?" When my answer isn't what they expect, it sometimes prompts what can be an involved conversation about why the answer wasn't a simple and immediate "Yes."

I think such conversations are useful in the long run.

But I can't help think if there were a good answer to the question my "blahg's" tagline poses, then maybe some of my weird answers to questions would make more sense to others sooner.

As a bonus, if I could distill the common themes behind those day-to-day answers, I might just have the makings of a manifesto that would at least frame our discussion for what unique qualities we bring to the "MozillaVerse" (or are the cool kids calling it "Mozillasphere" these days?)

Think of it as an attempt at a JoelOnSoftware knockoff, except... without all that annoying credibility and pestering about having created-Visual-Basic-for-Applications-but- before-it-was-called-that-because-I-only-worked-on-Excel-at-the-time getting in the way.

On the downside, I will admit: it is highly likely there will be fewer treatises on the speed of duck-typing.

Posted by preed at 8:08 AM | Permalink

You'd think 1.9 terrabytes of disk storage would be enough for Mozilla's builds, but... it's not.

In fact, with all the simultaneous releases going on lately, we've had to spend a lot of time babysitting the chronically under-spaced stage.m.o.

Fortunately, we now have a replacement and a plan to get us using it.

In addition to giving us another terrabyte of storage, we'll be able to reclaim some diskspace on the current arrays, and provide better verification of builds going out to the mirror farm.

The plan is to migrate stage.m.o and ftp.m.o to differet machines this Thursday. The downtime will start at 6 pm PDT and end at midnight.

Details of the plans can be found on the Wiki, including changes for contributors posting builds (they should be minimal).

If there are any questions or concerns, please feel free to email build@mozilla.org with them.

Posted by preed at 8:32 PM | Permalink | Comments (2)

Last Friday, after [*cough* too many *cough*] hours of flight time[due entirely to me dragging my own feet], I drove out to the Palo Alto Airport and took my private pilot checkride with FAA-designated pilot examiner Mike Shiflett.

The first part—the oral test—went smoothly... and quickly!

I then gathered the weather for the second part, the practical flight test. Everything looked great on paper, but after pre-flighting the plane and getting out to the runway, an almost-direct crosswind of 12-gusting-15 knots, made this part of the test more... interesting. Much more interesting.

But after ninety minutes of doing various types of takeoffs, landings, stalls, and flight maneuvers, the examiner flew us back to the airport.

I guess I managed to trick him into thinking I knew what I was doing.¹

To celebrate, I flew a couple of close friends to O69 today for my first $100 hamburger.

Both being photo nerds, they took some pretty incredible pictures.²

Posted by preed at 6:17 AM | Permalink | Comments (1)

Is there a way for me to filter a cookie based upon a regex matched against the cookie name, for each webpage, without the "Do you want to allow this site to set this cookie?"-window popping up? For instance, say I never wanted cookies with /^__utm\w+$/ to be set, even if I wanted other cookies from a site set?

A bit of Googling netted me this Netlib documentation, but seeing as it's dated 1998, I have little faith that it was even ever implemented.* (If I'm remembering the last time I looked at this correctly, I'm pretty sure I even did a quick LXR search on that pref name, too...)

An extension would be fine too... and I think I even searched for one too, but... alas, I came up with nothing.

Any pointers?

*No, I didn't bother trying this yet, since the effort to ask the LazyWeb is much lower than the effort that will be required to bang my head against the wall when it doesn't work. I figure that's precious time I could be spending fixing a Tinderbox... or something. ;-)

Posted by preed at 4:26 PM | Permalink | Comments (10)