lilmatt's blog

A number of people have asked me why I work for on Mozilla. They want to know what about Mozilla makes me want to be a part of it. Up until now I hadn't been able to give a cogent answer that wasn't simply a stream of consciousness exercise, full of insight but impossible to follow. However, after some lengthy discussions with folks, I may be able to explain this in thirty minutes or less.

My brother, Ph.D. does crazy-go-nuts immunology research. I have no doubt that if he hasn't already, he will eventually save someone's life through his work. It is clear to me that he makes people's lives better through his research. His ex-wife, Ph.D. is the head of the mental health and counseling department for a large university. She without question has saved lives, and directly helps people with her work on a daily basis. My wife is a financial aid counselor for an ivy-league university. She helps potential and current students get the funding they need to they can learn and become the next generation of crazy-go-nuts immunology researchers, head of mental health and counseling departments, and financial aid counselors, among other things.

Like my brother, his ex-wife, my wife, and all the rest of us, I have a set of skills and those skills are for the most part finite. Sure, I could attempt to learn absolutely anything, but for whatever reason, be it physical, genetic, or level of interest or enthusiasm, I just won't be as successful at some things as I am at others. With good fortune, many of my skills and interests relate to computers, networks, software, problem solving, and the Internet.

Mozilla, through the software it creates, the community it has developed and maintains, and the technical influence it wields on the Internet, makes people's lives better. Much like how an incandescent lamp converts electricity into light, helping Mozilla is how I feel I most efficiently, directly, and significantly can convert my skills and interests into real positive change for people, all over the world.

I hope this gives a folks a little more clarity into why I work for on Mozilla, and why I feel working for MoCo has been such a privilege.

For the last eight months I have been fortunate enough to have my work on Calendar funded by the Mozilla Corporation. During this time we've had two releases of both Sunbird and Lightning (0.3 and 0.3.1), and are on the cusp of a third (0.5). We have added tinderbox builds for our localizers, delivery of nightly builds to our testers (via AUS), crash reporting support to Sunbird, iTIP/iMIP invitation support to Lightning, read/write support for Google Calendar (via an extension), and improved CalDAV support, and have developed a community of regular contributors and testers. I am proud to have helped steward the Calendar project these past eight months and to have directly contributed to many of the above accomplishments.

Unfortunately, all good things must come to an end, and so my MoCo contract ends Saturday, March 31. On Monday, April 2, I begin working for Flock, the "social web browser" based on Firefox. I will continue to contribute to the Calendar project, and will continue driving the Sunbird/Lightning 0.5 release as much as I can, however I'll no longer be all-Calendar, all-the-time. At Flock, I want to push some of their work on the core of Mozilla back into Mozilla-proper, so that we all can benefit. I feel so lucky to have found work where I can continue to hack on Mozilla stuff full-time.

I'll be in Mountain View all of next week for indoctrination, and I hope to see folks for lunch, dinner, hockey or what have you. Working with everyone at MoCo has been a wonderful experience, and I deeply appreciate the opportunities I've had these past few months.

I'm already beginning to fulfill one of the goals I set out for myself when I started with Flock.  Yesterday, I opened my first bug to push some Flock code back upstream to Mozilla. I had queried Flock developers during my interview process on how much code they had already pushed back upstream.  The response was almost as if on cue.  Groaning (in unison) was followed by everyone's favourite (and unfortunately valid) complaint of patches sitting in review queues for months without any action.  That wasn't the only issue brought up, and many of the issues can and will be mitigated.  However, the point to take away was that while these developers were "bought in" to the whole Mozilla community/ecosystem and "giving back" thing, the issues they encountered with the review process basically skewed the diminishing returns ratio to the point that people stopped bothering. In reviewing this oft-repeated complaint, I came up with this short list of issues...

1. Outside contributors often don't know the drill.  There is a real art to getting code into the tree.  While the process is documented, it's the subtle details that determine whether this process involves much pain and suffering or instead is efficient and streamlined.  As an admittedly lengthy example: if the bug is hastily written, or doesn't describe a compelling enough problem to the reviewers and drivers that read it, there's a good chance it'll be WONTFIXed.  You have to context-switch out of "internal dev" mode and into "talking to folks that don't live in my codebase every day" mode.  drivers@m.o probably won't know all the issues you're facing because something's an AUTF8String instead of an AString.  Explain yourself well.  Write your bug summary, and then have a co-worker read it for sanity.  Link to the bug in your defect tracking system if it's available to the public, but remember that linking to your tracker doesn't excuse you from explaining yourself well in the bug on Mozilla's.
2. Patches sitting eternally in review queues demotivate contributors.  This one is obviously not outside contributor-specific, but is often experienced by them in an uneven ratio.  Outside folks often don't know who to request r? and a18xx? from, they pick someone whose queue is insanely long, or they use the despot-driven module owners page and don't know that it's out of date and so-and-so is no longer working on foo, etc.  Recently, pavlov's made serious progress in cleaning it up, but it's almost always going to be at least a little out of date.  Knowing who to request review from is as important as writing the bug.  Use bonsai to check the CVS check-in history for the component, directory or file you're touching.  Read the most recent check-in comments to get a sense of who the active reviewers are.  If you're still not sure, try pinging one of them in IRC.
3. Whomever checks in is responsible for bustage.  If a driver checks in your seemingly innocuous patch, they're on the hook to watch the tinderboxes for the next couple hours in case your patch breaks the build.  This is a non-trivial commitment of time on their part, and means it's more work than simply reviewing the patch and ticking the box.  This is where being a jerk in steps 1 and 2 comes back to bite you.  So as Chris Rock says, "Be polite."  If at all possible, be around on IRC in case you're needed to debug the bustage.  This is also where hiring existing Mozilla developers to work on your project can pay off.  If the reviewer personally knows the coder, and the coder has check-in rights, the burden on the reviewer is much less.  The reviewer knows that the coder will be the one watching for tinderbox flames for the next two hours, and so they're more likely to give that patch r+.

Let's return to the patch I submitted.  While it is both small and almost Flock-specific, I thought it was simple enough to gauge how much pain we as outside contributors truly faced when trying to "give back."  I must say that the process didn't feel much worse than it did being a calendar developer.  Like I was when on Sunbird, I am still one-step removed from Firefox, but I'm not Mr. Unknown.  In this case, I've met the reviewer, Gavin Sharp, in person.  I was able to discuss the patch with him a bit on IRC, and in doing so he gained some understanding of why we made this fix.  I was also able to address some of his concerns, thus the r+. So the moral of this story for me is that if you're working on a Mozilla-based product, hire an existing Mozilla contributor when possible, and engage the community in dialogue often.  Become known for good work, and even better behaviour.  Opening a bug, attaching a patch, and expecting it to stand on its own seems foolish given the scarcity of resources and a reverse scarcity of code.  Open lines of communication help avert your patches being from summarily rejected, and may make you think through why you needed to fork in the first place.  

Tags: mozilla, flock

Ok, so maybe 100% is a little overblown, but not by much.

The reality is that up until the 0.9.1/1.0 development cycle, Flock had little in the way of what Mozilla folks would think of as code review. There was a "buddy system" in place where your code buddy would look over your code post-checkin. However when deadlines approached and bug counts rose you can guess how religious folks were at checking someone else's code.

Around the time that we were finishing up Flock 0.9.0, we took a look at the state of our trunk versus the 0.9.0 release branch. It wasn't a "clown wreck", but it was too scary to consider releasing from. At this time I was asked to take on the role of configuration manager. This boiled down to "you don't check in without a=lilmatt since it's his butt on the line if the tree is borked". Since having me in the critical path puts us at risk of a bus error, we added some trusted friends to the approval pool, and thus drivers@flock was born.

Since that time, we instituted some coding style guidelines, as trying to read and review code from a bunch of developers, each with their own slightly different coding style, was rather... challenging. We also bit the bullet and added an explicit review step. So the process is 1) make patch, 2) get review, 3) land and bake on trunk, 4) get approval for release branch, 5) land on release branch. Sound familiar? My thought was to go with what you know works.

The end result is that the code in 0.9.1 is markedly better in quality than our earlier releases, and 1.0 is shaping up to be better still. Our code is also significantly more readable, which is good news for developers looking to add a provider for a particular web service we currently don't support.

I'm a pessimist and pragmatist by nature, so I'm naturally not satisfied. There's always more cruft to excise or cleaner interfaces to create. However, we've come an enormous way in a tiny amount of time, and that isn't something I can take credit for. My hat is off to my colleagues, who accepted and tried this review thing, and who actually made it work. Having your work critiqued is never an easy thing, nor is it often pleasant. However folks took it on faith that this would help us. Thanks folks.

Thanks also to Mozilla for refining a review process I could more or less copy verbatim. It works.

...and since I mentioned it, you can sign up for the 1.0 beta here.

Back in October 2005, Bart Decrem, then Flock's CEO, wrote the following regarding Flock and licensing its code:

"We currently plan to license open source code that’s created by Flock under the GPL license.  Modifications to Mozilla files will of course be made available under the MPL license.  We plan to ask that community contributors to our code assign copyrights to us, so that we will be able to license code under the MPL/GPL/LGPL triple licensing scheme as appropriate."

The reality is that very little (if any) Flock-original code has been relicensed and pushed back up to Mozilla, almost all Flock-original code is licensed solely under the GPL, and much of the code that is tri-licensed is stuff of little value such as Makefiles.

In my time at Flock, I've heard many theories from different people on why this is.  Some thought it was a conspiracy, meant to prevent the "evil MoCo" from taking the hard work of the nascent Flockers and putting them out of business.  Others said it was simply a philosophical decision on which license provides the most free-as-in-speech freedom.  I personally suspect somewhere in the middle lies reality.

Regardless of why Flock-original code is GPL, the fact remains that it is.  Another fact is that Mozilla will, with rare exception, not include code in Firefox that has not been tri-licensed.  Therefore, Mozilla can not randomly take Flock-original code and include it in Firefox.  When I asked internally about this, I was told something like "if there's some code Mozilla wants, they can ask and we'll consider it."  Great theory, but it's not as if someone from Flock is hanging out in front of Building K (MoCo HQ) actively offering bits of code.  In addition, the idea that Mozilla would be perusing our source repository in search of "the good bits" (pun intended) was laughable.  To my pragmatic mind it sounded as if we were saying "we'll help if asked", but are fully expecting never to be taken up on the offer.

The happy ending to this tale of licensing woe is that at least in one small case, Flock has made good on that promise.  I can't take credit for it, but I'd like to think that my constant nagging^W prodding about "being good community members" and "giving stuff back" had something to do with it.  Here's hoping we'll be able to find more "places" (again, pun intended) where Firefox can get some love back from Flock.

Blogged with the Flock Browser

Tags: flock, mozilla