The Inside Track on Firefox Development.

« Congrats Relicensing Project! | Main | Firefox Version Numbers »

April 7, 2006

Our Next Challenge?

The past year or so has been interesting. In this time, I've been able to meet a lot of new people and learn a lot of new things. Most importantly is that for the first time in as long as I can remember, I have had a chance to see the Mozilla project from the outside, as it would appear to someone who was trying to build on Mozilla technology or contribute directly to a project, but was not part of Netscape "back in the day" or an employee of the Mozilla Foundation/Corporation itself.

It's been an illuminating experience. From a technical perspective, it helped highlight APIs that I had developed without a clear understanding of how they would be used. The Extension Installation API was one example of this, and we were able to make some great improvements to it in 2005.

But perhaps more importantly it has shed some light on how people perceive Mozilla as an open source project. These perceptions are not the sort of things people express explicitly. You have to notice them.

The Difficulty of Involvement

This is sort of the uber-perception. I think some of the reasons for this include the following:

Where is the Discussion?

Which newsgroup/mailing list/IRC channel/wiki/talk page/bug/forum page do I need to track in order to know what's going on in a specific area? The answer is unsatisfactorily complex.

The traditional method of joining a project in the OSS world (where you join lists and IRC channels and lurk for a while, gradually ramping up your contributions) scales uneasily to a project the size of Mozilla. The amount of data a mere mortal would have to absorb in order to be productive quickly is staggering. I have in the past jocularly referred to it as the "learning wall". I wonder how many people just give up.

Madness to Method?

As a large project, Mozilla has thousands of source files across hundreds of directories. One of my coworkers here at Google commented that he tried to find something as simple as the browser window code a couple of years ago and couldn't, because it lived under the thoughtfully named "xpfe".

There's not a huge amount of documentation - and I'm not just talking about public API docs. I'm talking about the much needed diagrams that show how the various building blocks fit together, and in-code documentation for pretty much anything that isn't intuitive (which is a lot). I've written as little of this as anyone else.

Tone

In the past, I have not done the best I could to establish a tone for discourse that is conducive to productive development. My tendency was to snap when provoked. I made two mistakes of judgement here, one was ignoring the effect that this sort of thing would have on those watching, aside from the victim. The other was to think that regardless of the tone set by my actions, we as a group could work through any negative effects. Any work we relied on others for we could do ourselves. Or we could hire through it.

The Joy of Code

The flaw with this is that when your project's contributions come solely from companies, for better or for worse the activities of those paid contributors will align in some way with the interests of those companies. What this does not always allow for is the pursuit of the sort of improvements that are outside the scope of these interests. Such things often include raising general code quality, speculative feature development, feature polish and detail etc. I don't mean to say that companies are against these things, but they're often not the primary concern during a release crunch. And what companies like to have is shipping software.

Alternately, even in the absence of corporate support, if there is not enough redundancy that the same set of folk has to do the grunt work over and over, the risk of burnout is high.

I feel this because I have been incredibly "plan" focused over the past few years, formally during my time at Netscape and less formally but no less importantly during the run up to Firefox 1.0 and 1.5. What I notice is that I no longer have time to work on the sort of interesting side projects that I used to enjoy doing when I was first starting out.

For example, about six years ago I discovered a bug in the Bookmarks menu shortly after scrolling was implemented. When you moused into a submenu for a folder that was in the scrolled section, the sub menu popup was pushed off the bottom of the screen. I took a couple of days to learn the menu positioning code and fix the math error that was causing the bug. The exercise was good for me in a number of ways: I learned more about another section of the code, my general expertise was raised, and well.. I fixed the bug that was bothering
me.

I think we need to have a project that is accessible to volunteers for this reason. We also need to provide a way to allow those volunteers to grow if they want to, so that if you're one of the folk at the center you can have a chance to step aside for a moment and take a breather and code for the pure joy of it.

Full time paid contributors will always be a part of Open Source development. But I don't think release-focused agendas will ever be a substitute for the passion of folk who participate because of the joy of exploration and of contribution.

Looking Outward, Looking Forward

As a project, we have made overtures towards being a more inclusive lot. For some of the reasons I've listed here, I think as a project we're still more inward looking than outward. How many of us have thought about what we want to be doing in 5 years? Will we always be doing this? Will our roles remain the same? My opinion is that it's fast becoming time for us to start considering making personal sacrifices in our short term conveniences to make the project more accessible to new people. Do I know what we need to do? Not exactly. But I have some ideas: find ways to make our discussions, our public faces, and our code more accessible.

With Firefox we did an excellent job of building a world class product that people wanted to use. We have a new challenge now, one that is larger and scope and in the long run in my opinion considerably more important because the long term success of products like Firefox depend on it. How will we grow a world class development community? How will we ensure that the freedoms we enjoy are protected, forever?

Posted by ben at April 7, 2006 9:22 AM

Comments

Ben, I really think you're on the right track here. Mozilla/Firefox is building more than a piece of software. It's building a model for how large-scale, distributed, free software creation can continue sustainably. That might be even more important than the software itself. I hope your thinking and writing here is indicative of some serious thought and work going on at the foundation along these lines.

Posted by: Realish at April 7, 2006 9:53 AM

I was especially happy to read the Tone section; I think a nice tone goes a long way in alleviating many other problems.

Something that didn't catch my eye, but seems to be a major problem for people who actually go through the trouble to produce patches: those patches don't get integrated on a timely basis, or at all. Often times nobody will even look the patches. In cases where they are looked at, it often takes so long that the patch misses the release the developer meant it for. Slow responsiveness will turn people away, and make them hostile to the project (spreading bad publicity etc.) I know it is a lot of work to respond to patches, but I think it would be worth the investment to be more responsive towards them and the people who made them.

Posted by: Heikki Toivonen at April 7, 2006 11:25 AM

Best post yet! I don't know the answer for most of those questions, though do have my own experience with some.

It's nice to see someone else who recognizes some of the problems encountered in working from the outside (and your still more inside than most outsiders).

Posted by: Robert Accettura at April 7, 2006 11:58 AM

I think it's great that you and other Mozilla hackers are coming around to this conclusion. Although I am curious as to if and when it will have a practical impact...I have a patch in Bugzilla that has been awaiting review from you since 2004! This is not a thinly veiled attempt to get your attention; I'm sure the patch is bitrotted beyond recognition, and it's a trivial bug. Just saying that as a would-be casual Firefox contributor, it has been and continues to be excruciatingly difficult to "scratch your own itch". Anything you guys are doing to alleviate that problem would be great : ).

Posted by: Jon Henry at April 7, 2006 11:59 AM

Great post Ben!

It's a great chellange to keep people who are the community, once we're not anymore a "alternative, small, new project", not a forge, but a mature project with money involved around.
OS people who spend time volunteering to the projects usually prefer the former case.

Another thing is, will Mozilla open for new developers/volunteers? Everytime I speak with people (even here! Inside the Flock! I hear that stepping in to code for Mozilla project is very hard, and Mozilla project is very closed) - that's totally oposite from what happened to me (thanks Boris and Benjamin and Pike!), but I heard it so many times, from so many people that I tend to assume that my case was the rare one, and theirs are the common... Mozilla must do sth about it.

Posted by: gandalf at April 7, 2006 3:26 PM

Sometimes the drivers are not as communicative as they might be. One can contact them, but sometimes they give only a minimal responses, when just a little information could save everyone (including them) a lot of effort.

Posted by: AnotherGuest. at April 7, 2006 3:43 PM

Although you talk about some problems introduced by structural difficulties and lack of structural documentation, these are really just sub-issues in a general problem for a lot of open source projects (independent of scale!) that I'd call "initial investment".

The issue is how much effort is required for somebody to fix their first bug / add their first feature. Once you get used to the code base, these things may not be costs at all (they may even be advantages). But it is a barrier to ever getting "casual" development. Unfortunately, on many open source projects, the initial investment is extremely large. One place where this isn't a problem for Firefox is code that is entirely in one of the little languages, e.g extensions done with Javascript and XUL. But if you go want to go any deeper into the codebase, initial investment becomes a big issue, and I'm not sure it's something you can fix at this point. (The boundary between the two is even worse; when I wanted to figure how exactly some extension worked (I think it was magpie download tools), I was never ever able to figure out how all the pieces fit together.)

The two most significant issues are structure (which you talk about), which you can think about as understanding a tree/graph structure, and toolkit, which you can think about as being the leafy components you use to build up the system. Before STL came around, everybody rolled their own container libraries. Mozilla has created XPCOM, and I'm sure there are a number of other fundamental "leaf libraries" that it uses (it's been a long time since I looked at the code, so I don't know the details). Even a project as seemingly simple as the w3c's http library is incredibly daunting because of this sort of thing; it may have made the overall codebase cleaner, more effective, and more bug-free, but it adds a huge, huge cost to "casual" development (in the sense of first-time development).

As much as people hated the original NS4 code base, I was able to download it, build, do a little grepping, and add a minor feature I wanted (but only to the windows front-end, since they were all separate) in literally a couple of hours. Every time I've downloaded Mozilla/Firefox to consider fixing a bug (because there are tons of things I gripe about, which I'd be perfectly happy to fix myself), I've ended up putting in a few hours and then bailed, unwilling to spend the effort required to get familiar with the codebase. Just the sheer complexity of downloading-and-building something for the first time can be off-putting; many open source projects depend on many other libraries but do not include them in the source build, which means getting started becomes really painful (at least in windows; and I don't know if this is specifically true for Firefox).

Some of these issues can be addressed by documentation, but the issues with using lots of specific-to-the-project libraries is pretty much unconquerable, except by "don't do that", which isn't really an option at this point. Perhaps thoroughly documenting them from the perspective of not requiring people to understand them completely, only well enough to start hacking, would help.

Posted by: Sean Barrett at April 7, 2006 6:14 PM

In a post on her blog, Mitchell Baker mentioned a desire to transfer some of the Firefox related revenue coming to the Mozilla Corporation back to the community, perhaps through grant-making and donations programs. I think her and your post dovetail nicely in that re-investment of fund back into the community could help address some of your concerns. It's great to see a successful organisation explore these thoughts. However grants and donations, especially when targeted at individual efforts and developers, don't always provide the best ROI. I rather see the Mozilla organisation invest in the community at large, using one of the best practises in good investing: diversification.

I see three excellent opportunities for investment that would intrinsically diversify itself across the broader Mozilla community:

Education: Encouraging knowledge sharing and growth.
Tools: Lowering the barriers to entry into the community.
Infrastructure: Mimicking the internal support department found in large corporations.

EDUCATION:

Invest in the creation of an addons.mozilla.org like site where people can share any material they think might be useful to others.

Allow visitors to award points to entries they found useful.
Let contributors earn rewards for their efforts by exchanging points they have banked over time for Mozilla store credits or other goods. Award special prizes to the entry with the most points awarded each month.
Encourage the diversification of material through monthly/quarterly competitions:

Best JS / C++ / XUL sample code.
Best classroom course material featuring Firefox extension or XULrunner programming.

Sponsor the creation of books targeted at novice and intermediate programmers. If there exists a "Firefox for Dummies" then why doesn't there exists a "Firefox Extensions Building for Dummies" or "XUL Programming for Dummies"?

Get those thousands of MySpace users sharing simple extensions on their sites. Provide templates for simple extensions. For example: a step by step by step guide on modifying a base toolbar extension so it displays thumbnails of one's friends and maybe a supplementary "advance" tutorial that turns those thumbnails into links. Give the general public some toys to play with and see what happens. Make it easy for them to explore what their own imaginations can create.

TOOLS:

Setting up a Mozilla build environment is a daunting task unrelated to programming skill. Automating this task with a build environment installer would open up the community to a much wider pool of potential contributors. Similarly, provide pre-configured development software - compilers, debuggers. Be it open source solutions available free of charge or commercial solutions available at a price, the primary goal is to provide tools that allow contributors to concentrate on developing, not configuring.

INFRASTRUCTURE:

Large corporations have IS&T department that handle software installation and hardware configuration for their employees. The Mozilla community needs a virtual IS&T department, a team that helps relieve community members of those burdens by providing regularly updated and easily found tools like the build environment installer and pre-configured development software.

Also missing in the community but found in most large corporations is the internal "Help Desk" - a single resource where one can turn to with *any* question and be *guaranteed* to receive a response, if not immediately then eventually. Build a site where one could submit a question, be presented with a list of answers from similar questions submitted previously, and if still not satisfied, submit the question to an "Unanswered" queue.

Let the questions be anything:

"Does a bug report already exist for this problem?"
"How do I change my homepage?"
"Who owns the APIs for nsIGobbledygook?"
"Who is 'the server' and why is Firefox looking for him?"

Allow community members to answer questions in the queue.

Allow question submitters to award points to answers they found useful.
Let answerers earn rewards for their efforts by exchanging points they have banked over time for Mozilla store credits or other goods. Award special prizes to the answerers with the most points awarded each month

Posted by: Ian Pottinger at April 7, 2006 9:56 PM

I appreciate your humility and honesty. As a student who hasn't had much time to look over the codebase, but could potentially be interested in contributing to the Mozilla project after I graduate, seeing this type of direction for the organization is encouraging.

Posted by: Grant Hutchins at April 7, 2006 10:13 PM

A much more significant barrier to entry is that the Mozilla organization wastes contributors' time: Much work, including many patches and certainly most bug triaging, never amounts to anything -- it never gets reviewed/approved/fixed/etc. Volunteers aren't doing it for the money; they do it because they want to accomplish something. Donating time on something that never comes to fruition is pointless.

The lack of interpersonal skills amazes me: I can't think of an organization where I've had consistently worse experiences. Has any commercial business, or any polite person, ever treated you the way some Mozilla employees treat volunteers? At least Microsoft employees behave with normal, adult politeness when I deal with them. Why would anyone behave so rudely?

Mozilla has become a Cathedral: It's arrogant, rejects outsiders, and does much of its work behind closed doors. That must compromise the benefits of the Bazaar, such as the 'many eyes' that make bugs shallow. Personally, I now invest most of my volunteer time elsewhere. It's only because the project is so valuable that I even keep an eye on it.

Posted by: guanxi at April 8, 2006 7:37 AM

actually, guanxi, I'd recommend you to limit your anger because it's a very per case issue. Many people, including me, got amazing amount of help from Mozilla employes which allowed me to start working on Mozilla project, and, in result, got me into working on Flock.

As I said in my previous comment, I know that there's a lot o oposite cases, but I'd recomment not to generalize. Mozilla has a lot of wonderfull people who always accept to share their time between their actual work and helping newbies know stuff that they should read in FAQ's, but they're too lazy to do so.
I never met with any kind of "arogance" on #developers channel, while at least for a few times I met with it in KDE project, Gnome project, Wikipedia, and... many, many times, in PHP project.
So, it's not that bad, really :)

Posted by: gandalf at April 8, 2006 4:25 PM

Don't worry about me -- I have no personal complaints -- or about defending the project. Let's simply make it better:

We have a problem: What do we do about it?

Posted by: guanxi at April 8, 2006 10:22 PM

Bugs could be marked by lead developers with pointers to the following:

1) which files the developer thinks are good places to look (always and labouriously with what those files are for eg. NGLayout.c - this is the layout engine - its a good place to look for the reflow bug you having etc. etc.)

2) hunches on what could be wrong

3) a level of probable difficulty for the bug

4) a helper contact who is happy to be bugged by the person/people investigating

5) a private comments section for those people investigating the bug where people actually write down how they solved the bug in brief once they had fixed it.

6) a damn pretty web 2.0 overview of the starting point bugs that you guys could and do probably fix blind folder with hands tied behind you back typing with your nose. Don't fix them - put them up on this site with a deadline before you fix them (you would be the contact about this bug and you could make suggestions such as files to look in variables to check etc.) Add a points system that as you do more bug fixes you earn points and get a higher rating, eventually being able to work up to monied bugs.

7) Bugzilla is really damn ugly - would as many people use firefox if it was ugly?

Just my 10p and I make no claims that any of the above are practical/will work :-)

Posted by: Andrew Phillipo at April 9, 2006 10:25 AM

I've also had a patch bit rot, though this was many years before Firefox.

It may be that the maintainers - like you Ben - need to accept that in future your role may be as patch reviewer and leader, not writing new features. How many new code does Linus write? Not much. He spends most of his time doing code review and making sure patches from others flow smoothly. I think that's normal for any project of this size, and Mozilla has traditionally neglected it.

Posted by: Mike Hearn at April 11, 2006 5:37 AM

This Nielsen piece gave me the idea that it might be useful to usability test how the code is organized and documented:
http://www.useit.com/alertbox/maturity.html

Posted by: Ian at April 24, 2006 7:51 AM

Your blog is such interesting stuff KaylaX

Posted by: Mitchell at April 26, 2006 9:45 AM

Excellent post, Ben. There are some issues you raise here that i've been thinking about for the last while, and i think i, partially, have some answers. Two things: Consistency and Documentation.

Posted by: gary at April 26, 2006 6:50 PM

©1997-2006 Ben Goodger. All Rights Reserved.
Opinions expressed here are my own, and not those of any organization that I may be affiliated with.
Reload icon is © Stephen Horlander;
Firefox logo is by Jon Hicks, and is a trademark of The Mozilla Foundation.
GetFirefox buttons are from rakaz