Justin's Blog

I was reminded of the power of open source software yet again this weekend. A little background:

We here at Mozilla are big fans of OpenVPN. When we rebuilt are datacenter, we did a large search for the right VPN solution. Mozilla's requirements were somewhat specific:

* Had to work with all three platforms (mac, linux, windows)
* Needed to work with our LDAP infrastructure (i.e. not AD)
* Needed to work through NAT
* We needed to be able to give each user granular per-host access
* We wanted a solution that would allow just Mozilla traffic to traverse the VPN rather than forcing all traffic through the VPN

We looked at many options, most of which were commercial closed-source solutions (given the lack of options). Ideally, a client-less, SSL-based solution would have been ideal, but it was clear Firefox (!) and Mac support was not ready. We decided on OpenVPN as it met all of our requirements and had the added benifit of being open source and free!

< /background >

We've been happily using openvpn with TunnelBlick as our mac client. Justdave even created a custom installer for our users (pretty slick Dave :-) ). But along comes Leopard - with changes such that the low level network drivers don't function anymore (along with other issues in the GUI). With some research, mrz found that a OS X tuntap development team just released new drivers which support Leopard. Still, openvpn won't connect, TunnelBlick won't run, etc, so this weekend I set out to fix the issues. After 3-4 hours of figuring out how the TunnelBlick build setup works, fixing some bugs and adding in the new drivers, I have a working version of TunnelBlick, openvpn and tuntap drivers on Leopard.

What's the point of this rant? I could have *never* fixed this with a closed source VPN client. I'd be hamstrung by Cisco (yes, Cisco John) or some other network vendor while they gave me the normal story that Mac is not a large enough platform to dedicate resources too (nevermind that 90+% of Mozilla engineers use Mac hardware). Being able to look at the source, build system and composition of each of these apps made it possible to figure out what the issue was, fix it, and post this build for anyone else who needs it.

Makes me remember why what we do here at Mozilla is so important. So, if you need a Leopard version of TunnelBlick (with tuntap drivers and openvpn 2.0.9 with lzo support), here you go.

Posted by justin at 11:54 AM | Permalink

We recently ran an experiment, code named funnelcake (see polvi's blog post for more details) - this was an interesting project from IT's perspective for a few reasons.

First a little background - for one 24 hour period, we would need to serve *all* en-US and de downloads which originate from our website - not a small number. We estimate ~500k downloads a day overall, with a large percentage being en-US and de. Why would we want to host the downloads when we have an excellent mirror network setup, happily serving up our bits? We were interested in gathering statistics on how many people started, aborted or completed the downloads. We could do some of this by adding an FTP server of our own into bouncer, but is much more interesting to get an idea of the behavior seeing *all* the traffic. Also, we can correlate the logs later to number of active users and website behavior. Plus 24 hours won't kill my 95th percentile bandwidth bills :-)

Second, seeing all of the traffic allows us to get a great view of the diversity, amount and frequency of downloads. As you'll see below, it was quite an increase in our normal traffic.

Third, it's a great test to stress test our infrastructure, verifying we don't have any unexpected bottlenecks or performance issues. The good news here is the systems passed with flying colors.

Our setup was pretty simple - we built out three download servers with the archive.mozilla.org nfs share mounted. Slapped apache on them, added them to bouncer and we were off to the races. Here are the traffic graphs (you can probably tell when we switched things over):

Furthermore, Apache really impressed me. The servers were pushing upwards of 80mbs each off nfs, with a load of... 0.00 and cpu hovering around 5%. We sometimes got the occasional 0.10 spike, but all in all, pretty amazing. Graphs from one of the machines:

All in all, I was very happy with lack if impact on the systems and continued good performance.

Posted by justin at 12:00 PM | Permalink

I'm about mid way through my first trip to China (in Beijing) - first time to the far east for that matter, and I have to say it's a pretty interesting place. I've been all over europe and north america, but what has really struck me is how Beijing is similar to many other major international cities I've been to. Sure it's got it's unique attractions, food, people and activities - but isn't so different that I can't function or don't know how to fit in - in fact quite the opposite.

Now let me preface this by saying I am in the outer section of the city in a tech park, and haven't had time to go into the heart of the city (which I hope to do). But on the 12 hour (!) plane ride over, I had this notion that coming to China would be extremely exotic with very different ways of doing things.

Sure, the Internet access is not the best (i.e Great Firewall, international congestion, etc), food can be...adventurous (chicken neck, frog, snail, turtle, donkey, and others were all on the menu at tonight's restaurant), the weather & pollution aren't the best, politics aren't in line with what I'd vote for, but all in all - it's just a city, and a great one at that. People eat and hang out a lot, get work done in similar fashions and live their lives.

I think the differences in how people work, live, and interact in different cultures is incredibly interesting - hence why I think I am enjoying my time here so much. The trip has really highlighted that while there are a lot of differences in the way we choose to live, we often forget just how similar we all are :-)

More technical (read: nerdy) posts later on the Great Firewall, Internet access, colo's, and more.

Posted by justin at 5:09 AM | Permalink