Justin's Blog

I was reminded of the power of open source software yet again this weekend. A little background:

We here at Mozilla are big fans of OpenVPN. When we rebuilt are datacenter, we did a large search for the right VPN solution. Mozilla's requirements were somewhat specific:

* Had to work with all three platforms (mac, linux, windows)
* Needed to work with our LDAP infrastructure (i.e. not AD)
* Needed to work through NAT
* We needed to be able to give each user granular per-host access
* We wanted a solution that would allow just Mozilla traffic to traverse the VPN rather than forcing all traffic through the VPN

We looked at many options, most of which were commercial closed-source solutions (given the lack of options). Ideally, a client-less, SSL-based solution would have been ideal, but it was clear Firefox (!) and Mac support was not ready. We decided on OpenVPN as it met all of our requirements and had the added benifit of being open source and free!

< /background >

We've been happily using openvpn with TunnelBlick as our mac client. Justdave even created a custom installer for our users (pretty slick Dave :-) ). But along comes Leopard - with changes such that the low level network drivers don't function anymore (along with other issues in the GUI). With some research, mrz found that a OS X tuntap development team just released new drivers which support Leopard. Still, openvpn won't connect, TunnelBlick won't run, etc, so this weekend I set out to fix the issues. After 3-4 hours of figuring out how the TunnelBlick build setup works, fixing some bugs and adding in the new drivers, I have a working version of TunnelBlick, openvpn and tuntap drivers on Leopard.

What's the point of this rant? I could have *never* fixed this with a closed source VPN client. I'd be hamstrung by Cisco (yes, Cisco John) or some other network vendor while they gave me the normal story that Mac is not a large enough platform to dedicate resources too (nevermind that 90+% of Mozilla engineers use Mac hardware). Being able to look at the source, build system and composition of each of these apps made it possible to figure out what the issue was, fix it, and post this build for anyone else who needs it.

Makes me remember why what we do here at Mozilla is so important. So, if you need a Leopard version of TunnelBlick (with tuntap drivers and openvpn 2.0.9 with lzo support), here you go.

Posted by justin at 11:54 AM | Permalink

We recently ran an experiment, code named funnelcake (see polvi's blog post for more details) - this was an interesting project from IT's perspective for a few reasons.

First a little background - for one 24 hour period, we would need to serve *all* en-US and de downloads which originate from our website - not a small number. We estimate ~500k downloads a day overall, with a large percentage being en-US and de. Why would we want to host the downloads when we have an excellent mirror network setup, happily serving up our bits? We were interested in gathering statistics on how many people started, aborted or completed the downloads. We could do some of this by adding an FTP server of our own into bouncer, but is much more interesting to get an idea of the behavior seeing *all* the traffic. Also, we can correlate the logs later to number of active users and website behavior. Plus 24 hours won't kill my 95th percentile bandwidth bills :-)

Second, seeing all of the traffic allows us to get a great view of the diversity, amount and frequency of downloads. As you'll see below, it was quite an increase in our normal traffic.

Third, it's a great test to stress test our infrastructure, verifying we don't have any unexpected bottlenecks or performance issues. The good news here is the systems passed with flying colors.

Our setup was pretty simple - we built out three download servers with the archive.mozilla.org nfs share mounted. Slapped apache on them, added them to bouncer and we were off to the races. Here are the traffic graphs (you can probably tell when we switched things over):

Furthermore, Apache really impressed me. The servers were pushing upwards of 80mbs each off nfs, with a load of... 0.00 and cpu hovering around 5%. We sometimes got the occasional 0.10 spike, but all in all, pretty amazing. Graphs from one of the machines:

All in all, I was very happy with lack if impact on the systems and continued good performance.

Posted by justin at 12:00 PM | Permalink

As many of you may know, we released a major update to offer 1.5.0.12 -> 2.0.0.4. This is significant to the infrastructure for a few reasons. First off, all of these updates will be *full* updates, i.e. full browser downloads - no 300k mar file. That puts a large load on our mirror network (nothing they can't handle :-) ). Second, as people update from1.5.0.12 -> 2.0.0.4, we anticipate people will need to update addons for compatibility reasons.

We released at 3pm PDT yesterday - here are some of the stats so far:

* Just under 1 million people have been updated from 1.5.0.12 -> 2.0.0.4
* We are updating people at a rate of 20-30 per second (FF2 download rate was about 30/second)
* Mirrors are seeing about a 2-3x increase in bandwidth

I expect to see increased load throughout the day, but so far so good! Huge thanks to webdev with their help to optimize addons.mozilla.org - it's been a huge win for this release.

Posted by justin at 5:56 AM | Permalink

Here at Mozilla, we have been pretty happy VMWare customers (aside from some of the p2v migration hell). We are moving quite a bit of our infrastructure to VMs and it seems to be working out well.

Enter VMWare 3.0.

You might ask "why would you want to use the first rev a such a new release"? Good point - problem is, VMWare only supports one, that's right one 4 port ethernet card (Intel quad port). No problem - I order our servers with the supported Intel PRO/1000 Quad Port Server Adapter. This adapter has worked fine in other older boxes so I have no reason to think it won't work. Well, Intel has discontinued the "MT" version of the cards, and only sells the "GT" version of the cards. After 5 days of tedious interaction with VMware web support, we find why 2.5.3 won't see the card - 2.5.3 only supports the out of date and impossible to get "MT" version of the card.

VMWare 2.5.3 only supports one quad port card that you can't get, with no ETA on "GT" version support for 2.5.x - great - guess we'll try 3.0. So we get 3.0 install and it recognizes the "GT" version of the card - yay.

Now the real fun starts - I go to the web admin console that I have used in the past to configure the host - wait - there is nothing here?!? No network configuration, no VM config options, nothing! It appears VMWare has moved all the configuration to a .Net, windows-only application called "Infrastructure Client". I use a Mac, much of my team runs Linux. Why in the world would you take a web-based, platform independent configuration tool and move it to a .Net application?

So you ask why I need Parallels to run VMWare? That's why (I'm on a Mac, and need a Windows OS to configure VMWare). And it sucks. I think VMWare has a really great product and one that can help scale our infrastructure, but why focus the admin tools on just one OS? Seems the whole company is built around virtualization and cross-platform support, but has taken a step backwards with this change in 3.0. Hopefully VMWare will fix this issue by producing upcoming versions of Infrastructure Client by supporting it on different platforms.

Posted by justin at 8:10 AM | Permalink | Comments (1)