Justin's Blog

We recently ran an experiment, code named funnelcake (see polvi's blog post for more details) - this was an interesting project from IT's perspective for a few reasons.

First a little background - for one 24 hour period, we would need to serve *all* en-US and de downloads which originate from our website - not a small number. We estimate ~500k downloads a day overall, with a large percentage being en-US and de. Why would we want to host the downloads when we have an excellent mirror network setup, happily serving up our bits? We were interested in gathering statistics on how many people started, aborted or completed the downloads. We could do some of this by adding an FTP server of our own into bouncer, but is much more interesting to get an idea of the behavior seeing *all* the traffic. Also, we can correlate the logs later to number of active users and website behavior. Plus 24 hours won't kill my 95th percentile bandwidth bills :-)

Second, seeing all of the traffic allows us to get a great view of the diversity, amount and frequency of downloads. As you'll see below, it was quite an increase in our normal traffic.

Third, it's a great test to stress test our infrastructure, verifying we don't have any unexpected bottlenecks or performance issues. The good news here is the systems passed with flying colors.

Our setup was pretty simple - we built out three download servers with the archive.mozilla.org nfs share mounted. Slapped apache on them, added them to bouncer and we were off to the races. Here are the traffic graphs (you can probably tell when we switched things over):

Furthermore, Apache really impressed me. The servers were pushing upwards of 80mbs each off nfs, with a load of... 0.00 and cpu hovering around 5%. We sometimes got the occasional 0.10 spike, but all in all, pretty amazing. Graphs from one of the machines:

All in all, I was very happy with lack if impact on the systems and continued good performance.