Mozilla IT

Mozilla Scheduled Downtime - 6/15/2006, 7pm - 10pm PDT (0200 - 0500 UTC)

We will have a scheduled downtime window this coming Thursday from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT - Spreadfirefox disk replacement. We will be replacing a failed disk on the SFX machine. Downtime should be less than 5 minutes.
* 7pm PDT - MySQL security update. We will be patching our MySQL instances on mrdb01/02 and osdb01/02/03/04 - apps affected: addons, bugzilla, bonsai, reporter, spreadfirefox, all wikis, MDC and any other clustered database-backed systems. Expected downtime is 10 minutes.
* 8pm PDT - Continued DNS architecture migration. We will be migrating our primary DNS server to a new distributed system. We don't anticipate any downtime, but sporadic DNS service is possible.
* 8pm PDT - Mirror migration. We will be implementing various mirror module changes in coordination with our master mirrors TDS and OSL. We don't anticipate any downtime.
* 8:30pm PDT - Bonsai upgrades. We will be upgrading Bonsai on mecha and rheet (bonsai.mozilla.org and bonsai-www.mozilla.org). We anticipate 5-10 minutes of downtime.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Posted by justin at 9:31 PM | Permalink

We'd like to announce some proposed changes to how we mirror and
distribute our various products and builds. Our current mirror size is
approaching 500gb - much too large and costly to ask our mirrors to
support. Thus, we have come up with a new and improved mirror strategy
to help alleviate the space and bandwidth issues associated with
supporting such a large mirror site while retaining our ability to
distribute our software in a quick and efficient manner.

We will take comments until 6/8/06 with implementation scheduled for
6/9/06 assuming no major objections. Please take the time to read
through the changes and let us know if you have any concerns with the
proposed plan. Note - there should be no end-user effecting changes.
All of these modifications will be transparent to the user.

---
* There will be 2 rsync modules/data repositories:
o Releases - this will be all releases which have been
released with the last year. This includes rc's, beta's and alpha's. All
bouncer files will be contained within this module.
o FTP - this will contain all of the releases module plus all
old releases (beta's, alpha's and rc's) and any nightly builds.
ftp.mozilla.org and archive.mozilla.org will go to this data set.

* Anyone who would like to mirror Releases will *have* to mirror
from releases-rsync.mozilla.org (TDS and OSL - as most should be doing
now). We haven't enforced this and there are a few people who are still
mirroring off surf (i.e. stage). This has a few bad ramifications around
virus scanning, bandwidth, etc - we'll just start enforcing that people
use the correct rsync mirrors.

* Master mirrors (OSL and TDS) and standard mirrors won't rsync the
FTP module, instead only the releases module. This will mean that all
ftp.mozilla.org traffic will come directly to Mozilla. Our main mirrors
do not want to keep this much data online and we can easily handle
traffic. Any high-traffic builds/releases should be in the releases module.
---

We are in discussions with our major mirror sites to make sure the
changes are implemented with minimal downstream impact. Again, please
let me know if you have any questions/concerns/issues with the new setup.

Posted by justin at 4:39 PM | Permalink | Comments (1)