Mozilla IT

We will have a scheduled maintenance window tonight from 7pm to 9pm PDT. The following changes will take place:
* 7pm PDT (0200 UTC) - MXR software upgrade. We'll be updating MXR to add new functionality around Hg & SVN. 5 minutes of downtime is expected.
* 7pm PDT (0200 UTC) - people.mozilla.com OS upgrade. people.mozilla.com will be upgraded to RHEL 5. 2 hours of downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 12:23 PM | Permalink | TrackBacks (0)

Mozilla Scheduled Maintenance - 1/22/2008, 7pm - 8pm PST (0300 - 0400 1/23/2007 UTC)

We will have a scheduled maintenance window tonight from 7pm to 8pm PST. The following changes will take place:

* 7pm PST (0300 UTC) - addons.mozilla.org update. We'll be updating addons.mozilla.org to pick up recent API updates. No downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Posted by justin at 11:13 AM | Permalink | TrackBacks (0)

We will have a scheduled maintenance window tomorrow night from 7pm to 10pm PST. The following changes will take place:

* 7pm PST (0300 UTC) - *.mozilla.org, *.add-ons.mozilla.org cert renewal. We'll be installing a new certificate for *.mozilla.org and *.add-ons.mozilla.org. No downtime is expected.
* 7pm PST (0300 UTC) - Geo-load balancing configuration chance. We'll be changing the configuration of our load balancers to help reduce the number of RTT pings and more effectively route users to the correct data center. No downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 8:39 PM | Permalink | TrackBacks (0)

Mozilla Scheduled Downtime - 12/11/2007, 7pm - 10pm PST (0300 - 0600 12/12/2007 UTC)

We will have a scheduled downtime window tonight from 7pm to 10pm PST. The following changes will take place:

* 7pm PST (0300 UTC) - Bugzilla cluster migration. We'll be migrating our Bugzilla instance to a fault-tolerant cluster. 30 minutes or less of downtime is expected.
* 7:30pm PST (0330 UTC) - Kernel upgrades. We'll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:
- All developer and QA machines that are running RHEL (not CentOS)
* 7pm PST (0300 UTC) - Web application PHP 5 migration. We'll be migrating the following apps to PHP 5. 10 minutes or less of downtime is expected for each of the following:
- blog.mozilla.com
- labs.mozilla.com
- pastebin.mozilla.org
- quality.mozilla.org

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 9:33 AM | Permalink | TrackBacks (0)

Mozilla Scheduled Downtime - 12/6/2007, 7pm - 10pm PST (0300 - 0600 12/7/2007 UTC)

We will have a scheduled downtime window tomorrow night from 7pm to 10pm PST. The following changes will take place:

* 7pm PST (0300 UTC) - Bugzilla upgrade. We'll be upgrading bugzilla to address various security issues and feature enhancements. 30 minutes or less of downtime is expected.
* 7:30pm PST (0330 UTC) - Kernel upgrades. We'll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:
- Email for all domains we host
- Breakpad
- CVS, cvs-mirror, SVN, Hg
- quality.mozilla.org
- Bugzilla
- Bonsai
- LXR/MXR
- Tinderbox
- ftp.mozilla.org
- stage.mozilla.org
- irc.mozilla.org
- mail.mozilla.org
- lists.mozilla.org
- videos.mozilla.org
- all other dynamic and database-backed websites (addons, blog, crash-reports, developer, download(bouncer), firefoxflicks, labs, livetitles, mozilla-europe, pastebin, quality, reporter, survey, wiki, firefoxparty, worldfirefoxday, addons-facebook)

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Posted by justin at 12:58 PM | Permalink

We will have a scheduled downtime window tomorrow night from 8pm to 9pm PST. The following changes will take place:

* 8pm PST (0400 UTC) - download.mozilla.org database migration. We'll me migrating download.mozilla.org to a new set of database servers. 5 minutes or less of downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Posted by justin at 10:02 PM | Permalink

Mozilla Scheduled Downtime - 11/22/2007, 8pm - 9pm PST (0400 - 0500 11/23/2007 UTC)

We will have a scheduled downtime window tonight from 8pm to 9pm PST. The following changes will take place:

* 8pm PST (0400 UTC) - Bonsai upgrade. We'll be applying an update to Bonsai (bonsai.mozilla.org, bonsai-l10n.mozilla.org, bonsai-www.mozilla.org) to fix various security and functionality issues. Downtime is expected to be 15-30 minutes.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Posted by justin at 8:00 AM | Permalink

Mozilla Scheduled Downtime - 11/15/2007, 7pm - 10pm PST (0300 - 0600 11/16/2007 UTC)

We will have a scheduled downtime window tomorrow night from 7pm to 10pm PST. The following changes will take place:

* 7pm PST (0300 UTC) - addons.mozilla.org updates. We'll be updating code and testing the downtime notice of addons.mozilla.org. No downtime is expected.
* 7pm PST (0300 UTC) - bonsai and tinderbox server update. We'll be applying security updates to tinderbox and bonsai. 15 minutes of downtime is expected.
* 7pm PST (0300 UTC) - Kernel upgrades. Duration 4 hours. We'll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:
- VPN and Jumphosts at all locations
- any sites requiring an LDAP login
- phone services at all locations
- people.mozilla.com
- litmus.mozilla.org
- All developer and QA machines that are running RHEL

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 10:52 PM | Permalink

Mozilla Scheduled Downtime - 9/27/2007, 10pm - 2am PDT (0500 - 0900 9/28/2007 UTC)

We will have a scheduled downtime window this Thursday night from 10pm to 2am PDT. The following changes will take place:

* 10pm PDT (0500 UTC) - MediaWiki security upgrades. We'll be upgrading wiki.mozilla.org and various internal wikis to address security issues. No downtime is expected.
* 10pm PDT (0200 UTC) - stage.mozilla.org migration. Refer to http://weblogs.mozillazine.org/preed/2007/09/ftpmo_shiftreloaded.html for more details. Intermittent downtime is expected through the duration of the window.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Posted by justin at 6:50 PM | Permalink

Mozilla Scheduled Downtime - 9/13/2007, 7pm - 10pm PDT (0200 - 0500 9/14/2007 UTC)

We will have a scheduled downtime window tomorrow night from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Kernel upgrades. We'll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:

- Telephone service in the Toronto office
- Mozilla.com email (Zimbra)
- VPN and jumphosts at all locations
- people.mozilla.com
- Build Try Server
- Joey
- Any services that use LDAP authentication
- Internal DNS at the offices

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be
operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 9:11 PM | Permalink

Mozilla Scheduled Downtime - 9/11/2007, 7pm - 10pm PDT (0200 - 0500 9/12/2007 UTC)

We will have a scheduled downtime window tomorrow night from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Kernel upgrades. We'll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:
- Email for all domains we host
- Breakpad
- CVS, cvs-mirror, SVN, Hg
- quality.mozilla.org
- Bugzilla
- Bonsai
- LXR/MXR
- Tinderbox
- ftp.mozilla.org
- stage.mozilla.org
- irc.mozilla.org
- mail.mozilla.org
- lists.mozilla.org
- videos.mozilla.org
- all other dynamic and database-backed websites (addons, blog, crash-reports, developer, download(bouncer), firefoxflicks, labs, livetitles, mozilla-europe, pastebin, quality, reporter, survey, wiki, firefoxparty, worldfirefoxday, addons-facebook)

* 7pm PDT (0200 UTC) - VLAN ACL implementation. We'll be adding another layer of internal network security. No downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be
operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 8:50 PM | Permalink

Mozilla Scheduled Downtime - 8/28/2007, 7pm - 8pm PDT (0200 - 0300 8/29/2007 UTC)

We will have a scheduled downtime window this Tuesday night from 7pm to 8pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Tinderbox upgrade. Tinderbox code will be upgraded for various security fixes. 30 minutes or less of downtime is expected.
* 7pm PDT (0200 UTC) - developer.mozilla.org migration. Developer.mozilla.org will be migrated to our production cluster. No downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 5:08 AM | Permalink

Mozilla Scheduled Downtime - 8/15/2007, 7pm - 10pm PDT (0200 - 0500 8/16/2007 UTC)

We will have a scheduled downtime window tomorrow night from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Kernel upgrades. We'll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:
- LDAP logins for all sites that use LDAP
- Community build jumphost
- DNS and DHCP service in Mountain View
- wifi access in Mountain View
- VPN services at all colos and Mountain View
- people.mozilla.com
- Nagios monitoring services at all locations
- intranet.mozilla.org

* 7pm PDT (0200 UTC) - download.mozilla.org code update (Bouncer). Bouncer will be upgraded to support various new features. No downtime is expected.
* 7pm PDT (0200 UTC) - load balancer code upgrade. We'll be upgrading the code on our core load balancers to address various bug fixes. No downtime is expected.
* 8pm PDT (0300 UTC) - bugzilla upgrade. Bugzilla.mozilla.org will be upgraded to deploy various bug fixes. 15 minutes or less of downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be
operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 8:07 PM | Permalink

Mozilla Scheduled Downtime - 6/19/2007, 7pm - 10pm PDT (0200 - 0500 6/20/2007 UTC)

We will have a scheduled downtime window tonight from 7pm to 10pm
PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Pageload server migration. We'll be migrating the pageload server to new hardware (again) after a code update. No downtime is expected.
* 7pm PDT (0200 UTC) - LXR/MXR migration. LXR/MXR will move to new hardware. No downtime is expected.
* 8pm PDT (0300 UTC) - Upgrade services on irc.mozilla.org and restart Unreal. 5 minutes of downtime is expected.
* 8pm PDT (0300 UTC) - Database migrations. Spreadfirefox and wiki.m.o will be migrated to new hardware. 30 minutes of downtime is expected.

Please let me know if you have any reason why we should not proceed
with the planned maintenance. As always, we aim to keep downtime to
as little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window. Feel free to email
infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 1:04 PM | Permalink

Continue reading "Mozilla Scheduled Downtime - 6/12/2007, 7pm - 11pm PDT (0200 - 0600 6/13/2007 UTC)" »

Posted by justin at 7:28 PM | Permalink

Mozilla Scheduled Downtime - 6/7/2007, 7pm - 9pm PDT (0200 - 0400 6/8/2007 UTC)

We will have a scheduled downtime window this Thursday from 7pm to 9pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - pageload server migration. We'll be migrating the tinderbox pageload server to better, more supportable hardware - please refer to http://weblogs.mozillazine.org/preed/2007/06/farewall_axolotl_and_we_thank.html for more information. No downtime is expected.
* 7pm PDT (0200 UTC) - svn.mozilla.org downtime. svn will require downtime for maintenance. 2 hours of downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 10:16 PM | Permalink

Mozilla Scheduled Downtime - 5/31/2007, 7pm - 11pm PDT (0200 - 0600 6/1/2007 UTC)

We will have a scheduled downtime window Thursday night from 7pm to 11pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - CVS upgrade. We will be applying security updates to the libraries contained within the chroot jails used by the cvs servers. No downtime is expected.
* 7pm PDT (0200 UTC) - Planet.mozilla.org upgrade. A new version of Planet (Venus) will be deployed. No downtime is expected.
* 7pm PDT (0200 UTC) - Kernel updates. We will be doing various kernel upgrades which require a reboot. Affected systems/applications are as follows:

- FTP Staging
- IRC
- Mountain View and Toronto phone systems
- All database-backed applications

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Posted by justin at 7:27 PM | Permalink

Mozilla Scheduled Downtime - 5/24/2007, 7pm - 10pm PDT (0200 - 0500 5/25/2007 UTC)

We will have a scheduled downtime window this Thursday from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - build-graphs.mozilla.org migration. We'll be migrating build-graphs to better, more supportable hardware. No downtime is expected.
* 7pm PDT (0200 UTC) - download.mozilla.org migration. download.mozilla.org will be migrated to our San Jose colocation. No downtime is expected.
* 7:30pm PDT (0230 UTC) - Kernel updates. We will be doing various kernel upgrades which require a reboot. Affected systems/applications are as follows:

- Mozilla email services (messages should be held until services are back up)
- Anything related to CVS, Hg, SVN
- Talkback
- people.mozilla.com
- quality.mozilla.org
- developer.mozilla.org
- Bonsai
- Bugzilla
- Tinderbox
- LXR
- MXR
- Despot
- Hendrix
- Litmus

Downtime is expected to be 5 minutes or less per machine.

* 9pm PDT (0200 UTC) - Bugzilla update. Minor Bugzilla updates will be put into production. 5 minutes or less of downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Posted by justin at 8:56 PM | Permalink

Mozilla Scheduled Downtime - 5/8/2007, 7pm - 9pm PDT (0200 - 0400 5/9/2007 UTC)

We will have a scheduled downtime window tonight from 7pm to 9pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - AMO global cache testing/enabling. We'll be testing (and enabling if test go well) our global load balancing and caching with addons.mozilla.org in an effort to improve response time. There is no downtime anticipated, but as we are testing, we could run into unforeseen issues.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 11:58 AM | Permalink

We will have a scheduled downtime window tonight from 7pm to 9pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Litmus downtime. We'll be cloning the Litmus VM to bring up a staging box. Expected downtime is 2 hours.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 3:23 PM | Permalink

Mozilla Scheduled Downtime - 4/17/2007, 7pm - 8pm PDT (0200 - 0300 4/18/2007 UTC)

We will have a scheduled downtime window this Tuesday from 7pm to 8pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - CVS memory replacement. We'll be swapping out bad memory from our cvs server. Downtime is estimated at 15 minutes, but could be more if issues arise.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 4:18 PM | Permalink

Mozilla Scheduled Downtime - 3/29/2007, 7pm - 11pm PDT (0200 - 0600 3/30/2007 UTC)

We will have a scheduled downtime window this Thursday from 7pm to 11pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - CVS-WWW migration. We'll be moving cvs-www to the main cvs repo, thus changing the cvs-www root. Downtime is estimated at 2 hours, but could run the length of the window. If your old cvs connect string contained "cvs-www.mozilla.org" - you'll need to change.

Old cvs root: user%hostname@cvs-www.mozilla.org:/cvsroot
New cvs root: user%hostname@cvs.mozilla.org:/www

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Posted by justin at 7:12 PM | Permalink

Mozilla Scheduled Downtime - 3/22/2007, 7pm - 10pm PDT (0200 - 0500 3/23/2007 UTC)

We will have a scheduled downtime window this Thursday from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Addons.mozilla.org application update. We'll be updating addons to the next major revision. Expected downtime is 2 hours, but could be for the full maintenance window.

* 7pm PDT (0200 UTC) - Kernel updates. We'll updating the kernels on many production boxes. 5-10 minutes of downtime is expected per host. Services effected:

* LDAP authentication
* cvs-mirror.mozilla.org
* svn.mozilla.org
* ftp.mozilla.org
* stage.mozilla.org
* irc.mozilla.org
* bugzilla.mozilla.org
* despot.mozilla.org
* doctor.mozilla.org

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 8:44 AM | Permalink

Mozilla Scheduled Downtime - 3/15/2007, 7pm - 10pm PDT (0200 - 0500 3/16/2007 UTC)

We will have a scheduled downtime window this Thursday from 7pm to 11pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Mailman upgrades & mozilla.org relay migration. Mailman will be upgraded to fix multiple issues and moving mozilla.org mail to better hardware. Any lists ending in @lists.mozilla.org and @mozilla.org will be effected. Expected downtime is 2-3 hours, but mail will be queued for delivery. No mail will be lost.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 7:30 PM | Permalink

Mozilla Scheduled Downtime - 3/13/2007, 7pm - 11pm PDT (0200 - 0600 3/14/2007 UTC)

We will have a scheduled downtime window this Tuesday from 7pm to 11pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Database replication changes. We'll be adding additional replication functionality to one of our production databases. 1-2 hours of downtime is expected, but could be more if issues arise.

* 7pm PDT (0200 UTC) - Kernel updates. We'll updating the kernels on many production boxes. 5-10 minutes of downtime is expected per host. Services effected:

* Everything in the San Jose colocation that uses a database
* LDAP authentication on all services
* mozilla.org inbound mail
* Litmus
* Oink
* cvs-mirror
* spreadfirefox.com mail
* build-graphs
* landfill.bugzilla.org
* network connectivity for Mountain View office
* San Jose VPN connectivity

* 8pm PST (0300 UTC) - load balancer configuration change. We'll be changing the addons configuration on our load balancers. No expected downtime.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 9:55 PM | Permalink

Mozilla Scheduled Downtime - 3/6/2007, 7pm - 11pm PST (0300 - 0700 3/7/2007 UTC)

We will have a scheduled downtime window this Tuesday from 7pm to 11pm PST. The following changes will take place:

* 7pm PST (0300 UTC) - mozilla.com mail upgrade. We'll be upgrading Zimbra to 4.5.3 and testing failover. Expected downtime is 2-3 hours, but could run the length of the window.

* 7pm PST (0300 UTC) - load balancer software upgrade. We'll be upgrading the software on our load balancers. No expected downtime.

* 8pm PST (0400 UTC) - static site & cvs-mirror migration to global load balancing. The following sites will be added into the global load balancing pool: aus, fxfeeds, sfx, redirect-com, redirect-org, firefox.com, cvs-mirror. No expected downtime.

* 8pm PST (0400 UTC) - kernel upgrades. The following hosts will receive kernel upgrades and a reboot: dm-webtools02,notorious,mradm01,osadm01,dm-cvs01,osdb01,osdb02,osdb03,osdb04,khan-vm,cm-mail01,cm-mail02,dm-mail01,dm-mail02,dm-nagios01,dm-stats01,dm-stats02,dm-peep01,dm-svn01,dm-svn02,im-ns02,dracula,giles,cm-vpn01. Intermittent downtime should be expected for all of these hosts through the window.

Affected services: (all will experience downtime of 5 minutes or so)

Public sites:
* download.mozilla.org (no database access)
* mailing lists that are gatewayed to newsgroups (news will still be
available, mail will be queued, so no messages will be lost, web
interface will be unavailable)

Developer sites:
* developer.mozilla.org

* tinderbox.mozilla.org
* bonsai.mozilla.org
* lxr.mozilla.org
* mxr.mozilla.org
* webtools.mozilla.org

* cvs.mozilla.org
* svn.mozilla.org
* hg.mozilla.org

Internal sites:
* mail.mozilla.com (Zimbra) (mail will be queued, no messages lost, but
webmail, pop3, and imap will be unavailable)
* smtp.mozilla.org (mail will be queued, no messages lost, outbound mail
will be unavailable)
* people.mozilla.com
* stats.mozilla.org
* mpt-vpn.mozilla.org (any existing VPN connection will be dropped)

Additionally:
* Any site in the San Jose colo that uses LDAP authentication will have
a few minutes where any authentication attempts will result in failures.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 10:07 PM | Permalink

Mozilla Scheduled Downtime - 2/22/2007, 7pm - 10pm PST (0300 - 0600 2/16/2007 UTC)

We will have a scheduled downtime window this Thursday from 7pm to 10pm PST. The following changes will take place:

* 7pm PST (0300 UTC) - Addons.mozilla.org performance tests. We will be running performance/load tests on addons.mozilla.org. Expected downtime is 1-2 hours, but could be for the full maintenance window if more troubleshooting is needed and could effect blog.mozilla.com, firefoxflicks.com, labs.mozilla.com, spreadfirefox, survey, and wiki.mozilla.org.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 7:58 AM | Permalink

Mozilla Scheduled Downtime - 2/15/2007, 7pm - 11pm PST (0300 - 0700 2/16/2007 UTC)

We will have a scheduled downtime window this Thursday from 7pm to 11pm PST. The following changes will take place:

* 7pm PST (0300 UTC) - Addons.mozilla.org application update. We ran into some unexpected code and infrastructure issues with the last release which have been resolved. Expected downtime is 2 hours, but could be for the full maintenance window if more troubleshooting is needed.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 6:04 PM | Permalink

Mozilla Scheduled Downtime - 2/8/2007, 7pm - 10pm PST

We will have a scheduled downtime window this Thursday from 7pm to 10pm
PST. The following changes will take place:

* 7pm PST - Mozilla.com/org GSLB turn-up. Mozilla.com/org web sites
will be migrated to globally load balanced addresses. No downtime is
expected.
* 7pm PST - Webtools migration. We will be migrating all applications
on webtools01 (bonsai, bonsai-l10n, tinderbox, lxr, mxr, browser-buster)
to new hardware. 1 hour of downtime is expected.
* 8pm PST - Bugzilla memory upgrade. We'll be adding memory to our
bugzilla server. 5-15 minutes of downtime is expected.

Please let me know if you have any reason why we should not proceed with
the planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window. Feel free to email
infra at mozilla.org if you see issues past the planned downtime.

-Justin

Posted by justin at 9:55 PM | Permalink

Mozilla Scheduled Downtime - 2/6/2007, 8pm - 9pm PST

We will have a scheduled downtime window this tonight from 8pm to 9pm PST. The following changes will take place:

* 8pm PST - Migrate mozilla.com to DynDNS proxy. We will be migrating external DNS response to DynDNS who will proxy to our name servers. No downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

-Justin

Posted by justin at 9:59 AM | Permalink

Mozilla Scheduled Downtime - 1/25/2007, 7pm - 10pm PST

We will have a scheduled downtime window this Thursday night from 7pm to 10pm PST. The following changes will take place:

* 7pm PST - Global load balancing tests. We will be running some tests on our global loadbalancing setup to ensure proper functionality. Intermittent downtime of www.mozilla.com/org is possible for the duration of the window, but not expected.
* 8pm PST - Wordpress upgrade. We will be upgrading blogs.mozilla.com to a new version of WordPress. Estimated downtime: 5 min

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

-Justin

Posted by justin at 5:18 PM | Permalink

Mozilla Scheduled Downtime - 12/7/2006, 7pm - 10pm PST

We will have a scheduled downtime window this coming Thursday from 7pm to 10pm PST. The following changes will take place:

* 7pm PST - Load balancer software upgrade. We will be updating the code on our primary load balancers to address specific known issues. There is no estimated downtime as they are in a fault tolerant configuration, but downtime is possible.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

-Justin

Posted by justin at 9:27 PM | Permalink

Mozilla Scheduled Downtime - 10/17/2006, 7pm - 10pm PDT (0200 - 0500 UTC)

We will have a scheduled downtime window this coming Thursday from 7pm
to 10pm PDT. The following changes will take place:

* 7pm PDT - SAN/NAS interfaces changes. We need to make modifications
to the network setup of one of our storage arrays. Small periods of
downtime may be seen of the following systems: mozilla.com mail,
talkback, cvs, and other systems that rely on this array. CVS will be
rebooted as part of this to fix an iLo issue. Downtime should be
minimal.
* 7pm PDT - spreadfirefox.com migration. We will be migrating SFX to
our California datacenter. Up to 2 hours of downtime is expected.
* 8pm PDT - mozilla.org mail migration. We will be migrating
mozilla.org mail to our California datacenter. No downtime is expected.

Please let me know if you have any reason why we should not proceed with
the planned maintenance. As always, we aim to keep downtime to as
little as possible, but unexpected complications can arise causing
longer downtime periods than expected. All systems should be
operational by the end of the maintenance window. Feel free to email
infra@mozilla.org if you see issues past the planned downtime.

Posted by justin at 5:24 PM | Permalink

Mozilla Scheduled Downtime - 9/21/2006, 7pm - 10pm PDT (0200 - 0500 UTC)

We will have a scheduled downtime window this coming Thursday from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT - Database migrations. We have 2 new database servers to help reduce some of the load. Databases to migrate are: addons, devmowiki, firefoxflicks, labs, quality_wiki, reporter, userpanel and survey. Applications may be down or in a read-only state for the duration of the window.
* 8pm PDT - cm-vpn01 network change. cm-vpn01 will need to be rebooted for a network change. 15 min of downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Posted by justin at 3:29 PM | Permalink

Mozilla Scheduled Downtime - 9/14/2006, 6pm - 12pm PDT (0100 - 0700 UTC)

We will have a scheduled downtime window this coming Thursday from 6pm to 12pm PDT. The following changes will take place:

* 6pm PDT - Stage machine/storage migration. Both stage.mozilla.org and ftp.mozilla.org will be effected - both sites may be unavailable for periods during this window. See http://wiki.mozilla.org/Build:StageMigrationNotes for details on what is changing.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Posted by justin at 2:36 PM | Permalink

Mozilla Scheduled Downtime - 9/12/2006, 7pm - 10pm PDT (0200 - 0500 UTC)

We will have a scheduled downtime window this coming Tuesday from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT - Update firewall code. This is needed to address a few minor software issues. No expected downtime, but intermittent network connectivity is possible during the window.
* 7pm PDT - Intranet wiki migration. All internal wikis will be migrated to new database and app servers to improve performance and scalability. 30-60 min of downtime is expected but could extend through the window.
* 8pm PDT - developer.mozilla.org migration. D.m.o will be moved behind our caching loadbalancer farm to improve performance, scalability and reliability. No downtime is expected.
* 8pm PDT - Move mozilla.com mail to internal mail relays. Mozilla.com email will be re-routed to our internal mail relays (no end-user effect). No downtime is expected but delayed email is possible during the downtime window.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Posted by justin at 8:28 PM | Permalink

HP believes we are running into a 2 platform hardwares bug with our CVS server (recommended memory layout changes and ASR issues - http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1024238&admit=-682735245+1156660583117+28353475). We have been provided a new memory layout and a work-around for the ASR issue - will be implementing these changes tonight a 9pm PDT via a reboot. Estimated downtime is between 15 and 30 minutes - please let me know if you have any conflicts or issues with the reboot.

We apologize for the instability and recent maintenance to CVS. We are working around the clock to get this system to a stable and reliable state.

Posted by justin at 1:10 PM | Permalink

Mozilla Scheduled Downtime - 8/22/2006, 7pm - 10pm PDT (0200 - 0500 UTC)

We will have a scheduled downtime window this coming Tuesday from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT - NAS network settings changes. We will be updating network settings on our NAS. No expected downtime, but intermittent network filesystem access is possible through the window.
* 7pm PDT - Webtools migration. Bonsai, LXR and Tinderbox will be migrated to a new machine. 30-60 min of downtime is expected but could extend through the window.
* 8pm PDT - Updating firewall rules to internal networks. This will help improve network isolation and security. No expected downtime, but intermittent network connectivity is possible.
* 8pm PDT - SSH access to/from mecha will be disabled. If you use this host as a jumphost, you need an account on the new VPN server (announced multiple times).

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Posted by justin at 4:41 PM | Permalink

Mozilla Scheduled Downtime - 8/10/2006, 8pm - 11pm PDT (0300 - 0600 UTC)

We will have a scheduled downtime window this coming Thursday from 8pm to 11pm PDT. The following changes will take place:

* 8pm PDT - One of our primary database servers will be brought down to replace faulty hardware. This may effect addons, bonsai, firefoxday and other dynamic sites. Expected downtime is less than 15 minutes, but could run for the full window if issues are encountered.
* 8pm PDT - Adding firewall rules to internal networks. This will help improve network isolation and security. No expected downtime, but intermittent network connectivity is possible.
* 9pm PDT - irc.mozilla.org ssl certificate upgrade and software upgrade. 5 minutes or less of downtime is expected.
* 9pm PDT - NAS network settings changes. We will be updating network settings on our NAS. No expected downtime, but intermittent network filesystem access is possible for the remainder of the maintenance window.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Posted by justin at 9:30 PM | Permalink

Mozilla Scheduled Downtime - 8/5/2006, 8am - 12pm PDT (1500 - 1900 UTC)

We will have a scheduled downtime window this coming Saturday from 8am to 12pm PDT. The following changes will take place:

* 8am PDT - addons.mozilla.org will be migrated to our new datacenter. Expect intermittent availability and read-only versions of the site for the length of the scheduled window.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 8:04 AM | Permalink

Just a friendly reminder of the upcoming maintenance this weekend for our datacenter migration (original notice). Also please keep in mind the following:

* We will be auditing user accounts as we migrate each machine. We will be removing accounts which have not been used or we think are not needed. If this is done in error, please open a ticket and let us know who you are and why you need access to the system.

* Machines will be re-addressed (IP). Please check all scripts and logins to make sure they work after the migration. You may get ssh warnings upon login due to the IP changes.

* Firewall rules have been audited and cleaned up. While we made every effort to maintain all network access needed, we could have missed a rule or deleted something in error. Again, if you feel something has been closed/removed in error, please open a ticket.

All of these changes are meant to enhance security and stability of our systems. Please let us know if have any issues after the move.

Posted by justin at 9:27 AM | Permalink

On Saturday, 7/15/2006 we will be moving our primary datacenter to new facilities. This will include internal and external addressing changes and physically moving to new networks. We have prioritized systems in order of impact to users and will make every effort to bring high priority systems up first. The following applications/systems will be effected by the move (not in order of priority):

Build systems, QA systems, CVS prim