Mozilla IT

Mozilla Scheduled Downtime - 3/29/2007, 7pm - 11pm PDT (0200 - 0600 3/30/2007 UTC)

We will have a scheduled downtime window this Thursday from 7pm to 11pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - CVS-WWW migration. We'll be moving cvs-www to the main cvs repo, thus changing the cvs-www root. Downtime is estimated at 2 hours, but could run the length of the window. If your old cvs connect string contained "cvs-www.mozilla.org" - you'll need to change.

Old cvs root: user%hostname@cvs-www.mozilla.org:/cvsroot
New cvs root: user%hostname@cvs.mozilla.org:/www

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Posted by justin at 7:12 PM | Permalink

As many of you know, we have attempted to release a new version of addons a few times. With each attempt the webdev team makes incremental & substantial improvements, but we haven't been able to get the application to a reasonable performance level.

Most of the issue is that we simply have a difficult time reproducing the massive amount of traffic that comes into the site on a daily basis. This, coupled with the complexity of a complete application rewrite, has kept us from formally releasing.

In order help the webdev team better debug and pinpoint the performance issues, we will be periodically switching the production site to new code at low load times to test with real world traffic. The site will be totally functional and these switches should result in no downtime. Once the release is final, we'll be sure to announce it far and wide.

Thanks again for your continued patience and support - we are all working hard to deliver a quality amo release.

-Justin

Posted by justin at 1:11 PM | Permalink

Mozilla Scheduled Downtime - 3/22/2007, 7pm - 10pm PDT (0200 - 0500 3/23/2007 UTC)

We will have a scheduled downtime window this Thursday from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Addons.mozilla.org application update. We'll be updating addons to the next major revision. Expected downtime is 2 hours, but could be for the full maintenance window.

* 7pm PDT (0200 UTC) - Kernel updates. We'll updating the kernels on many production boxes. 5-10 minutes of downtime is expected per host. Services effected:

* LDAP authentication
* cvs-mirror.mozilla.org
* svn.mozilla.org
* ftp.mozilla.org
* stage.mozilla.org
* irc.mozilla.org
* bugzilla.mozilla.org
* despot.mozilla.org
* doctor.mozilla.org

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 8:44 AM | Permalink

Mozilla Scheduled Downtime - 3/15/2007, 7pm - 10pm PDT (0200 - 0500 3/16/2007 UTC)

We will have a scheduled downtime window this Thursday from 7pm to 11pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Mailman upgrades & mozilla.org relay migration. Mailman will be upgraded to fix multiple issues and moving mozilla.org mail to better hardware. Any lists ending in @lists.mozilla.org and @mozilla.org will be effected. Expected downtime is 2-3 hours, but mail will be queued for delivery. No mail will be lost.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 7:30 PM | Permalink

Mozilla Scheduled Downtime - 3/13/2007, 7pm - 11pm PDT (0200 - 0600 3/14/2007 UTC)

We will have a scheduled downtime window this Tuesday from 7pm to 11pm PDT. The following changes will take place:

* 7pm PDT (0200 UTC) - Database replication changes. We'll be adding additional replication functionality to one of our production databases. 1-2 hours of downtime is expected, but could be more if issues arise.

* 7pm PDT (0200 UTC) - Kernel updates. We'll updating the kernels on many production boxes. 5-10 minutes of downtime is expected per host. Services effected:

* Everything in the San Jose colocation that uses a database
* LDAP authentication on all services
* mozilla.org inbound mail
* Litmus
* Oink
* cvs-mirror
* spreadfirefox.com mail
* build-graphs
* landfill.bugzilla.org
* network connectivity for Mountain View office
* San Jose VPN connectivity

* 8pm PST (0300 UTC) - load balancer configuration change. We'll be changing the addons configuration on our load balancers. No expected downtime.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 9:55 PM | Permalink

Mozilla Scheduled Downtime - 3/6/2007, 7pm - 11pm PST (0300 - 0700 3/7/2007 UTC)

We will have a scheduled downtime window this Tuesday from 7pm to 11pm PST. The following changes will take place:

* 7pm PST (0300 UTC) - mozilla.com mail upgrade. We'll be upgrading Zimbra to 4.5.3 and testing failover. Expected downtime is 2-3 hours, but could run the length of the window.

* 7pm PST (0300 UTC) - load balancer software upgrade. We'll be upgrading the software on our load balancers. No expected downtime.

* 8pm PST (0400 UTC) - static site & cvs-mirror migration to global load balancing. The following sites will be added into the global load balancing pool: aus, fxfeeds, sfx, redirect-com, redirect-org, firefox.com, cvs-mirror. No expected downtime.

* 8pm PST (0400 UTC) - kernel upgrades. The following hosts will receive kernel upgrades and a reboot: dm-webtools02,notorious,mradm01,osadm01,dm-cvs01,osdb01,osdb02,osdb03,osdb04,khan-vm,cm-mail01,cm-mail02,dm-mail01,dm-mail02,dm-nagios01,dm-stats01,dm-stats02,dm-peep01,dm-svn01,dm-svn02,im-ns02,dracula,giles,cm-vpn01. Intermittent downtime should be expected for all of these hosts through the window.

Affected services: (all will experience downtime of 5 minutes or so)

Public sites:
* download.mozilla.org (no database access)
* mailing lists that are gatewayed to newsgroups (news will still be
available, mail will be queued, so no messages will be lost, web
interface will be unavailable)

Developer sites:
* developer.mozilla.org

* tinderbox.mozilla.org
* bonsai.mozilla.org
* lxr.mozilla.org
* mxr.mozilla.org
* webtools.mozilla.org

* cvs.mozilla.org
* svn.mozilla.org
* hg.mozilla.org

Internal sites:
* mail.mozilla.com (Zimbra) (mail will be queued, no messages lost, but
webmail, pop3, and imap will be unavailable)
* smtp.mozilla.org (mail will be queued, no messages lost, outbound mail
will be unavailable)
* people.mozilla.com
* stats.mozilla.org
* mpt-vpn.mozilla.org (any existing VPN connection will be dropped)

Additionally:
* Any site in the San Jose colo that uses LDAP authentication will have
a few minutes where any authentication attempts will result in failures.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.com if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 10:07 PM | Permalink