Mozilla IT

HP believes we are running into a 2 platform hardwares bug with our CVS server (recommended memory layout changes and ASR issues - http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1024238&admit=-682735245+1156660583117+28353475). We have been provided a new memory layout and a work-around for the ASR issue - will be implementing these changes tonight a 9pm PDT via a reboot. Estimated downtime is between 15 and 30 minutes - please let me know if you have any conflicts or issues with the reboot.

We apologize for the instability and recent maintenance to CVS. We are working around the clock to get this system to a stable and reliable state.

Posted by justin at 1:10 PM | Permalink

Mozilla Scheduled Downtime - 8/22/2006, 7pm - 10pm PDT (0200 - 0500 UTC)

We will have a scheduled downtime window this coming Tuesday from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT - NAS network settings changes. We will be updating network settings on our NAS. No expected downtime, but intermittent network filesystem access is possible through the window.
* 7pm PDT - Webtools migration. Bonsai, LXR and Tinderbox will be migrated to a new machine. 30-60 min of downtime is expected but could extend through the window.
* 8pm PDT - Updating firewall rules to internal networks. This will help improve network isolation and security. No expected downtime, but intermittent network connectivity is possible.
* 8pm PDT - SSH access to/from mecha will be disabled. If you use this host as a jumphost, you need an account on the new VPN server (announced multiple times).

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Posted by justin at 4:41 PM | Permalink

Mozilla Scheduled Downtime - 8/10/2006, 8pm - 11pm PDT (0300 - 0600 UTC)

We will have a scheduled downtime window this coming Thursday from 8pm to 11pm PDT. The following changes will take place:

* 8pm PDT - One of our primary database servers will be brought down to replace faulty hardware. This may effect addons, bonsai, firefoxday and other dynamic sites. Expected downtime is less than 15 minutes, but could run for the full window if issues are encountered.
* 8pm PDT - Adding firewall rules to internal networks. This will help improve network isolation and security. No expected downtime, but intermittent network connectivity is possible.
* 9pm PDT - irc.mozilla.org ssl certificate upgrade and software upgrade. 5 minutes or less of downtime is expected.
* 9pm PDT - NAS network settings changes. We will be updating network settings on our NAS. No expected downtime, but intermittent network filesystem access is possible for the remainder of the maintenance window.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Posted by justin at 9:30 PM | Permalink

Mozilla Scheduled Downtime - 8/5/2006, 8am - 12pm PDT (1500 - 1900 UTC)

We will have a scheduled downtime window this coming Saturday from 8am to 12pm PDT. The following changes will take place:

* 8am PDT - addons.mozilla.org will be migrated to our new datacenter. Expect intermittent availability and read-only versions of the site for the length of the scheduled window.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Thanks,
-Justin

Posted by justin at 8:04 AM | Permalink