02232008_outage.html

Feb 23rd, 2008 Outage

THIS WAS A SCHEDULED OUTAGE

9:52pm - A *mostly* all clear has been issued. PBO GPS & BSM Dataflow and real-time GPS is not available quite yet. An email will be sent when this changes. This will be the last update to this page for this particular outage. Please check your email for future updates... Thanks!

7:40pm - All Infrastructure and Facility services are up. PBO services are being tested, and we're awaiting confirmation of functionality. All clear will be sent very soon now...

6:50pm - All infrastructure services are up, and it appears the clear majority of the Facility and PBO services are up, but awaiting confirmation from their respective groups. An all clear will be sent this evening, but it may not be sent until 8-8:30pm. At this point services will not go down, we'll only see services coming online from this point. UNAVCO IT staff are now offsite.

6:10pm - Most all infrastructure services are up. Most PBO & Facility services are in the process of coming up. Email, DNS, Windows domain, VPN, and NTP are now up and running and will remain so. An all clear will probably be issued within the hour. Thanks again for bearing with us!

5:20pm - All contractors were offsite around 3:40pm. Staff required to rework some of the electrical feeds from PDUs. Most all power is restored to racks. Network is coming up. Look for services to come back in the next few hours. Email, and other services may not be available yet. Or if they are, might go away, before becoming steady. Thanks for your patience.

2:40pm - Most machines are cabled up to the new power system. Anticipate powering on the majority of gear in the next 2 hours. Network test, then servers online. Contractors offsite soon.

10:30am - Most of the old power bus & PDU are in the warehouse. Staff are now verifying new power bus locations, and cabling up machines to new PDU's/UPS's. Fire monitoring systems are still running on battery and are being tapped into the new power bus now.

7:40am - Power in machine room off -- uncabling old power bus commences

7:00am - Contractors show up -- hold until power is ready to be pulled from old power bus

5:30am - Staff onsite, and PDU work continues