University of Cambridge Computer Laboratory - Power outage affecting William Gates Building – Incident details

Power outage affecting William Gates Building

Resolved
Major outage
Started 5 months agoLasted 187 hours 23 minutes

Affected

Internal Services

Operational from 11:12 PM to 11:21 PM, Major outage from 11:21 PM to 2:12 AM, Operational from 2:12 AM to 6:36 PM

Caelum Console (server management)

Operational from 11:12 PM to 11:21 PM, Major outage from 11:21 PM to 2:12 AM, Operational from 2:12 AM to 6:36 PM

Other Internal Services

Operational from 11:12 PM to 11:21 PM, Major outage from 11:21 PM to 2:12 AM, Operational from 2:12 AM to 6:36 PM

Datacentres

Partial outage from 11:12 PM to 11:21 PM, Major outage from 11:21 PM to 2:13 AM, Operational from 2:13 AM to 6:36 PM

GN09

Partial outage from 11:12 PM to 11:21 PM, Major outage from 11:21 PM to 2:13 AM, Operational from 2:13 AM to 6:36 PM

Virtual Machine Hosting

Operational from 11:12 PM to 11:21 PM, Major outage from 11:21 PM to 2:13 AM, Operational from 2:13 AM to 6:36 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident was previously resolved.

  • Monitoring
    UTC
    Monitoring

    WGB and GN09 infrastructure is operational again. Most GN09 servers and services have been restored.

    It is likely that some systems may not have started correctly, or may have entered a broken state if running elsewhere but relying on systems in GN09. This is especially likely if they use filer or other networked storage that lost power. Contact service-desk@cst.cam.ac.uk to report any issues.

    The UPS supplying power to part of GN09 has apparently suffered a fault whilst powering back up after the outage, and one third of its capacity is currently inoperative; resilience is reduced until repairs can be arranged.

  • Update
    UTC
    Update

    Power appears to have been restored, and some systems have come back up automatically, however full restoration of services will take time and will only begin once we are confident that the power supply is stable and the UPS batteries have had some time to recharge.

    We are aware of at least one apparent hardware failure arising from this outage so far.

  • Identified
    UTC
    Identified

    Servers in GN09 not connected to a UPS supply, and machines in offices, have lost power. Servers in GN09 connected to a UPS supply are likely to lose power in a few minutes. UK Power Networks has confirmed the outage and expects power to be restored around 01:30-02:30. Most departmental IT services will be unavailable until further notice.

  • Investigating
    UTC
    Investigating

    We are aware of a possible site-wide power outage affecting the William Gates Building, including servers in GN09.