Completed
William Gates Building planned power outage

Status
Resolved after about 8 hours
Started
November 25, 2023 at 8:00 AM
Completed
November 26, 2023 at 1:21 AM
Affects
Network
Virtual Machine Hosting
GPUs
Datacentres
GN09
  • Completed
    November 25, 2023 at 3:35 PM
    Completed
    November 25, 2023 at 3:35 PM

    This maintenance has been completed successfully.

    If any IT systems are still not working, please contact sys-admin. If any electrical circuits remain down, please contact building-services.

  • Update
    November 25, 2023 at 12:46 PM
    In progress
    November 25, 2023 at 12:46 PM

    Power to the building is gradually being restored. GN09 servers will gradually be powered back up when the cooling has reached temperature.

    Office power and networking in parts of the building will remain down as the maintenance work in wiring cupboards is ongoing.

  • In progress
    November 25, 2023 at 10:03 AM
    In progress
    November 25, 2023 at 10:03 AM

    The start of the electrical work was delayed by two hours due to a generator fault. The work is now under way but is likely to run past 12:00.

  • Planned
    November 25, 2023 at 8:00 AM
    Planned
    November 25, 2023 at 8:00 AM

    The William Gates Building will be without power for the morning of 25th November, due to planned work on our electrical switch gear to facilitate the upcoming commissioning of a substantial amount of solar power generation.

    Electrical circuits in the William Gates Building datacentre, GN09, which are connected via the UPS should remain powered throughout the maintenance, running from a backup generator.

    Other electrical circuits will go down; in particular, a few research servers are connected only to non-UPS circuits. A list of these will be circulated.

    GN09 will also be without active cooling for the duration of this work; we will have temporary air blowers in place to reduce the buildup of hot air but we may need to shut down high-powered servers (for example GPU and FPGA servers) depending on the weather and temperature.

    The following list of machines in GN09 will lose power during this maintenance. If possible, please shut them down before the maintenance starts (otherwise we will try to shut them down by pressing the power button). Once we have announced that the maintenance is complete, you can start them again from the Caelum Console. Please wait for an announcement that the maintenance is complete before attempting to do so.

    Some other servers may be shut down as well, in particular GPU and FPGA servers, to reduce the electrical and/or thermal load in GN09.

    • virtual machines on the GPU cluster (dev-gpu-…, dev-cpu-… and others as notified separately)
    • grumpy
    • gxp06
    • tarawera
    • ngongotaha
    • all quorum servers
    • stix
    • story
    • L51 Raspberry Pi cluster
    • godzilla
    • tiger
    • baume
    • ctsrd-slave2
    • rama
    • cat
    • chericloud-switch
    • rado
    • wenger
    • wolf0/1/2
    • edale
    • glencoe
    • sakura
    • ran
    • nana
    • momo
    • gilling
    • sigyn
    • idun
    • heimdall
    • nikola01/02/03/04
    • acritarch
    • morello101-dev/102-dev/103-dev
    • sleepy
    • doc
    • sherwood
    • behemoth
    • leviathan
    • excalibur
    • bam
    • kinabalu
    • daintree
    • marpe
    • iphito
    • doris
    • asteria
    • all POETS servers
    • mauao
    • any other GPU or FPGA server observed to be drawing a lot of power on Friday evening or Saturday morning
  • Update
    November 25, 2023 at 7:08 AM
    In progress
    November 25, 2023 at 7:08 AM

    Shutdown of listed servers is now beginning.

  • In progress
    November 24, 2023 at 10:14 PM
    In progress
    November 24, 2023 at 10:14 PM

    VMs on the GPU cluster are now shutting down. They can be started again from Xen Orchestra once the electrical work is finished, tomorrow afternoon.

  • Planned
    November 24, 2023 at 6:12 PM
    Planned
    November 24, 2023 at 6:12 PM

    Shutdown of GPU VMs will begin at 22:00 today (Friday).

    Shutdown of affected physical servers will begin at 07:00 on Saturday morning.