The core router and switch in the William Gates Building (gatwick) seemingly crashed and rebooted at around 14:27.

This appears to have been due to a software bug triggered by a routine configuration change. Although gatwick is a virtual switch/router comprising two independent physical systems, it seems that the entire virtual switch/router (both physical systems) rebooted simultaneously.

This type of device takes a long time to reboot; in this case there would have been a little over 12 minutes during which the William Gates Building office network was cut off from the University network and the internet (followed by a further few minutes of instability). This would also have affected connectivity to filer and a few other core services hosted in the WGB.

Investigation is ongoing into the reason for this outage.

Some functions of Caelum Console are temporarily unavailable due to a server problem which will be rectified as soon as possible.

We are aware of performance problems on the storage server that hosts dev-gpu/dev-cpu VM disks and data.

One cause on a user VM causing a very high I/O load has been found and temporarily mitigated. We will continue to monitor the situation as load still seems higher than expected.