University of Cambridge Computer Laboratory - Notice history

All systems operational

100.0% uptime

Caelum Console (server management) - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

Request Tracker - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

Other Internal Services - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

External Services - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

Network - Operational

99.99% uptime
Mar 2024 · 100.0%Apr · 99.96%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 202499.96% uptime
May 2024100.0% uptime
100.0% uptime

GN09 - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

WCDC - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime
100.0% uptime

Main VM Pool (WCDC) - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

GPUs - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

Secondary VM Hosts - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

Xen Orchestra - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime
100.0% uptime

Filer - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

Archive Server - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

Data Replication - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime

Other Secondary Storage Systems - Operational

100.0% uptime
Mar 2024 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2024100.0% uptime
Apr 2024100.0% uptime
May 2024100.0% uptime
100.0% uptime

Third Party: Fastmail → General Availability - Operational

Third Party: Fastmail → Mail delivery - Operational

Third Party: Fastmail → Web client and mobile app - Operational

Third Party: Fastmail → Mail access (IMAP/POP) - Operational

Third Party: Fastmail → Login & sessions - Operational

Third Party: Fastmail → Contacts (CardDAV) - Operational

Notice history

View current status

May 2024

MAY20
DS-Print maintenance
CompletedMaintenance1 hour
  • Completed20 May, 2024 at 07:30UTC
    Maintenance has completed successfully
  • In progress20 May, 2024 at 06:30UTC
    Maintenance is now in progress
  • Planned20 May, 2024 at 06:30UTC

    UIS has announced scheduled maintenance on the DS-Print service on the morning of Monday 20th May. Printing to the DS-Print printers in the Department will not be possible between 7:30am and 8:30am, and multi-function devices will be unresponsive. The Department's locally managed printers will be unaffected.

    UIS are replacing a security certificate and installing an update for one of the technologies we use to provide the service.

MAY6
WGB emergency network maintenance
CompletedMaintenance27 minutes
  • Completed06 May, 2024 at 22:42UTC
    Maintenance has completed successfully.
  • In progress06 May, 2024 at 22:15UTC
    Maintenance is now in progress
  • Planned06 May, 2024 at 22:15UTC

    We will be updating the software on the core router/switch in the William Gates Building (gatwick) in order to attempt to mitigate the ongoing crashes (https://cl.instatus.com/clvva1e4b43187b8n2hqywstc0). This upgrade cannot be performed "live", so there will be approximately 20-30 minutes' outage of the William Gates Building office network, and of filer. Other servers in GN09 should be largely unaffected.

WGB network problem under investigation
ResolvedPartial outage4 hours 32 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    The core switch/router in the William Gates Building (gatwick) appears to have crashed and rebooted; perhaps a reoccurrence of issues a month ago (https://cl.instatus.com/clur1lte237417blopt08gvelk). Networking should have returned (initially via one switch of the redundant pair that constitutes gatwick, whilst the other switch restarts).

Apr 2024

APR19
RT upgrade
CompletedMaintenance7 hours 8 minutes
  • Completed19 April, 2024 at 23:08UTC
    Maintenance has completed successfully.
  • In progress19 April, 2024 at 16:00UTC
    Maintenance is now in progress
  • Planned19 April, 2024 at 16:00UTC

    We will be upgrading RT during this time. Email to sys-admin, building-services and other email addresses that use ticket numbers will be delayed and not acted upon until the upgrade is complete.

    The upgrade may take all weekend as it involves a time-consuming database conversion.

gatwick (WGB core network) crashed
ResolvedPartial outage119 hours 31 minutes
  • ResolvedUTC
    This incident has been resolved.
  • UpdateUTC

    gatwick crashed and rebooted again at around 06:22, again triggered by a routine configuration update.

    We had, earlier in the night, attempted to install a software update but due to an unrelated issue, the routers refused to do an 'In Service Software Upgrade' - i.e. the upgrade would have caused more disruption - so we chose to roll back and delay this update until Cisco published their notes about this particular version.

  • UpdateUTC

    The routers have remained stable since the crash, but we're going to do some further testing out-of-hours, and install a software update. There may be some further disruption whilst that happens.

  • UpdateUTC

  • MonitoringUTC

    The core router and switch in the William Gates Building (gatwick) seemingly crashed and rebooted at around 14:27.

    This appears to have been due to a software bug triggered by a routine configuration change. Although gatwick is a virtual switch/router comprising two independent physical systems, it seems that the entire virtual switch/router (both physical systems) rebooted simultaneously.

    This type of device takes a long time to reboot; in this case there would have been a little over 12 minutes during which the William Gates Building office network was cut off from the University network and the internet (followed by a further few minutes of instability). This would also have affected connectivity to filer and a few other core services hosted in the WGB.

    Investigation is ongoing into the reason for this outage.

Mar 2024

Reduced Caelum Console functionality
ResolvedDegraded performance8 hours 39 minutes
  • ResolvedUTC

  • MonitoringUTC

    A fix has been implemented for the misbehaving Caelum live state database (which had failed, filled the disk with error logs, and caused more problems).

  • IdentifiedUTC

    Some functions of Caelum Console are temporarily unavailable due to a server problem which will be rectified as soon as possible.

Lab GPU clusters: poor I/O performance
ResolvedDegraded performance1 hour 54 minutes
  • ResolvedUTC

    We believe this issue has been resolved, and are in communication with the user who accidentally caused the poor performance. We're also considering hardware upgrades for the storage server.

  • MonitoringUTC

    We are aware of performance problems on the storage server that hosts dev-gpu/dev-cpu VM disks and data.

    One cause on a user VM causing a very high I/O load has been found and temporarily mitigated. We will continue to monitor the situation as load still seems higher than expected.

Previous

Mar 2024 to May 2024

Next