University of Cambridge Computer Laboratory - Notice history

100% - uptime

Caelum Console (server management) - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime

Request Tracker - Operational

100% - uptime
May 2025 · 100.0%Jun · 98.65%Jul · 100.0%
May 2025100.0% uptime
Jun 202598.65% uptime
Jul 2025100.0% uptime

Other Internal Services - Operational

100% - uptime
May 2025 · 99.98%Jun · 99.59%Jul · 100.0%
May 202599.98% uptime
Jun 202599.59% uptime
Jul 2025100.0% uptime

External Services - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime

Network - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 99.99%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 202599.99% uptime
100% - uptime

GN09 - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime

WCDC - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime
100% - uptime

Main VM Pool (WCDC) - Operational

100% - uptime
May 2025 · 99.95%Jun · 99.59%Jul · 100.0%
May 202599.95% uptime
Jun 202599.59% uptime
Jul 2025100.0% uptime

GPUs - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime

Secondary VM Hosts - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime

Xen Orchestra - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime
100% - uptime

Filer - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime

Archive Server - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime

Data Replication - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime

Other Secondary Storage Systems - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%
May 2025100.0% uptime
Jun 2025100.0% uptime
Jul 2025100.0% uptime
100% - uptime

Third Party: Fastmail → General Availability - Operational

Third Party: Fastmail → Mail delivery - Operational

Third Party: Fastmail → Web client and mobile app - Operational

Third Party: Fastmail → Mail access (IMAP/POP) - Operational

Third Party: Fastmail → Login & sessions - Operational

Third Party: Fastmail → Contacts (CardDAV) - Operational

Notice history

Jul 2025

Various services inaccessible from the internet
  • Resolved
    Resolved

    We believe this incident has been resolved.

  • Monitoring
    Monitoring

    UIS implemented a fix and we are currently monitoring the result. Please report to service-desk@cl.cam.ac.uk CC mas90@cam.ac.uk if you are aware of any remaining networking issues or unavailable services.

  • Update
    Update

    UIS are working on fixing this for us. Some services that were being blocked by UIS have now been repaired. However inbound email to departmental systems such as Request Tracker and the legacy email platform is still blocked.

  • Update
    Update

    Mail to the small number of users still on the legacy filer-based email platform (i.e. using ~/.forward to route email) is also currently not working as a result of this incident. This affects: djg11 ejb1 fhk1 jac22 jmb25 km10 pb22 pes20 rnc1

    Though we hope that UIS will rectify this for us soon, note that the legacy email platform will not last for much longer anyway and these people should plan their migration to another system. We can upon request reconfigure your email to forward directly to another address, for example your @cam.ac.uk Exchange Online mailbox, bypassing your .forward. Pending restoration of the IT service desk, please contact mas90@cam.ac.uk from your @cam.ac.uk address or from a known external email address (as your @cl/@cst address will not be working) if you would like to discuss this.

  • Update
    Update

    This outage is also preventing email from reaching departmental request-tracker services including the IT service desk, Building Services, Purchasing etc..

  • Identified
    Identified

    Since UIS's network maintenance this morning, they appear to have blocked some connections into the departmental network that had previously been allowed. For example, the CL MSA is currently unreachable from the internet so depending on your configuration you may be unable to send mail, and DNS servers on our network are unreachable. We are querying this urgently with UIS.

University network maintenance: complete internet outage
  • Completed
    15 July, 2025 at 15:28
    Completed
    15 July, 2025 at 15:28
    Maintenance has completed successfully.
  • In progress
    15 July, 2025 at 06:00
    In progress
    15 July, 2025 at 06:00
    Maintenance is now in progress
  • Planned
    15 July, 2025 at 06:00
    Planned
    15 July, 2025 at 06:00

    UIS have announced network maintenance on 15th-16th July which will result in the entire University having no internet connection for at least an hour starting at 7am on 15th July. They aim to have the work completed by 9am on 15th July but may overrun.

    Anyone working locally, for example in their office, will experience disruption, as the internet and cloud services will be inaccessible. Many on-premises systems will also be inaccessible as they rely on cloud services for some functions such as authentication of user accounts. This will affect all wired and wifi connections in the William Gates Building, including eduroam and wgb.

    Anyone working remotely and trying to connect to on-premises University services (such as CHRIS, CUFS and any cl.cam.ac.uk system such as filer, slogin or an office computer) will experience disruption, as University servers will be inaccessible from the internet.

    Anyone working remotely but only connecting to cloud services (for example, Microsoft applications such as Teams, SharePoint, Office 365, Exchange/Outlook) should not experience any disruption.

    The full announcement from UIS follows:

    "We will be undertaking work to update and enhance our network service, equipment and security posture on Tuesday 15 July, between 07:00 and 09:00. We’re also reserving 16 July for any additional work we need to complete, but we aim to be finished on 15 July.

    There will be some disruption to network connectivity between the University Data Network (UDN) and the internet during this work. Users should plan to avoid critical work on the network during the maintenance period.

    This work includes the network maintenance that we postponed in December 2024.

    What’s happening

    We plan to change the network infrastructure between Janet and the UDN. This will include new border routers, network address translation (NAT) infrastructure, and a replacement for the intrusion prevention system (IPS). This will cause disruption to network connectivity between the UDN and the internet.  

    There will be disruption for some users between 07:00 and 09:00 on 15 July, as follows:

    • Users working remotely and connecting to cloud services (for example, Microsoft applications such as Teams) should not experience any disruption. 

    • Users working remotely and connecting to on-premise University services (such as CHRIS, CUFS) will experience disruption. 

    • Users working on the University network (for example, working in the office) will experience disruption to the internet and cloud services.

    • Connectivity within the University network, (for example, working in an office and connecting to a University service such as CHRIS) should not experience any disruption.

    There will be changes to the way the central NAT service is configured. Read further details about our NAT service.

    What you should do 

    Users should plan to avoid critical work on the network during 07:00 to 09:00 on the morning of 15 July. This includes hybrid meetings on the University estate. IT officers should advise their users of this disruption. 

    We will issue a reminder about this work a week before it is scheduled to take place.

    Any issues? 

    If you have any queries about this work, please contact the Service Desk."

    (Note that for the Department of Computer Science and Technology, the changes to the NAT service only affect eduroam.)

Jun 2025

GPU VM cluster maintenance
  • Completed
    22 June, 2025 at 14:50
    Completed
    22 June, 2025 at 14:50
    Maintenance has completed successfully.
  • Update
    22 June, 2025 at 14:33
    In progress
    22 June, 2025 at 14:33

    Some capacity to run user VMs is now back online. You may try to start your VM again via Xen Orchestra if you need it. If it fails to start, try again after half an hour when there should be more capacity available.

  • Update
    22 June, 2025 at 13:51
    In progress
    22 June, 2025 at 13:51

    Storage server maintenance is complete. The shared server dev-gpu-2 is coming back up. VM hypervisor upgrades are now beginning so personal dev-* VMs will remain down.

  • In progress
    22 June, 2025 at 13:00
    In progress
    22 June, 2025 at 13:00
    Maintenance is now in progress
  • Planned
    22 June, 2025 at 13:00
    Planned
    22 June, 2025 at 13:00

    The GPU VM cluster which hosts dev-gpu-* and dev-cpu-* virtual machines, and the associated storage server, requires some urgent software updates and hardware maintenance in order to rectify a couple of known problems. We propose to do this on Sunday; however this could be rescheduled if this would be particularly disruptive (contact mas90 ASAP if so).

    All dev-gpu-* and dev-cpu-* VMs plus the shared servers dev-gpu-1, dev-gpu-2, dev-cpu-1, dev-gpu-acs and dev-cpu-acs must be shut down during this maintenance, as the storage server that holds VM disks and home directories will be unavailable for a short time. Capacity to host VMs will gradually be restored during the maintenance as each VM host is updated and brought back online.

VM storage server repair (xene-pool1)
  • Completed
    17 June, 2025 at 16:49
    Completed
    17 June, 2025 at 16:49
    Maintenance has completed successfully.
  • In progress
    17 June, 2025 at 16:00
    In progress
    17 June, 2025 at 16:00
    Maintenance is now in progress
  • Planned
    17 June, 2025 at 16:00
    Planned
    17 June, 2025 at 16:00

    Following on from the earlier unscheduled VM storage outage, we need to replace a failed memory module in the storage server that backs one of our departmental VM pools in order to restore performance and reliablity.

    This requires us to shut down all VMs on xene-pool1, which will affect the following departmental services:

    • cl-student-ssh - Undergraduate SSH server

    • MSA (partial outage, one of two servers affected)

    • Request Tracker

    • VPN2 (partial outage, one of two servers affected and new connections are already steered towards the other server)

    • Departmental database server (SQL Server / svr-win-db / db-*)

    • Windows Remote Desktop service

    • dbwebserver

    • WSUS (Windows Updates)

    And it will affect the following user VMs:

    • cl-teaching-ecad

    • dev-compilers0

    • egress

    • knot

    • lmserv-mentor

    • svr-papers

    • svr-www-ecad

    • svr-yg386-web

    These will be shut down soon after 5pm and will remain off for approximately an hour. The at-risk window is given as 2.5 hours due to uncertainty with the exact timing.

    We will take the opportunity to do some routine maintenance (software and firmware updates) of the storage system at the same time, in order to avoid a future need to do more scheduled maintenance.

VM storage fault (xene-pool1)
  • Resolved
    Resolved
    This incident has been resolved. However the same VMs will need to be shut down when a replacement part arrives. This will be communicated separately.
  • Monitoring
    Monitoring
    The fault has been mitigated and affected VMs are now back online. The VMs will have to be shut down again within a few days to replace a failed hardware component. Some users connected to VPN2 may be disconnected shortly as one of the VPN gateway servers needs rebooting even though it is still partially working. Besides this, please contact [service-desk@cst.cam.ac.uk](mailto:service-desk@cst.cam.ac.uk) in case of any remaining problems.
  • Investigating
    Investigating
    Overnight a hardware fault took down the storage server that backs one of our main departmental VM pools (xene-pool1). All VMs running on that pool failed, which included the departmental database server, dbwebserver, Request Tracker, cl-student-ssh, part of the MSA service and the Windows Remote Desktop service.

May 2025

Legacy CUPS printing from Macs disrupted
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

    Nevertheless we suggest that you get DS-Print set up on your devices anyway, as this system will fully replace the legacy CUPS server soon.

  • Investigating
    Investigating

    We are investigating reports that printing to legacy printers from Macs is currently disrupted, possibly due to a Bonjour problem. We suggest using DS-Print as a workaround.

May 2025 to Jul 2025

Next