University of Cambridge Computer Laboratory - Notice history

We have implemented a workaround and have brought archive-smb back into service, with reduced resilience pending replacement of a failed system SSD.

We observe that both power feeds in WCDC have now been restored. However as we have had no information from UIS about this incident, we do not yet know whether power can be considered stable.

The archive-smb outage is ongoing and tracked in a separate incident. We believe that all other departmental systems are working again.

This incident has been resolved. GN09 is fully operational. Most servers that were previously running have been restarted.

If you have a physical server that is not running, you may be able to start it yourself via https://console.caelum.cl.cam.ac.uk as usual, or contact service-desk@cst.cam.ac.uk.

VMs that were not set to start automatically have not been restarted. You can start VMs when you need them via https://xo.cl.cam.ac.uk as usual.

Contact service-desk@cst.cam.ac.uk if there are any remaining issues.

Cooling has been restored and is expected to remain stable. The cause of the chiller shutting down was the chilled water circulation pumps stopping for some other reason, which will be investigated next week but which we expect to have been an isolated incident. The chiller still has one alarm present which is not preventing operation but is still being investigated.

We are taking the opportunity of GN09 being shut down to perform some routine firmware and software updates on network hardware and storage systems, so we will not start turning servers back on quite yet, but expect to be able to do so shortly.

Progress has been made; the chiller is running again but there is a problem still under investigation. We are hopeful that servers can be turned back on again today, but will await the all-clear from the chiller technician.

Most servers in GN09 are now off, and must remain off until further notice. The emergency technician has arrived and is investigating.

We believe that Mimecast has unblocked us.

There are some unrelated issues with some mailing lists still under investigation, not connected in any way (as far as we know) with the Mimecast problem; if you experience any more problems please contact service-desk@cst.cam.ac.uk.

We believe that UIS has successfully worked around this issue, and email sent to mailing lists from departmental addresses should now work.

However, we now also believe that this was a symptom of a broader problem with email to one particular email anti-spam service provider, Mimecast. Email to other institutions which also use Mimecast may also be affected. We are working on getting this resolved.

If you do encounter the issue, you may be able to get email through successfully by sending from an @cam.ac.uk address.

As a workaround, messages should get through if you send mail using Outlook from your @cam.ac.uk address to the relevant internal @lists.cam.ac.uk address for the mailing list; if you don't know what that address is for a particular list, contact service-desk@cst.cam.ac.uk.