University of Cambridge Computer Laboratory - Notice history

gatwick crashed and rebooted again at around 06:22, again triggered by a routine configuration update.

We had, earlier in the night, attempted to install a software update but due to an unrelated issue, the routers refused to do an 'In Service Software Upgrade' - i.e. the upgrade would have caused more disruption - so we chose to roll back and delay this update until Cisco published their notes about this particular version.

The routers have remained stable since the crash, but we're going to do some further testing out-of-hours, and install a software update. There may be some further disruption whilst that happens.

A fix has been implemented for the misbehaving Caelum live state database (which had failed, filled the disk with error logs, and caused more problems).

We believe this issue has been resolved, and are in communication with the user who accidentally caused the poor performance. We're also considering hardware upgrades for the storage server.

UIS has declared the DDoS-related disruption to be resolved, though they note that there may be some residual issues this morning, including delays to email delivery.

We observe that some email was delayed for several hours overnight; Jisc's mitigation of the DDoS attack seems to have blocked some legitimate connections into and out of the University, particularly email-related ones.

Jisc have confirmed that the attacks are across multiple UK universities.

We observe that Jisc are implementing strategies to reduce the impact of the attacks, and the network is becoming more usable, though performance is likely to be impaired and some things may not work.

UIS has posted an update: "The University is currently experiencing a DDoS attack. CSIRT are liaising with JISC - we'll update when we know more."

We believe the disruption has ended; please report any ongoing issues with filer access to sys-admin.