University of Cambridge Computer Laboratory - Internet connectivity disrupted – Incident details

All systems operational

Internet connectivity disrupted

Resolved
Major outage
Started 2 months agoLasted 62 hours 44 minutes

Affected

External Services

Operational from 8:52 PM to 1:59 PM, Major outage from 1:59 PM to 2:26 PM, Partial outage from 2:26 PM to 9:24 PM, Operational from 9:24 PM to 11:36 AM

Network

Partial outage from 8:52 PM to 1:59 PM, Major outage from 1:59 PM to 2:26 PM, Partial outage from 2:26 PM to 9:24 PM, Operational from 9:24 PM to 11:36 AM

Updates
  • Resolved
    UTC
    Resolved

    Update from UIS:

    We’ve moved the service to the standby firewall, and it is currently stable with no further issues observed.                                                       

    We’re continuing to investigate the underlying cause with Fortinet and will provide a further update once we have more information.

    We will close this incident now as it is no longer affecting us, but may add a further update here if we receive one.

  • Monitoring
    UTC
    Monitoring

    We observe that internet connectivity appears to be working again at the moment, though we have had no information yet from UIS so cannot guarantee that this will remain the case. The advice to work from home tomorrow still stands for now.

  • Update
    UTC
    Update

    The Departmental Silver Team has decided to recommend that you work from home tomorrow if you can.

  • Update
    UTC
    Update

    Partial connectivity (about 40%) has been restored as of 15:26.

    This remains a University-wide problem but we have had no information from UIS yet.

    If connecting from the internet into our network, only about 40% of our servers will be reachable. If you are unable to connect directly to the system you need, try going via CL VPN1, CL VPN2 or the UIS VPN. If connecting via SSH, you could try connecting via slogin or some other departmental server such as ely.

    Internet connectivity in William Gates Building offices, as well as the rest of the University including eduroam, other departments and colleges, is likely to be barely usable.

    Our main departmental website www.cst.cam.ac.uk remains accessible from the internet as it is hosted outside the University; however pages on www.cl.cam.ac.uk are affected.

  • Update
    UTC
    Update

    We observe that the University is now completely offline. This situation is beyond our control and we await UIS to repair the fault.

  • Identified
    UTC
    Identified

    We are aware that certain hosts within our network are unable to reach certain hosts on the internet.

    (Specifically, any given IP address on our network is unable to reach a different seemingly-random 15% of internet IP addresses.)

    We believe that this may be due to a malfunction or misconfiguration of the University firewall and have contacted the team in UIS responsible for this.