University of Cambridge Computer Laboratory - Caelum Console, vpnpassword and sshkeys.cl unavailable – Incident details

All systems operational

Caelum Console, vpnpassword and sshkeys.cl unavailable

Resolved
Major outage
Started over 1 year agoLasted about 2 hours

Affected

Internal Services

Major outage from 11:25 AM to 11:55 AM, Partial outage from 11:25 AM to 11:55 AM, Operational from 11:55 AM to 12:59 PM

Caelum Console (server management)

Major outage from 11:25 AM to 11:55 AM, Operational from 11:55 AM to 12:59 PM

Other Internal Services

Partial outage from 11:25 AM to 11:55 AM, Operational from 11:55 AM to 12:59 PM

Updates
  • Resolved
    UTC
    Resolved

    We have updated a component (uwsgi) of the web front end, which we hope will help with the deadlock problem.

  • Monitoring
    UTC
    Monitoring

    The server has been rebooted (and one bug fixed in the SSH key management backend) to restore service.

    We are continuing to investigate a recurring problem that has caused the web frontends to stop responding a few times now.

  • Investigating
    UTC
    Investigating

    The Caelum Console application for managing servers in datacentres, along with the VPN2 password application and SSH key management application, are currently unavailable. We are investigating.