mastodon.online - Kubernetes node outage – Incident details

System Under Maintenance

Kubernetes node outage

Resolved
Major outage
Started 4 months agoLasted 39 minutes

Affected

Website & API

Major outage from 2:21 PM to 2:42 PM, Degraded performance from 2:42 PM to 2:59 PM

Background queues

Major outage from 2:21 PM to 2:42 PM, Degraded performance from 2:42 PM to 2:59 PM

Media storage

Major outage from 2:21 PM to 2:42 PM, Degraded performance from 2:42 PM to 2:59 PM

Streaming API

Major outage from 2:21 PM to 2:42 PM, Degraded performance from 2:42 PM to 2:59 PM

Updates
  • Resolved
    Resolved

    Performance issues have been resolved.

  • Investigating
    Investigating

    Node have all been restored, but the underlying cause of the issue seems to still be present (i.e. massive resource utilization across all nodes). Still investigating.

  • Identified
    Identified

    The nodes hosting mastodon.online for kubernetes seem to be offline. We are currently working to replace them with new nodes as quickly as possible.