mastodon.online - DB connection issues – Incident details

DB connection issues

ResolvedMajor outage20 hours 38 minutes

Affected

Website & API

Major outage from 3:31 PM to 7:07 PM, Operational from 7:07 PM to 12:10 PM

Background queues

Major outage from 3:31 PM to 7:07 PM, Operational from 7:07 PM to 12:10 PM

Media storage

Operational from 3:31 PM to 12:10 PM

Streaming API

Operational from 3:31 PM to 12:10 PM

  • ResolvedUTC

    We have switched back to our original networking, and all functionality and performance is restored. Hetzner has confirmed that is was indeed a vSwitch issue that caused the network outage.

  • MonitoringUTC

    We have implemented a fix involving tailscale to route around the networking issue, and functionality appears to be restored. We will continue monitoring the situation.

  • UpdateUTC

    We have learned that the backbone connection between our cloud servers and the baremetals that host our database is nonfunctional for any new cloud servers. We are currently working with Hetzner to resolve this issue.

  • InvestigatingUTC

    We are currently experiencing an issue with our DB connection pooler connecting to the database. We are currently investigating.