mastodon.online - DB connection issues – Incident details

All systems operational

DB connection issues

Resolved
Major outage
Started 9 days agoLasted about 21 hours

Affected

Website & API

Major outage from 3:31 PM to 7:07 PM, Operational from 7:07 PM to 12:10 PM

Background queues

Major outage from 3:31 PM to 7:07 PM, Operational from 7:07 PM to 12:10 PM

Media storage

Operational from 3:31 PM to 12:10 PM

Streaming API

Operational from 3:31 PM to 12:10 PM

Updates
  • Resolved
    Resolved

    We have switched back to our original networking, and all functionality and performance is restored. Hetzner has confirmed that is was indeed a vSwitch issue that caused the network outage.

  • Monitoring
    Monitoring

    We have implemented a fix involving tailscale to route around the networking issue, and functionality appears to be restored. We will continue monitoring the situation.

  • Update
    Update

    We have learned that the backbone connection between our cloud servers and the baremetals that host our database is nonfunctional for any new cloud servers. We are currently working with Hetzner to resolve this issue.

  • Investigating
    Investigating

    We are currently experiencing an issue with our DB connection pooler connecting to the database. We are currently investigating.