mastodon.online - Notice history

All systems operational

Website & API - Operational

100% - uptime
Jan 2025 · 100.0%Feb · 99.41%Mar · 100.0%

Previous page

Next page

Jan 2025
Feb 2025
Mar 2025

Background queues - Operational

100% - uptime
Jan 2025 · 100.0%Feb · 99.41%Mar · 100.0%

Previous page

Next page

Jan 2025
Feb 2025
Mar 2025

Media storage - Operational

100% - uptime
Jan 2025 · 100.0%Feb · 99.95%Mar · 100.0%

Previous page

Next page

Jan 2025
Feb 2025
Mar 2025

Streaming API - Operational

100% - uptime
Jan 2025 · 100.0%Feb · 99.95%Mar · 100.0%

Previous page

Next page

Jan 2025
Feb 2025
Mar 2025

Notice history

Mar 2025

No notices reported this month

Feb 2025

DB connection issues
  • Resolved
    Resolved

    We have switched back to our original networking, and all functionality and performance is restored. Hetzner has confirmed that is was indeed a vSwitch issue that caused the network outage.

  • Monitoring
    Monitoring

    We have implemented a fix involving tailscale to route around the networking issue, and functionality appears to be restored. We will continue monitoring the situation.

  • Update
    Update

    We have learned that the backbone connection between our cloud servers and the baremetals that host our database is nonfunctional for any new cloud servers. We are currently working with Hetzner to resolve this issue.

  • Investigating
    Investigating

    We are currently experiencing an issue with our DB connection pooler connecting to the database. We are currently investigating.

Kubernetes node outage
  • Resolved
    Resolved

    Performance issues have been resolved.

  • Investigating
    Investigating

    Node have all been restored, but the underlying cause of the issue seems to still be present (i.e. massive resource utilization across all nodes). Still investigating.

  • Identified
    Identified

    The nodes hosting mastodon.online for kubernetes seem to be offline. We are currently working to replace them with new nodes as quickly as possible.

Jan 2025

Timelines are not updating with some clients
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    The database software developer has been very responsive about this issue. They helped us troubleshoot it and provided us with a new version that solved the problem.

    We are currently monitoring the system closely, but everything seems to work fine now.

  • Identified
    Identified

    Last week, we switched our Redis database to another software. This led to user reports of timelines not updating anymore with using some clients, like Ivory or Pachli.

    After investigating the issue, we discovered that this new software returns invalid data in some specific cases, which are triggered by the API those Mastodon clients use.

    We have reached out to the software developers to troubleshoot this and are hopeful it will get fixed soon. In the meantime, we are also working on a way to roll back to our previous software in case we can not get this fixed quickly.

    We will post more details on this incident when possible.

Jan 2025 to Mar 2025

Next