Duration: 1 hour and 14 minutes

Affected Components: Background queues, Website & API

May 12, 08:56:56 GMT+0
Identified - Hetzner is currently experiencing an outage due to planned maintenance that is affecting our infrastructure. We are attempting a workaround, and will be back up as soon as possible..

May 12, 09:23:19 GMT+0
Monitoring - We've implemented a workaround, and the site is functional again. We will continue to monitor the situation during Hetzner's maintenance work..

May 12, 10:11:03 GMT+0
Resolved - Hetzner has concluded their maintenance..

Duration: 15 minutes

Affected Components: Background queues, Website & API

Jan 29, 09:45:40 GMT+0
Investigating - The site seems to be experiencing connection issues with related services such as the database. We are currently investigating the issue..

Jan 29, 09:53:58 GMT+0
Monitoring - Site connectivity has been restored. We are still working to identify the root cause of the outage..

Jan 29, 10:00:47 GMT+0
Resolved - The connectivity issues seem to have been caused by a database migration job causing a lock. All functionality has been restored..

Duration: 33 minutes

Affected Components: Background queues, Website & API, Streaming API

Oct 17, 08:31:01 GMT+0
Investigating - We are experiencing a DB outage and are currently investigating..

Oct 17, 08:37:54 GMT+0
Monitoring - DB connectivity has been fixed. The read replica had become unresponsive. We are currently monitoring the situation to make sure everything remains stable..

Oct 17, 09:04:15 GMT+0
Resolved - This incident has been resolved..

Duration: 17 minutes

Affected Components: Website & API, Background queues

May 28, 06:30:01 GMT+0
Identified - Maintenance is now in progress.

May 28, 06:46:31 GMT+0
Completed - Maintenance has completed successfully..

May 28, 06:30:00 GMT+0
Identified - We will be updating the database to the next minor version, as well as some OS components that will require a restart. Minor downtime of roughly a minute is expected..

Duration: 20 hours and 38 minutes

Affected Components: Media storage, Background queues, Website & API, Streaming API

Feb 12, 15:31:42 GMT+0
Investigating - We are currently experiencing an issue with our DB connection pooler connecting to the database. We are currently investigating..

Feb 12, 16:43:48 GMT+0
Investigating - We have learned that the backbone connection between our cloud servers and the baremetals that host our database is nonfunctional for any new cloud servers. We are currently working with Hetzner to resolve this issue..

Feb 12, 19:07:52 GMT+0
Monitoring - We have implemented a fix involving tailscale to route around the networking issue, and functionality appears to be restored. We will continue monitoring the situation..

Feb 13, 12:10:11 GMT+0
Resolved - We have switched back to our original networking, and all functionality and performance is restored. Hetzner has confirmed that is was indeed a vSwitch issue that caused the network outage..

Duration: 39 minutes

Affected Components: Media storage, Background queues, Website & API, Streaming API

Feb 11, 14:21:27 GMT+0
Identified - The nodes hosting [mastodon.online](http://mastodon.online) for kubernetes seem to be offline. We are currently working to replace them with new nodes as quickly as possible..

Feb 11, 14:42:38 GMT+0
Investigating - Node have all been restored, but the underlying cause of the issue seems to still be present (i.e. massive resource utilization across all nodes). Still investigating..

Feb 11, 14:59:58 GMT+0
Resolved - Performance issues have been resolved..

Duration: 7 days, 23 hours and 10 minutes

Affected Components: Website & API

Feb 5, 14:39:53 GMT+0
Resolved - This incident has been resolved..

Jan 28, 15:30:00 GMT+0
Identified - Last week, we switched our Redis database to another software. This led to user reports of timelines not updating anymore with using some clients, like Ivory or Pachli. After investigating the issue, we discovered that this new software returns invalid data in some specific cases, which are triggered by the API those Mastodon clients use. We have reached out to the software developers to troubleshoot this and are hopeful it will get fixed soon. In the meantime, we are also working on a way to roll back to our previous software in case we can not get this fixed quickly. We will post more details on this incident when possible..

Feb 4, 17:06:33 GMT+0
Monitoring - The database software developer has been very responsive about this issue. They helped us troubleshoot it and provided us with a new version that solved the problem. We are currently monitoring the system closely, but everything seems to work fine now..

Duration: 11 hours and 47 minutes

Affected Components: Media storage, Background queues, Website & API, Streaming API

Nov 28, 21:22:09 GMT+0
Investigating - All application pods were briefly offline, and are currently struggling to come back. We are investigating the issue..

Nov 28, 21:31:29 GMT+0
Monitoring - All application components have been restarted and resumed functionality. Current assumption is an outage in Hetzner's Helsinki region. We will continue monitoring the cluster and Hetzner for updates..

Nov 29, 09:09:21 GMT+0
Resolved - Outage was likely the result of maintenance work being done on Hetzner networking. Considering this resolved..

Duration: 3 hours and 8 minutes

Affected Components: Background queues

Sep 16, 06:55:16 GMT+0
Investigating - We are currently unable to send emails to users due to issues with Postmark. We are currently investigating the issue..

Sep 16, 10:02:46 GMT+0
Resolved - Issue was identified as an SSL error on Postmark's side. This has now been resolved, and emails are working again..

Duration: 18 minutes

Affected Components: Background queues, Website & API, Streaming API

Aug 19, 16:06:22 GMT+0
Identified - Due to a hardware failure with our provider, [mastodon.online](http://mastodon.online) is not available. We are working on the issue..

Aug 19, 16:24:16 GMT+0
Resolved - This incident has been resolved, all services are now operational..

Duration: 2 days, 2 hours and 56 minutes

Affected Components: Media storage

Feb 14, 18:18:58 GMT+0
Identified - Our storage provider is having issues in the region we use to store media files on [mastodon.online](http://mastodon.online). They are working to resolve this, in the meantime some media files might not load correctly, or new posts can fail to be posted..

Feb 14, 21:18:12 GMT+0
Identified - Our provider is still working on restoring access to the media (images and videos) files storage. They estimate the recovery will last until at least 06:00 UTC tomorrow. Until this is completed, newly posts from the federation will not display media, and it is not possible to write new posts containing images or videos..

Feb 15, 07:57:33 GMT+0
Identified - Our media storage service is back online. Old statuses may still be missing images, we are working on fetching them again..

Feb 15, 16:32:36 GMT+0
Identified - We finished ingesting media file for every status from a remote instance that was missing data due to our provider's outage. Unfortunately, they lost some stored files and any media (image or video) attached to statuses written by [mastodon.online](http://mastodon.online) users between 2024-02-13 23:58 UTC and 2024-02-14 16:11 UTC have been lost, and we do not have a way to recover them. Some remote user profile pictures and banners might still be outdated, but they should be refreshed in the coming hours..

Feb 16, 21:15:28 GMT+0
Resolved - This incident has been resolved. We re-fetched external content (media files, profile pictures and banners) that were missing due to the incident. Unfortunately, they lost some stored files and any media (image or video) attached to statuses written by [mastodon.online](http://mastodon.online) users between 2024-02-13 23:58 UTC and 2024-02-14 16:11 UTC have been lost, and we do not have a way to recover them. We will work with our provider to understand better where their failure came from and ensure they are taking the needed steps to avoid this to happen again..

Duration: 30 minutes

Affected Components: Background queues, Website & API

Aug 7, 10:30:01 GMT+0
Identified - Maintenance is now in progress.

Aug 7, 11:00:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 7, 10:30:00 GMT+0
Identified - We will be upgrading PosgreSQL for mastodon.online. The upgrade should be a simple switchover and only result in a few minutes of downtime at most, hopefully less..

Duration: 1 hour and 3 minutes

Affected Components: Background queues

Jul 20, 19:05:37 GMT+0
Identified - We noticed a problem with our email deliveries (new signups, lost password…). We are working on it, it should be fixed in the next hours..

Jul 20, 20:08:09 GMT+0
Resolved - This incident has been resolved. E-mails are not correctly delivered. If you created an account in the last two days and did not receive an email, it should be delivered in the coming hours..

Duration: 1 day, 1 hour and 33 minutes

Affected Components: Background queues, Website & API

Jul 11, 14:34:07 GMT+0
Resolved - This incident has been resolved..

Jul 10, 13:01:32 GMT+0
Investigating - There is currently an issue with Hetzner's network connection between Hetzner Cloud and Bare Metal, which is affecting our backend services. We are looking into the cause..

Jul 10, 14:10:51 GMT+0
Monitoring - We have moved our Redis server into Hetzner Cloud, where there are no network connectivity issues, and functionality has been restored. We will continue monitoring for any issues..

Duration: 1 hour and 26 minutes

Affected Components: Background queues, Website & API, Streaming API

Jun 27, 08:51:15 GMT+0
Identified - There is currently a network incident with our hosting provider, preventing our Redis server to be unreachable. We are waiting for an ETA from our provider, and are preparing setting up a new Redis server if the issue is not resolved quick enough..

Jun 27, 09:27:07 GMT+0
Monitoring - Our provider seems to have fixed the network issue and the services are back online. Your feeds might be a few minutes late while we are processing the messages posted during the downtime. We are closely monitoring the situation until our provider confirms everything is operational on their side..

Jun 27, 10:03:01 GMT+0
Identified - Network to the Redis server is down again, we are waiting for an update from our hosting provider..

Jun 27, 10:17:13 GMT+0
Resolved - This incident has been resolved and our provider confirmed the issue has been fixed on their side. Everything is back online and all messages in the backlog have been processed..

Affected Components: Background queues

Jun 6, 18:58:44 GMT+0
Resolved - There was a small error in the DB migration steps when upgrading to the latest nightly. This was due to an older DB version being used in production. We have rolled back to a previous version and are continuing to monitor..

Duration: 1 hour and 36 minutes

Affected Components: Website & API

May 5, 21:04:21 GMT+0
Investigating - We are currently investigating this incident..

May 5, 21:46:12 GMT+0
Identified - We identified the issue and are working on it..

May 5, 22:40:36 GMT+0
Resolved - This incident has been resolved. An internal TLS certificate was renewed automatically, but did not get properly reloaded into our load balancer after renewal. It took us a long time to identify this, as no errors were visible to point us to the source of the issue. We identified the root cause and will take steps to prevent this from happening in the future..

Duration: 23 minutes

Affected Components: Website & API, Streaming API

Apr 27, 04:22:07 GMT+0
Investigating - We are currently investigating this incident..

Apr 27, 04:42:50 GMT+0
Identified - Problem was misconfiguration in the Helm chart. We're working on a fix..

Apr 27, 04:44:45 GMT+0
Resolved - This incident has been resolved..

Affected Components: Media storage

Mar 16, 22:50:01 GMT+0
Resolved - Early morning Feb 24 we were indirectly made aware of a misconfiguration on our object storage domain (files.mastodon.online) that allowed anyone to see the list of all uploaded files. Within 30 minutes this mistake was corrected. However, we have reasons to believe that the issue has existed since Feb 2, when we began upgrading our infrastructure. Normally Mastodon relies on long, randomly generated file names with high entropy to ensure that certain files are accessed only by those who know the link. However, that misconfiguration allowed that measure to be bypassed. Most files in our object storage are public in nature–profile pictures, custom emojis, images and videos attached to public posts. But there is a type of file that should never be accessed by anyone but its owner, and it’s the user’s archive takeout. When a user requests an archive takeout of their account, all of their posts, favorites and bookmarks are put into an archive which remains accessible for 7 days after. At the time of the incident, 1,037 such user archive takeouts were available. We have immediately deleted them to prevent anyone from downloading them. We have notified the affected users and we are changing the Mastodon software to not rely on high entropy links for access control to user archive takeouts any longer, as well as adding an automated check into the admin dashboard to detect similar misconfigurations and notify other server operators about them. Security is important to us and we are continuously improving our processes as we scale our organization from one employee to multiple to ensure that mistakes like this do not happen in the future..

Duration: 2 hours and 39 minutes

Affected Components: Website & API, Background queues, Streaming API

Mar 13, 22:40:58 GMT+0
Identified - There is an issue in our Kubernetes cluster that we are working to resolve..

Mar 14, 01:19:53 GMT+0
Resolved - This incident has been resolved..

Duration: 11 hours and 19 minutes

Affected Components: Media storage

Mar 1, 08:14:19 GMT+0
Resolved - This was caused by a network maintenance from our hosting provider. The maintenance has ended and our response times and error rates are back to normal..

Feb 28, 20:55:00 GMT+0
Identified - There are intermittent failures with uploading to object storage, which results in "Not available" previews..