mastodon.online Status - Incident history https://status.mastodon.online mastodon.online Fri, 25 Sep 2026 07:45:12 +0000 Website and API are down Type: Incident Duration: 23 minutes Affected Components: Website & API Sep 25, 07:45:12 GMT+0 - Investigating - We are currently investigating this incident. Sep 25, 08:08:23 GMT+0 - Resolved - This was caused by a migration, which has been rolled back. All services are back online. Type: Incident

Duration: 23 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 25</var>, <var data-var='time'>07:45:12</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Sep <var data-var='date'> 25</var>, <var data-var='time'>08:08:23</var> GMT+0</small><br><strong>Resolved</strong> - This was caused by a migration, which has been rolled back. All services are back online..</p> ]]> Fri, 25 Sep 2026 07:45:12 +0000 https://status.mastodon.online/incident/cmugnohex00511ao45n6yc01p https://status.mastodon.online/incident/cmugnohex00511ao45n6yc01p Hetzner Outage Type: Incident Duration: 1 hour and 14 minutes Affected Components: Background queues, Website & API May 12, 08:56:56 GMT+0 - Identified - Hetzner is currently experiencing an outage due to planned maintenance that is affecting our infrastructure. We are attempting a workaround, and will be back up as soon as possible. May 12, 09:23:19 GMT+0 - Monitoring - We've implemented a workaround, and the site is functional again. We will continue to monitor the situation during Hetzner's maintenance work. May 12, 10:11:03 GMT+0 - Resolved - Hetzner has concluded their maintenance. Type: Incident

Duration: 1 hour and 14 minutes

Affected Components: ,

<p><small>May <var data-var='date'> 12</var>, <var data-var='time'>08:56:56</var> GMT+0</small><br><strong>Identified</strong> - Hetzner is currently experiencing an outage due to planned maintenance that is affecting our infrastructure. We are attempting a workaround, and will be back up as soon as possible..</p> <p><small>May <var data-var='date'> 12</var>, <var data-var='time'>09:23:19</var> GMT+0</small><br><strong>Monitoring</strong> - We've implemented a workaround, and the site is functional again. We will continue to monitor the situation during Hetzner's maintenance work..</p> <p><small>May <var data-var='date'> 12</var>, <var data-var='time'>10:11:03</var> GMT+0</small><br><strong>Resolved</strong> - Hetzner has concluded their maintenance..</p> ]]> Tue, 12 May 2026 08:56:56 +0000 https://status.mastodon.online/incident/cmp2ecvnl0eukugtt809ksbzm https://status.mastodon.online/incident/cmp2ecvnl0eukugtt809ksbzm Site connectivity issues Type: Incident Duration: 15 minutes Affected Components: Background queues, Website & API Jan 29, 09:45:40 GMT+0 - Investigating - The site seems to be experiencing connection issues with related services such as the database. We are currently investigating the issue. Jan 29, 09:53:58 GMT+0 - Monitoring - Site connectivity has been restored. We are still working to identify the root cause of the outage. Jan 29, 10:00:47 GMT+0 - Resolved - The connectivity issues seem to have been caused by a database migration job causing a lock. All functionality has been restored. Type: Incident

Duration: 15 minutes

Affected Components: ,

<p><small>Jan <var data-var='date'> 29</var>, <var data-var='time'>09:45:40</var> GMT+0</small><br><strong>Investigating</strong> - The site seems to be experiencing connection issues with related services such as the database. We are currently investigating the issue..</p> <p><small>Jan <var data-var='date'> 29</var>, <var data-var='time'>09:53:58</var> GMT+0</small><br><strong>Monitoring</strong> - Site connectivity has been restored. We are still working to identify the root cause of the outage..</p> <p><small>Jan <var data-var='date'> 29</var>, <var data-var='time'>10:00:47</var> GMT+0</small><br><strong>Resolved</strong> - The connectivity issues seem to have been caused by a database migration job causing a lock. All functionality has been restored..</p> ]]> Thu, 29 Jan 2026 09:45:40 +0000 https://status.mastodon.online/incident/cmkz9qtbo0y08ytjoxnxehe0o https://status.mastodon.online/incident/cmkz9qtbo0y08ytjoxnxehe0o DB availability issues Type: Incident Duration: 33 minutes Affected Components: Background queues, Website & API, Streaming API Oct 17, 08:31:01 GMT+0 - Investigating - We are experiencing a DB outage and are currently investigating. Oct 17, 08:37:54 GMT+0 - Monitoring - DB connectivity has been fixed. The read replica had become unresponsive. We are currently monitoring the situation to make sure everything remains stable. Oct 17, 09:04:15 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 33 minutes

Affected Components: , ,

<p><small>Oct <var data-var='date'> 17</var>, <var data-var='time'>08:31:01</var> GMT+0</small><br><strong>Investigating</strong> - We are experiencing a DB outage and are currently investigating..</p> <p><small>Oct <var data-var='date'> 17</var>, <var data-var='time'>08:37:54</var> GMT+0</small><br><strong>Monitoring</strong> - DB connectivity has been fixed. The read replica had become unresponsive. We are currently monitoring the situation to make sure everything remains stable..</p> <p><small>Oct <var data-var='date'> 17</var>, <var data-var='time'>09:04:15</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 17 Oct 2025 08:31:01 +0000 https://status.mastodon.online/incident/cmgula8cw01xz116srkdmuwsw https://status.mastodon.online/incident/cmgula8cw01xz116srkdmuwsw Database minor upgrade Type: Maintenance Duration: 17 minutes Affected Components: Website & API, Background queues May 28, 06:30:01 GMT+0 - Identified - Maintenance is now in progress May 28, 06:46:31 GMT+0 - Completed - Maintenance has completed successfully. May 28, 06:30:00 GMT+0 - Identified - We will be updating the database to the next minor version, as well as some OS components that will require a restart. Minor downtime of roughly a minute is expected. Type: Maintenance

Duration: 17 minutes

Affected Components: ,

<p><small>May <var data-var='date'> 28</var>, <var data-var='time'>06:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>May <var data-var='date'> 28</var>, <var data-var='time'>06:46:31</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>May <var data-var='date'> 28</var>, <var data-var='time'>06:30:00</var> GMT+0</small><br><strong>Identified</strong> - We will be updating the database to the next minor version, as well as some OS components that will require a restart. Minor downtime of roughly a minute is expected..</p> ]]> Wed, 28 May 2025 06:30:00 +0000 https://status.mastodon.online/maintenance/cmb6pcayy000gbiaclmfwbq14 https://status.mastodon.online/maintenance/cmb6pcayy000gbiaclmfwbq14 DB connection issues Type: Incident Duration: 20 hours and 38 minutes Affected Components: Media storage, Background queues, Website & API, Streaming API Feb 12, 15:31:42 GMT+0 - Investigating - We are currently experiencing an issue with our DB connection pooler connecting to the database. We are currently investigating. Feb 12, 16:43:48 GMT+0 - Investigating - We have learned that the backbone connection between our cloud servers and the baremetals that host our database is nonfunctional for any new cloud servers. We are currently working with Hetzner to resolve this issue. Feb 12, 19:07:52 GMT+0 - Monitoring - We have implemented a fix involving tailscale to route around the networking issue, and functionality appears to be restored. We will continue monitoring the situation. Feb 13, 12:10:11 GMT+0 - Resolved - We have switched back to our original networking, and all functionality and performance is restored. Hetzner has confirmed that is was indeed a vSwitch issue that caused the network outage. Type: Incident

Duration: 20 hours and 38 minutes

Affected Components: , , ,

<p><small>Feb <var data-var='date'> 12</var>, <var data-var='time'>15:31:42</var> GMT+0</small><br><strong>Investigating</strong> - We are currently experiencing an issue with our DB connection pooler connecting to the database. We are currently investigating..</p> <p><small>Feb <var data-var='date'> 12</var>, <var data-var='time'>16:43:48</var> GMT+0</small><br><strong>Investigating</strong> - We have learned that the backbone connection between our cloud servers and the baremetals that host our database is nonfunctional for any new cloud servers. We are currently working with Hetzner to resolve this issue..</p> <p><small>Feb <var data-var='date'> 12</var>, <var data-var='time'>19:07:52</var> GMT+0</small><br><strong>Monitoring</strong> - We have implemented a fix involving tailscale to route around the networking issue, and functionality appears to be restored. We will continue monitoring the situation..</p> <p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>12:10:11</var> GMT+0</small><br><strong>Resolved</strong> - We have switched back to our original networking, and all functionality and performance is restored. Hetzner has confirmed that is was indeed a vSwitch issue that caused the network outage..</p> ]]> Wed, 12 Feb 2025 15:31:42 +0000 https://status.mastodon.online/incident/cm722jt5p005y134nbp7jwfm2 https://status.mastodon.online/incident/cm722jt5p005y134nbp7jwfm2 Kubernetes node outage Type: Incident Duration: 39 minutes Affected Components: Media storage, Background queues, Website & API, Streaming API Feb 11, 14:21:27 GMT+0 - Identified - The nodes hosting [mastodon.online](http://mastodon.online) for kubernetes seem to be offline. We are currently working to replace them with new nodes as quickly as possible. Feb 11, 14:42:38 GMT+0 - Investigating - Node have all been restored, but the underlying cause of the issue seems to still be present (i.e. massive resource utilization across all nodes). Still investigating. Feb 11, 14:59:58 GMT+0 - Resolved - Performance issues have been resolved. Type: Incident

Duration: 39 minutes

Affected Components: , , ,

<p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>14:21:27</var> GMT+0</small><br><strong>Identified</strong> - The nodes hosting [mastodon.online](http://mastodon.online) for kubernetes seem to be offline. We are currently working to replace them with new nodes as quickly as possible..</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>14:42:38</var> GMT+0</small><br><strong>Investigating</strong> - Node have all been restored, but the underlying cause of the issue seems to still be present (i.e. massive resource utilization across all nodes). Still investigating..</p> <p><small>Feb <var data-var='date'> 11</var>, <var data-var='time'>14:59:58</var> GMT+0</small><br><strong>Resolved</strong> - Performance issues have been resolved..</p> ]]> Tue, 11 Feb 2025 14:21:27 +0000 https://status.mastodon.online/incident/cm70kln2t005g11wh6yeaxd4g https://status.mastodon.online/incident/cm70kln2t005g11wh6yeaxd4g Timelines are not updating with some clients Type: Incident Duration: 7 days, 23 hours and 10 minutes Affected Components: Website & API Feb 5, 14:39:53 GMT+0 - Resolved - This incident has been resolved. Jan 28, 15:30:00 GMT+0 - Identified - Last week, we switched our Redis database to another software. This led to user reports of timelines not updating anymore with using some clients, like Ivory or Pachli. After investigating the issue, we discovered that this new software returns invalid data in some specific cases, which are triggered by the API those Mastodon clients use. We have reached out to the software developers to troubleshoot this and are hopeful it will get fixed soon. In the meantime, we are also working on a way to roll back to our previous software in case we can not get this fixed quickly. We will post more details on this incident when possible. Feb 4, 17:06:33 GMT+0 - Monitoring - The database software developer has been very responsive about this issue. They helped us troubleshoot it and provided us with a new version that solved the problem. We are currently monitoring the system closely, but everything seems to work fine now. Type: Incident

Duration: 7 days, 23 hours and 10 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>14:39:53</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jan <var data-var='date'> 28</var>, <var data-var='time'>15:30:00</var> GMT+0</small><br><strong>Identified</strong> - Last week, we switched our Redis database to another software. This led to user reports of timelines not updating anymore with using some clients, like Ivory or Pachli. After investigating the issue, we discovered that this new software returns invalid data in some specific cases, which are triggered by the API those Mastodon clients use. We have reached out to the software developers to troubleshoot this and are hopeful it will get fixed soon. In the meantime, we are also working on a way to roll back to our previous software in case we can not get this fixed quickly. We will post more details on this incident when possible..</p> <p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>17:06:33</var> GMT+0</small><br><strong>Monitoring</strong> - The database software developer has been very responsive about this issue. They helped us troubleshoot it and provided us with a new version that solved the problem. We are currently monitoring the system closely, but everything seems to work fine now..</p> ]]> Tue, 28 Jan 2025 15:30:00 +0000 https://status.mastodon.online/incident/cm6pdomff002d14lxsrpb5bmy https://status.mastodon.online/incident/cm6pdomff002d14lxsrpb5bmy Site outage Type: Incident Duration: 11 hours and 47 minutes Affected Components: Media storage, Background queues, Website & API, Streaming API Nov 28, 21:22:09 GMT+0 - Investigating - All application pods were briefly offline, and are currently struggling to come back. We are investigating the issue. Nov 28, 21:31:29 GMT+0 - Monitoring - All application components have been restarted and resumed functionality. Current assumption is an outage in Hetzner's Helsinki region. We will continue monitoring the cluster and Hetzner for updates. Nov 29, 09:09:21 GMT+0 - Resolved - Outage was likely the result of maintenance work being done on Hetzner networking. Considering this resolved. Type: Incident

Duration: 11 hours and 47 minutes

Affected Components: , , ,

<p><small>Nov <var data-var='date'> 28</var>, <var data-var='time'>21:22:09</var> GMT+0</small><br><strong>Investigating</strong> - All application pods were briefly offline, and are currently struggling to come back. We are investigating the issue..</p> <p><small>Nov <var data-var='date'> 28</var>, <var data-var='time'>21:31:29</var> GMT+0</small><br><strong>Monitoring</strong> - All application components have been restarted and resumed functionality. Current assumption is an outage in Hetzner's Helsinki region. We will continue monitoring the cluster and Hetzner for updates..</p> <p><small>Nov <var data-var='date'> 29</var>, <var data-var='time'>09:09:21</var> GMT+0</small><br><strong>Resolved</strong> - Outage was likely the result of maintenance work being done on Hetzner networking. Considering this resolved..</p> ]]> Thu, 28 Nov 2024 21:22:09 +0000 https://status.mastodon.online/incident/cm41tlrd2006610rnsk01s7hc https://status.mastodon.online/incident/cm41tlrd2006610rnsk01s7hc Email sending unavailable Type: Incident Duration: 3 hours and 8 minutes Affected Components: Background queues Sep 16, 06:55:16 GMT+0 - Investigating - We are currently unable to send emails to users due to issues with Postmark. We are currently investigating the issue. Sep 16, 10:02:46 GMT+0 - Resolved - Issue was identified as an SSL error on Postmark's side. This has now been resolved, and emails are working again. Type: Incident

Duration: 3 hours and 8 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>06:55:16</var> GMT+0</small><br><strong>Investigating</strong> - We are currently unable to send emails to users due to issues with Postmark. We are currently investigating the issue..</p> <p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>10:02:46</var> GMT+0</small><br><strong>Resolved</strong> - Issue was identified as an SSL error on Postmark's side. This has now been resolved, and emails are working again..</p> ]]> Mon, 16 Sep 2024 06:55:16 +0000 https://status.mastodon.online/incident/cm14nhr80000po54npj3hsm70 https://status.mastodon.online/incident/cm14nhr80000po54npj3hsm70 Service is down Type: Incident Duration: 18 minutes Affected Components: Background queues, Website & API, Streaming API Aug 19, 16:06:22 GMT+0 - Identified - Due to a hardware failure with our provider, [mastodon.online](http://mastodon.online) is not available. We are working on the issue. Aug 19, 16:24:16 GMT+0 - Resolved - This incident has been resolved, all services are now operational. Type: Incident

Duration: 18 minutes

Affected Components: , ,

<p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>16:06:22</var> GMT+0</small><br><strong>Identified</strong> - Due to a hardware failure with our provider, [mastodon.online](http://mastodon.online) is not available. We are working on the issue..</p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>16:24:16</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved, all services are now operational..</p> ]]> Mon, 19 Aug 2024 16:06:22 +0000 https://status.mastodon.online/incident/cm016um9d002jfg4r8aevqs48 https://status.mastodon.online/incident/cm016um9d002jfg4r8aevqs48 Media storage is unavailable Type: Incident Duration: 2 days, 2 hours and 56 minutes Affected Components: Media storage Feb 14, 18:18:58 GMT+0 - Identified - Our storage provider is having issues in the region we use to store media files on [mastodon.online](http://mastodon.online). They are working to resolve this, in the meantime some media files might not load correctly, or new posts can fail to be posted. Feb 14, 21:18:12 GMT+0 - Identified - Our provider is still working on restoring access to the media (images and videos) files storage. They estimate the recovery will last until at least 06:00 UTC tomorrow. Until this is completed, newly posts from the federation will not display media, and it is not possible to write new posts containing images or videos. Feb 15, 07:57:33 GMT+0 - Identified - Our media storage service is back online. Old statuses may still be missing images, we are working on fetching them again. Feb 15, 16:32:36 GMT+0 - Identified - We finished ingesting media file for every status from a remote instance that was missing data due to our provider's outage. Unfortunately, they lost some stored files and any media (image or video) attached to statuses written by [mastodon.online](http://mastodon.online) users between 2024-02-13 23:58 UTC and 2024-02-14 16:11 UTC have been lost, and we do not have a way to recover them. Some remote user profile pictures and banners might still be outdated, but they should be refreshed in the coming hours. Feb 16, 21:15:28 GMT+0 - Resolved - This incident has been resolved. We re-fetched external content (media files, profile pictures and banners) that were missing due to the incident. Unfortunately, they lost some stored files and any media (image or video) attached to statuses written by [mastodon.online](http://mastodon.online) users between 2024-02-13 23:58 UTC and 2024-02-14 16:11 UTC have been lost, and we do not have a way to recover them. We will work with our provider to understand better where their failure came from and ensure they are taking the needed steps to avoid this to happen again. Type: Incident

Duration: 2 days, 2 hours and 56 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 14</var>, <var data-var='time'>18:18:58</var> GMT+0</small><br><strong>Identified</strong> - Our storage provider is having issues in the region we use to store media files on [mastodon.online](http://mastodon.online). They are working to resolve this, in the meantime some media files might not load correctly, or new posts can fail to be posted..</p> <p><small>Feb <var data-var='date'> 14</var>, <var data-var='time'>21:18:12</var> GMT+0</small><br><strong>Identified</strong> - Our provider is still working on restoring access to the media (images and videos) files storage. They estimate the recovery will last until at least 06:00 UTC tomorrow. Until this is completed, newly posts from the federation will not display media, and it is not possible to write new posts containing images or videos..</p> <p><small>Feb <var data-var='date'> 15</var>, <var data-var='time'>07:57:33</var> GMT+0</small><br><strong>Identified</strong> - Our media storage service is back online. Old statuses may still be missing images, we are working on fetching them again..</p> <p><small>Feb <var data-var='date'> 15</var>, <var data-var='time'>16:32:36</var> GMT+0</small><br><strong>Identified</strong> - We finished ingesting media file for every status from a remote instance that was missing data due to our provider's outage. Unfortunately, they lost some stored files and any media (image or video) attached to statuses written by [mastodon.online](http://mastodon.online) users between 2024-02-13 23:58 UTC and 2024-02-14 16:11 UTC have been lost, and we do not have a way to recover them. Some remote user profile pictures and banners might still be outdated, but they should be refreshed in the coming hours..</p> <p><small>Feb <var data-var='date'> 16</var>, <var data-var='time'>21:15:28</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. We re-fetched external content (media files, profile pictures and banners) that were missing due to the incident. Unfortunately, they lost some stored files and any media (image or video) attached to statuses written by [mastodon.online](http://mastodon.online) users between 2024-02-13 23:58 UTC and 2024-02-14 16:11 UTC have been lost, and we do not have a way to recover them. We will work with our provider to understand better where their failure came from and ensure they are taking the needed steps to avoid this to happen again..</p> ]]> Wed, 14 Feb 2024 18:18:58 +0000 https://status.mastodon.online/incident/clsm48v6w25586bcojz440fhzq https://status.mastodon.online/incident/clsm48v6w25586bcojz440fhzq Database upgrade Type: Maintenance Duration: 30 minutes Affected Components: Background queues, Website & API Aug 7, 10:30:01 GMT+0 - Identified - Maintenance is now in progress Aug 7, 11:00:00 GMT+0 - Completed - Maintenance has completed successfully Aug 7, 10:30:00 GMT+0 - Identified - We will be upgrading PosgreSQL for mastodon.online. The upgrade should be a simple switchover and only result in a few minutes of downtime at most, hopefully less. Type: Maintenance

Duration: 30 minutes

Affected Components: ,

<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>10:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>11:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>10:30:00</var> GMT+0</small><br><strong>Identified</strong> - We will be upgrading PosgreSQL for mastodon.online. The upgrade should be a simple switchover and only result in a few minutes of downtime at most, hopefully less..</p> ]]> Mon, 7 Aug 2023 10:30:00 +0000 https://status.mastodon.online/maintenance/cll0oq9bf0592beoibnisrlys https://status.mastodon.online/maintenance/cll0oq9bf0592beoibnisrlys Email delivery issues Type: Incident Duration: 1 hour and 3 minutes Affected Components: Background queues Jul 20, 19:05:37 GMT+0 - Identified - We noticed a problem with our email deliveries (new signups, lost password…). We are working on it, it should be fixed in the next hours. Jul 20, 20:08:09 GMT+0 - Resolved - This incident has been resolved. E-mails are not correctly delivered. If you created an account in the last two days and did not receive an email, it should be delivered in the coming hours. Type: Incident

Duration: 1 hour and 3 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 20</var>, <var data-var='time'>19:05:37</var> GMT+0</small><br><strong>Identified</strong> - We noticed a problem with our email deliveries (new signups, lost password…). We are working on it, it should be fixed in the next hours..</p> <p><small>Jul <var data-var='date'> 20</var>, <var data-var='time'>20:08:09</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. E-mails are not correctly delivered. If you created an account in the last two days and did not receive an email, it should be delivered in the coming hours..</p> ]]> Thu, 20 Jul 2023 19:05:37 +0000 https://status.mastodon.online/incident/clkbivsoc163452caoco9tiikbe https://status.mastodon.online/incident/clkbivsoc163452caoco9tiikbe Hetzner Network Issues Type: Incident Duration: 1 day, 1 hour and 33 minutes Affected Components: Background queues, Website & API Jul 11, 14:34:07 GMT+0 - Resolved - This incident has been resolved. Jul 10, 13:01:32 GMT+0 - Investigating - There is currently an issue with Hetzner's network connection between Hetzner Cloud and Bare Metal, which is affecting our backend services. We are looking into the cause. Jul 10, 14:10:51 GMT+0 - Monitoring - We have moved our Redis server into Hetzner Cloud, where there are no network connectivity issues, and functionality has been restored. We will continue monitoring for any issues. Type: Incident

Duration: 1 day, 1 hour and 33 minutes

Affected Components: ,

<p><small>Jul <var data-var='date'> 11</var>, <var data-var='time'>14:34:07</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jul <var data-var='date'> 10</var>, <var data-var='time'>13:01:32</var> GMT+0</small><br><strong>Investigating</strong> - There is currently an issue with Hetzner's network connection between Hetzner Cloud and Bare Metal, which is affecting our backend services. We are looking into the cause..</p> <p><small>Jul <var data-var='date'> 10</var>, <var data-var='time'>14:10:51</var> GMT+0</small><br><strong>Monitoring</strong> - We have moved our Redis server into Hetzner Cloud, where there are no network connectivity issues, and functionality has been restored. We will continue monitoring for any issues..</p> ]]> Mon, 10 Jul 2023 13:01:32 +0000 https://status.mastodon.online/incident/cljwvh37i271303bwoj14b8gacl https://status.mastodon.online/incident/cljwvh37i271303bwoj14b8gacl Website is down Type: Incident Duration: 1 hour and 26 minutes Affected Components: Background queues, Website & API, Streaming API Jun 27, 08:51:15 GMT+0 - Identified - There is currently a network incident with our hosting provider, preventing our Redis server to be unreachable. We are waiting for an ETA from our provider, and are preparing setting up a new Redis server if the issue is not resolved quick enough. Jun 27, 09:27:07 GMT+0 - Monitoring - Our provider seems to have fixed the network issue and the services are back online. Your feeds might be a few minutes late while we are processing the messages posted during the downtime. We are closely monitoring the situation until our provider confirms everything is operational on their side. Jun 27, 10:03:01 GMT+0 - Identified - Network to the Redis server is down again, we are waiting for an update from our hosting provider. Jun 27, 10:17:13 GMT+0 - Resolved - This incident has been resolved and our provider confirmed the issue has been fixed on their side. Everything is back online and all messages in the backlog have been processed. Type: Incident

Duration: 1 hour and 26 minutes

Affected Components: , ,

<p><small>Jun <var data-var='date'> 27</var>, <var data-var='time'>08:51:15</var> GMT+0</small><br><strong>Identified</strong> - There is currently a network incident with our hosting provider, preventing our Redis server to be unreachable. We are waiting for an ETA from our provider, and are preparing setting up a new Redis server if the issue is not resolved quick enough..</p> <p><small>Jun <var data-var='date'> 27</var>, <var data-var='time'>09:27:07</var> GMT+0</small><br><strong>Monitoring</strong> - Our provider seems to have fixed the network issue and the services are back online. Your feeds might be a few minutes late while we are processing the messages posted during the downtime. We are closely monitoring the situation until our provider confirms everything is operational on their side..</p> <p><small>Jun <var data-var='date'> 27</var>, <var data-var='time'>10:03:01</var> GMT+0</small><br><strong>Identified</strong> - Network to the Redis server is down again, we are waiting for an update from our hosting provider..</p> <p><small>Jun <var data-var='date'> 27</var>, <var data-var='time'>10:17:13</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved and our provider confirmed the issue has been fixed on their side. Everything is back online and all messages in the backlog have been processed..</p> ]]> Tue, 27 Jun 2023 08:51:15 +0000 https://status.mastodon.online/incident/clje1t4ky212550azoglx5rye1t https://status.mastodon.online/incident/clje1t4ky212550azoglx5rye1t DB migration error Type: Incident Affected Components: Background queues Jun 6, 18:58:44 GMT+0 - Resolved - There was a small error in the DB migration steps when upgrading to the latest nightly. This was due to an older DB version being used in production. We have rolled back to a previous version and are continuing to monitor. Type: Incident

Affected Components:

<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>18:58:44</var> GMT+0</small><br><strong>Resolved</strong> - There was a small error in the DB migration steps when upgrading to the latest nightly. This was due to an older DB version being used in production. We have rolled back to a previous version and are continuing to monitor..</p> ]]> Tue, 6 Jun 2023 18:58:44 +0000 https://status.mastodon.online/incident/clikn9h6j112324b2ofly96cats https://status.mastodon.online/incident/clikn9h6j112324b2ofly96cats Service is unavailable Type: Incident Duration: 1 hour and 36 minutes Affected Components: Website & API May 5, 21:04:21 GMT+0 - Investigating - We are currently investigating this incident. May 5, 21:46:12 GMT+0 - Identified - We identified the issue and are working on it. May 5, 22:40:36 GMT+0 - Resolved - This incident has been resolved. An internal TLS certificate was renewed automatically, but did not get properly reloaded into our load balancer after renewal. It took us a long time to identify this, as no errors were visible to point us to the source of the issue. We identified the root cause and will take steps to prevent this from happening in the future. Type: Incident

Duration: 1 hour and 36 minutes

Affected Components:

<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>21:04:21</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>May <var data-var='date'> 5</var>, <var data-var='time'>21:46:12</var> GMT+0</small><br><strong>Identified</strong> - We identified the issue and are working on it..</p> <p><small>May <var data-var='date'> 5</var>, <var data-var='time'>22:40:36</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. An internal TLS certificate was renewed automatically, but did not get properly reloaded into our load balancer after renewal. It took us a long time to identify this, as no errors were visible to point us to the source of the issue. We identified the root cause and will take steps to prevent this from happening in the future..</p> ]]> Fri, 5 May 2023 21:04:21 +0000 https://status.mastodon.online/incident/clhb1nr780302gznj7h6ht6d6 https://status.mastodon.online/incident/clhb1nr780302gznj7h6ht6d6 API and streaming API inaccessible Type: Incident Duration: 23 minutes Affected Components: Website & API, Streaming API Apr 27, 04:22:07 GMT+0 - Investigating - We are currently investigating this incident. Apr 27, 04:42:50 GMT+0 - Identified - Problem was misconfiguration in the Helm chart. We're working on a fix. Apr 27, 04:44:45 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 23 minutes

Affected Components: ,

<p><small>Apr <var data-var='date'> 27</var>, <var data-var='time'>04:22:07</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 27</var>, <var data-var='time'>04:42:50</var> GMT+0</small><br><strong>Identified</strong> - Problem was misconfiguration in the Helm chart. We're working on a fix..</p> <p><small>Apr <var data-var='date'> 27</var>, <var data-var='time'>04:44:45</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Thu, 27 Apr 2023 04:22:07 +0000 https://status.mastodon.online/incident/clgymc2n3320418bnncwycenli1 https://status.mastodon.online/incident/clgymc2n3320418bnncwycenli1 Object storage misconfiguration Type: Incident Affected Components: Media storage Mar 16, 22:50:01 GMT+0 - Resolved - Early morning Feb 24 we were indirectly made aware of a misconfiguration on our object storage domain (files.mastodon.online) that allowed anyone to see the list of all uploaded files. Within 30 minutes this mistake was corrected. However, we have reasons to believe that the issue has existed since Feb 2, when we began upgrading our infrastructure. Normally Mastodon relies on long, randomly generated file names with high entropy to ensure that certain files are accessed only by those who know the link. However, that misconfiguration allowed that measure to be bypassed. Most files in our object storage are public in nature–profile pictures, custom emojis, images and videos attached to public posts. But there is a type of file that should never be accessed by anyone but its owner, and it’s the user’s archive takeout. When a user requests an archive takeout of their account, all of their posts, favorites and bookmarks are put into an archive which remains accessible for 7 days after. At the time of the incident, 1,037 such user archive takeouts were available. We have immediately deleted them to prevent anyone from downloading them. We have notified the affected users and we are changing the Mastodon software to not rely on high entropy links for access control to user archive takeouts any longer, as well as adding an automated check into the admin dashboard to detect similar misconfigurations and notify other server operators about them. Security is important to us and we are continuously improving our processes as we scale our organization from one employee to multiple to ensure that mistakes like this do not happen in the future. Type: Incident

Affected Components:

<p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>22:50:01</var> GMT+0</small><br><strong>Resolved</strong> - Early morning Feb 24 we were indirectly made aware of a misconfiguration on our object storage domain (files.mastodon.online) that allowed anyone to see the list of all uploaded files. Within 30 minutes this mistake was corrected. However, we have reasons to believe that the issue has existed since Feb 2, when we began upgrading our infrastructure. Normally Mastodon relies on long, randomly generated file names with high entropy to ensure that certain files are accessed only by those who know the link. However, that misconfiguration allowed that measure to be bypassed. Most files in our object storage are public in nature–profile pictures, custom emojis, images and videos attached to public posts. But there is a type of file that should never be accessed by anyone but its owner, and it’s the user’s archive takeout. When a user requests an archive takeout of their account, all of their posts, favorites and bookmarks are put into an archive which remains accessible for 7 days after. At the time of the incident, 1,037 such user archive takeouts were available. We have immediately deleted them to prevent anyone from downloading them. We have notified the affected users and we are changing the Mastodon software to not rely on high entropy links for access control to user archive takeouts any longer, as well as adding an automated check into the admin dashboard to detect similar misconfigurations and notify other server operators about them. Security is important to us and we are continuously improving our processes as we scale our organization from one employee to multiple to ensure that mistakes like this do not happen in the future..</p> ]]> Thu, 16 Mar 2023 22:50:01 +0000 https://status.mastodon.online/incident/clfbpf1ru3142hvocvvsed3vj https://status.mastodon.online/incident/clfbpf1ru3142hvocvvsed3vj Website is down Type: Incident Duration: 2 hours and 39 minutes Affected Components: Website & API, Background queues, Streaming API Mar 13, 22:40:58 GMT+0 - Identified - There is an issue in our Kubernetes cluster that we are working to resolve. Mar 14, 01:19:53 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 2 hours and 39 minutes

Affected Components: , ,

<p><small>Mar <var data-var='date'> 13</var>, <var data-var='time'>22:40:58</var> GMT+0</small><br><strong>Identified</strong> - There is an issue in our Kubernetes cluster that we are working to resolve..</p> <p><small>Mar <var data-var='date'> 14</var>, <var data-var='time'>01:19:53</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Mon, 13 Mar 2023 22:40:58 +0000 https://status.mastodon.online/incident/clf7ervju145599ejob2p0tgwp4 https://status.mastodon.online/incident/clf7ervju145599ejob2p0tgwp4 Remote media not loading Type: Incident Duration: 11 hours and 19 minutes Affected Components: Media storage Mar 1, 08:14:19 GMT+0 - Resolved - This was caused by a network maintenance from our hosting provider. The maintenance has ended and our response times and error rates are back to normal. Feb 28, 20:55:00 GMT+0 - Identified - There are intermittent failures with uploading to object storage, which results in "Not available" previews. Type: Incident

Duration: 11 hours and 19 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>08:14:19</var> GMT+0</small><br><strong>Resolved</strong> - This was caused by a network maintenance from our hosting provider. The maintenance has ended and our response times and error rates are back to normal..</p> <p><small>Feb <var data-var='date'> 28</var>, <var data-var='time'>20:55:00</var> GMT+0</small><br><strong>Identified</strong> - There are intermittent failures with uploading to object storage, which results in "Not available" previews..</p> ]]> Tue, 28 Feb 2023 20:55:00 +0000 https://status.mastodon.online/incident/cleoqt8zj429541i9n6l8t2rutp https://status.mastodon.online/incident/cleoqt8zj429541i9n6l8t2rutp