mastodon.online - Media storage is unavailable – Incident details

All systems operational

Media storage is unavailable

Resolved
Operational
Started 8 months agoLasted 2 days

Affected

Media storage

Operational from 6:18 PM to 9:15 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved.

    We re-fetched external content (media files, profile pictures and banners) that were missing due to the incident.

    Unfortunately, they lost some stored files and any media (image or video) attached to statuses written by mastodon.online users between 2024-02-13 23:58 UTC and 2024-02-14 16:11 UTC have been lost, and we do not have a way to recover them.

    We will work with our provider to understand better where their failure came from and ensure they are taking the needed steps to avoid this to happen again.

  • Identified
    Update

    We finished ingesting media file for every status from a remote instance that was missing data due to our provider's outage.

    Unfortunately, they lost some stored files and any media (image or video) attached to statuses written by mastodon.online users between 2024-02-13 23:58 UTC and 2024-02-14 16:11 UTC have been lost, and we do not have a way to recover them.

    Some remote user profile pictures and banners might still be outdated, but they should be refreshed in the coming hours.

  • Identified
    Update

    Our media storage service is back online.

    Old statuses may still be missing images, we are working on fetching them again.

  • Identified
    Update

    Our provider is still working on restoring access to the media (images and videos) files storage.

    They estimate the recovery will last until at least 06:00 UTC tomorrow.

    Until this is completed, newly posts from the federation will not display media, and it is not possible to write new posts containing images or videos.

  • Identified
    Identified

    Our storage provider is having issues in the region we use to store media files on mastodon.online.

    They are working to resolve this, in the meantime some media files might not load correctly, or new posts can fail to be posted.