mastodon.social - Notice history

All systems operational

Website & API - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 99.83%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Background queues - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 99.86%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Media storage - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 99.86%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Streaming API - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 99.86%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Notice history

Feb 2025

No notices reported this month

Jan 2025

Degraded performance
  • Resolved
    Resolved

    Data has been restored, and performance issues have been mitigated.

  • Identified
    Identified

    We've experienced an issue with our recent cache storage upgrade, causing some ephermal data loss (e.g. home feeds losing scrollback or showing the number of active users as "0"). We are working on fixing the underlying issue and restoring/regenerating the lost data.

  • Investigating
    Investigating
    We are currently investigating this incident.
Timelines are not updating with some clients
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    The database software developer has been very responsive about this issue. They helped us troubleshoot it and provided us with a new version that solved the problem.

    We are currently monitoring the system closely, but everything seems to work fine now.

  • Identified
    Identified

    Last week, we switched our Redis database to another software. This led to user reports of timelines not updating anymore with using some clients, like Ivory or Pachli.

    After investigating the issue, we discovered that this new software returns invalid data in some specific cases, which are triggered by the API those Mastodon clients use.

    We have reached out to the software developers to troubleshoot this and are hopeful it will get fixed soon. In the meantime, we are also working on a way to roll back to our previous software in case we can not get this fixed quickly.

    We will post more details on this incident when possible.

Website inaccessible
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
Database upgrade
  • Completed
    January 21, 2025 at 11:44 AM
    Completed
    January 21, 2025 at 11:44 AM

    Database is successfully upgraded, all configuration errors have been cleaned up. Thank you to everyone for your patience.

  • Update
    January 21, 2025 at 11:00 AM
    In progress
    January 21, 2025 at 11:00 AM

    The upgrade was successful and mastodon.social is now running on the newer database servers.

    There is a small delay in processing due to the backlog of request accumulated during our downtime, but everything should be back to real-time soon.

    We are closely monitoring the servers and will update this status once everything is back to normal.

  • Update
    January 21, 2025 at 9:57 AM
    In progress
    January 21, 2025 at 9:57 AM

    We encountered a configuration issue while performing the update. We are working on a fix and the service should be back online soon.

  • Update
    January 21, 2025 at 9:31 AM
    In progress
    January 21, 2025 at 9:31 AM
    Maintenance is now in progress.
  • Update
    January 21, 2025 at 9:00 AM
    In progress
    January 21, 2025 at 9:00 AM
    Maintenance is now in progress
  • In progress
    January 21, 2025 at 9:00 AM
    In progress
    January 21, 2025 at 9:00 AM

    We will be upgrading the database systems to new hardware, as well as a new major version of PostgreSQL. The upgrade should be very quick, but some small downtime will still occur for around a minute.

Backend DB connection issues
  • Resolved
    Resolved

    No acknowledgement from Hetzner, but everything remains stable. Considering this closed.

  • Monitoring
    Monitoring

    DB connections are stable again. Current suspect is a Hetzner networking issue somewhere in the backend. Currently monitoring.

  • Investigating
    Investigating

    There appears to be an issue with our nodes connecting to the DB reliably. We are currently investigating the issue.

Dec 2024

No notices reported this month

Dec 2024 to Feb 2025

Next