mastodon.social - Database connection issue – Incident details

Experiencing partially degraded performance

Database connection issue

Resolved
Major outage
Started over 2 years agoLasted about 8 hours

Affected

Website & API

Major outage from 12:20 PM to 12:44 PM, Operational from 12:44 PM to 3:06 PM, Major outage from 3:06 PM to 3:39 PM, Operational from 3:39 PM to 8:08 PM

Background queues

Major outage from 12:20 PM to 12:44 PM, Operational from 12:44 PM to 3:06 PM, Major outage from 3:06 PM to 3:39 PM, Operational from 3:39 PM to 8:08 PM

Updates
  • Resolved
    UTC
    Resolved

    Our provider identified the source of the network issues. It was caused by a router bug, and they have been able to update them to a new version that fixes the issue.

  • Monitoring
    UTC
    Monitoring

    The network issue has been fixed by our provider and all services are back online.

    We are monitoring the situation, but Hetzner has not yet announced that the underlying issue is fixed, so it may break again.

  • Identified
    UTC
    Identified

    Our hosting provider is having a network issue again, causing mastodon.social to be unavailable.

    We are monitoring the situation to ensure everything comes back online as soon as the network is working again.

  • Monitoring
    UTC
    Monitoring

    Database connectivity is restored. Continuing to monitor the situation while the Hetzner status remains open.

  • Identified
    UTC
    Identified

    Confirmed that Hetzner is experiencing an outage with their vSwitches, which we are using for our database connection. We will continue to monitor.

    https://status.hetzner.com/incident/9406c500-9c8b-48be-9591-a73691134096

  • Investigating
    UTC
    Investigating

    We are currently experiencing an issue connecting to the database. Investigation is ongoing.