mastodon.social - Backend outage – Incident details

All systems operational

Backend outage

Resolved
Major outage
Started about 1 month agoLasted about 3 hours

Affected

Website & API

Major outage from 10:33 PM to 10:42 PM, Degraded performance from 10:42 PM to 1:13 AM

Background queues

Major outage from 10:33 PM to 10:42 PM, Degraded performance from 10:42 PM to 1:13 AM

Updates
  • Resolved
    Resolved

    Kubernetes node issues have been resolved, and service is now fully restored.

  • Identified
    Identified

    Website is operative again. Issue seems to be related to kubernetes pods becoming completely unresponsive. Continuing to investigate.

  • Investigating
    Investigating

    We are experiencing issues with our redis caching system, as well as kubernetes node errors. We are currently investigating the issue.