mastodon.social - Connectivity/timeout issues with sidekiq – Incident details

Connectivity/timeout issues with sidekiq

Resolved763 hours 22 minutes

Affected

Website & API

Degraded performance from 2:00 PM to 6:42 AM, Operational from 6:42 AM to 9:22 AM

Background queues

Degraded performance from 2:00 PM to 6:42 AM, Operational from 6:42 AM to 9:22 AM

  • ResolvedUTC

    Sidekiq has been stable, so considering this incident resolved.

  • MonitoringUTC

    We found out the cause for this issue (within a dependency) and we deployed a fix 8 hours ago. So far, it seems to have completely fixed the issue.

  • InvestigatingUTC

    There appears to be an issue with connection timeouts to our Sidekiq runners, causing some operations to take longer than expected, or require multiple tries. We are currently investigating this issue.