mastodon.social - Connectivity/timeout issues with sidekiq – Incident details

Experiencing partially degraded performance

Connectivity/timeout issues with sidekiq

Resolved
Operational
Started 7 months agoLasted about 1 month

Affected

Website & API

Degraded performance from 2:00 PM to 6:42 AM, Operational from 6:42 AM to 9:22 AM

Background queues

Degraded performance from 2:00 PM to 6:42 AM, Operational from 6:42 AM to 9:22 AM

Updates
  • Resolved
    UTC
    Resolved

    Sidekiq has been stable, so considering this incident resolved.

  • Monitoring
    UTC
    Monitoring

    We found out the cause for this issue (within a dependency) and we deployed a fix 8 hours ago. So far, it seems to have completely fixed the issue.

  • Investigating
    UTC
    Investigating

    There appears to be an issue with connection timeouts to our Sidekiq runners, causing some operations to take longer than expected, or require multiple tries. We are currently investigating this issue.