mastodon.social - Under heavy load – Incident details

All systems operational

Under heavy load

Resolved
Degraded performance
Started about 1 year agoLasted 2 days

Affected

Website & API

Degraded performance from 7:51 PM to 4:14 PM

Background queues

Degraded performance from 7:51 PM to 4:14 PM

Updates
  • Resolved
    Resolved

    All queues are caught up, and performance is currently stable.

  • Monitoring
    Update

    All pending e-mails have now been delivered and new emails (on sign-up or password reset for example) are properly delivered.

    We are still lagging behind on ingesting content from other Mastodon servers, but this is resorbing and we expect to be back to normal in a few hours.

    The load on our infrastructure is closely monitored and we are working on various optimisations to better handle this newload, as well as adding more compute power to our infrastructure.

  • Monitoring
    Monitoring

    We switched to a new email provider and our backlog of emails (Subscription confirmation, password reset emails…) is shrinking and should be back to normal in the coming hours.

    Additional resources and optimisation have been deployed to help with the web services, we are still seeing some errors but the situation should be much better now.

    There is still a significant delay on statuses processing, but we hope the new resources will help getting it under control.

  • Investigating
    Investigating

    Additional DB resources have been added to help ease the load. Performance should steadily return to normal as background tasks catch up.

    The rapid influx of new users caused us to exceed our email sending limit, and as such emails are no longer being delivered. We are currently working on extending these limits.

  • Monitoring
    Monitoring

    We are experiencing a higher-than-average number of active users, and performance may be slightly degraded as a result.