mastodon.social - Elasticsearch issues – Incident details

All systems operational

Show current status

Elasticsearch issues

ResolvedMajor outage1 hour 2 minutes

Affected

Website & API

Major outage from 9:24 PM to 9:46 PM, Degraded performance from 9:46 PM to 3:07 PM

Background queues

Major outage from 9:24 PM to 9:46 PM, Degraded performance from 9:46 PM to 10:26 PM

  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    Search functionality has been restored and volume timeouts seem to have been fixed. We will continue to monitor the situation.

  • IdentifiedUTC

    The issue seems to be incredibly high timeouts on Hetzner volumes, causing components with high-throughput volumes like Elasticsearch to degrade and fail. Search has temporarily been disabled while we investigate this issue and attempt to work around it.

  • InvestigatingUTC

    There appears to be an issue with our Elasticsearch cluster that is causing the instance to fail. We are currently investigating the issue.