mastodon.social - Elasticsearch issues – Incident details

Experiencing partially degraded performance

Elasticsearch issues

Monitoring
Major outage
Started about 1 month ago

Affected

Website & API

Major outage from 9:24 PM to 9:46 PM, Degraded performance from 9:46 PM to 12:00 AM

Background queues

Major outage from 9:24 PM to 9:46 PM, Degraded performance from 9:46 PM to 12:00 AM

Updates
  • Monitoring
    UTC
    Monitoring

    Search functionality has been restored and volume timeouts seem to have been fixed. We will continue to monitor the situation.

  • Identified
    UTC
    Identified

    The issue seems to be incredibly high timeouts on Hetzner volumes, causing components with high-throughput volumes like Elasticsearch to degrade and fail. Search has temporarily been disabled while we investigate this issue and attempt to work around it.

  • Investigating
    UTC
    Investigating

    There appears to be an issue with our Elasticsearch cluster that is causing the instance to fail. We are currently investigating the issue.