Duration: 27 minutes

Affected Components: Website & API, Streaming API, Background queues

Feb 25, 21:08:32 GMT+0
Investigating - Hetzner, our cloud provider, appears to be having a significant outage, leaving many of our resources unavailable. We are monitoring the issue..

Feb 25, 21:16:30 GMT+0
Monitoring - Services have been restored and Hetzner connectivity seems to have returned. Will continue to monitor the situation..

Feb 25, 21:35:12 GMT+0
Resolved - This incident has been resolved..

Duration: 12 days, 18 hours and 22 minutes

Affected Components: Website & API, Background queues

Dec 17, 14:00:00 GMT+0
Investigating - There appears to be an issue with connection timeouts to our Sidekiq runners, causing some operations to take longer than expected, or require multiple tries. We are currently investigating this issue..

Jan 7, 06:42:57 GMT+0
Monitoring - We found out the cause for this issue (within a dependency) and we deployed a fix 8 hours ago. So far, it seems to have completely fixed the issue..

Mar 30, 08:22:20 GMT+0
Resolved - Sidekiq has been stable, so considering this incident resolved..

Duration: 8 minutes

Affected Components: Website & API, Background queues

Nov 3, 17:20:27 GMT+0
Identified - [mastodon.social](http://mastodon.social) is currently unresponsive to a database issue. We identified the problem and are working on fixing it..

Nov 3, 17:28:47 GMT+0
Resolved - This incident has been resolved, everything is now operational again..

Duration: 6 days, 2 hours and 35 minutes

Jun 17, 13:50:20 GMT+0
Identified - We paused sending emails from [mastodon.social](http://mastodon.social) (including password recovery and account confirmation emails) due to an identified issue that sent duplicate emails about our upcoming Terms of Service changes to some users. We are working on fixing this and avoiding more duplicate emails to be sent..

Jun 17, 17:42:30 GMT+0
Monitoring - Email sending has been removed. Servers are working through the backlog, and any pending email should arrive in the next few hours..

Jun 18, 05:40:36 GMT+0
Identified - The backlog of emails has been sent, and emails are reaching out their recipients now. Some emails sent in the last 12 hours are stuck with our email provider and we are working with them to fix this situation..

Jun 23, 16:25:19 GMT+0
Resolved - This incident has been resolved last Thursday, and email have been delivered correctly since then..

Duration: 2 hours and 40 minutes

Affected Components: Website & API, Background queues

Jun 11, 22:33:26 GMT+0
Investigating - We are experiencing issues with our redis caching system, as well as kubernetes node errors. We are currently investigating the issue..

Jun 11, 22:42:52 GMT+0
Identified - Website is operative again. Issue seems to be related to kubernetes pods becoming completely unresponsive. Continuing to investigate..

Jun 12, 01:13:29 GMT+0
Resolved - Kubernetes node issues have been resolved, and service is now fully restored..

Duration: 16 minutes

Affected Components: Background queues, Website & API

May 28, 06:30:01 GMT+0
Identified - Maintenance is now in progress.

May 28, 06:30:00 GMT+0
Identified - We will be updating the database to the next minor version, as well as some OS components that will require a restart. Minor downtime of roughly a minute is expected..

May 28, 06:46:24 GMT+0
Completed - Maintenance has completed successfully..

Duration: 11 days, 20 hours and 41 minutes

Affected Components: Website & API

Jan 29, 16:49:03 GMT+0
Investigating - We are currently investigating this incident..

Jan 29, 17:30:58 GMT+0
Identified - We've experienced an issue with our recent cache storage upgrade, causing some ephermal data loss (e.g. home feeds losing scrollback or showing the number of active users as "0"). We are working on fixing the underlying issue and restoring/regenerating the lost data..

Feb 10, 13:30:27 GMT+0
Resolved - Data has been restored, and performance issues have been mitigated..

Duration: 8 days, 1 hour and 45 minutes

Affected Components: Website & API

Feb 5, 14:39:41 GMT+0
Resolved - This incident has been resolved..

Jan 28, 12:55:00 GMT+0
Identified - Last week, we switched our Redis database to another software. This led to user reports of timelines not updating anymore with using some clients, like Ivory or Pachli. After investigating the issue, we discovered that this new software returns invalid data in some specific cases, which are triggered by the API those Mastodon clients use. We have reached out to the software developers to troubleshoot this and are hopeful it will get fixed soon. In the meantime, we are also working on a way to roll back to our previous software in case we can not get this fixed quickly. We will post more details on this incident when possible..

Feb 4, 17:06:12 GMT+0
Monitoring - The database software developer has been very responsive about this issue. They helped us troubleshoot it and provided us with a new version that solved the problem. We are currently monitoring the system closely, but everything seems to work fine now..

Duration: 30 minutes

Affected Components: Website & API

Jan 25, 08:43:04 GMT+0
Resolved - This incident has been resolved..

Jan 25, 08:12:40 GMT+0
Investigating - We are currently investigating this incident..

Duration: 2 hours and 44 minutes

Affected Components: Streaming API, Media storage, Background queues, Website & API

Jan 21, 09:00:00 GMT+0
Identified - We will be upgrading the database systems to new hardware, as well as a new major version of PostgreSQL. The upgrade should be very quick, but some small downtime will still occur for around a minute..

Jan 21, 09:31:23 GMT+0
Identified - Maintenance is now in progress..

Jan 21, 09:57:20 GMT+0
Identified - We encountered a configuration issue while performing the update. We are working on a fix and the service should be back online soon..

Jan 21, 09:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 21, 11:00:54 GMT+0
Identified - The upgrade was successful and [mastodon.social](http://mastodon.social) is now running on the newer database servers. There is a small delay in processing due to the backlog of request accumulated during our downtime, but everything should be back to real-time soon. We are closely monitoring the servers and will update this status once everything is back to normal..

Jan 21, 11:44:00 GMT+0
Completed - Database is successfully upgraded, all configuration errors have been cleaned up. Thank you to everyone for your patience..

Duration: 2 hours and 55 minutes

Affected Components: Website & API

Jan 16, 07:13:58 GMT+0
Investigating - There appears to be an issue with our nodes connecting to the DB reliably. We are currently investigating the issue..

Jan 16, 07:18:39 GMT+0
Monitoring - DB connections are stable again. Current suspect is a Hetzner networking issue somewhere in the backend. Currently monitoring..

Jan 16, 10:09:20 GMT+0
Resolved - No acknowledgement from Hetzner, but everything remains stable. Considering this closed..

Duration: 18 hours and 31 minutes

Affected Components: Streaming API, Background queues, Website & API

Nov 13, 16:17:05 GMT+0
Identified - There was an issue with our cluster autoscaler where it seems to have scaled down the number of nodes we have for our cluster, resulting in strained resources. We are trying to scale back up, but Hetzner is having availability issues in the region we are deployed in. We are working to resolve the issue as quickly as possible..

Nov 14, 10:48:00 GMT+0
Resolved - Capacity issues have been resolved, and the autoscaler has been adjusted..

Duration: 3 hours and 13 minutes

Affected Components: Background queues

Sep 16, 06:50:00 GMT+0
Investigating - We are currently unable to send emails to users due to issues with Postmark. We are currently investigating the issue..

Sep 16, 10:02:51 GMT+0
Resolved - Issue was identified as an SSL error on Postmark's side. This has now been resolved, and emails are working again..

Duration: 4 minutes

Affected Components: Website & API

Mar 15, 19:11:51 GMT+0
Identified - We've identified an inefficient database query in the most recent update and are performing a rollback to last good version. Sorry for the inconvenience..

Mar 15, 19:15:52 GMT+0
Resolved - This incident has been resolved..

Duration: 7 hours and 48 minutes

Affected Components: Background queues, Website & API

Mar 12, 12:20:54 GMT+0
Investigating - We are currently experiencing an issue connecting to the database. Investigation is ongoing..

Mar 12, 12:31:04 GMT+0
Identified - Confirmed that Hetzner is experiencing an outage with their vSwitches, which we are using for our database connection. We will continue to monitor. <https://status.hetzner.com/incident/9406c500-9c8b-48be-9591-a73691134096>.

Mar 12, 12:44:58 GMT+0
Monitoring - Database connectivity is restored. Continuing to monitor the situation while the Hetzner status remains open..

Mar 12, 15:06:13 GMT+0
Identified - Our hosting provider is having a network issue again, causing [mastodon.social](http://mastodon.social) to be unavailable. We are monitoring the situation to ensure everything comes back online as soon as the network is working again..

Mar 12, 15:39:56 GMT+0
Monitoring - The network issue has been fixed by our provider and all services are back online. We are monitoring the situation, but Hetzner has not yet announced that the underlying issue is fixed, so it may break again..

Mar 12, 20:08:41 GMT+0
Resolved - Our provider identified the source of the network issues. It was caused by a router bug, and they have been able to update them to a new version that fixes the issue..

Duration: 3 minutes

Affected Components: Streaming API, Background queues, Website & API

Jan 22, 23:13:23 GMT+0
Resolved - The fault has been fixed..

Jan 22, 23:10:41 GMT+0
Investigating - We were notified of a switch fault in the data center of our hosting provider..

Duration: 4 hours and 37 minutes

Affected Components: Website & API

Jan 15, 05:42:06 GMT+0
Investigating - We are currently investigating this incident..

Jan 15, 09:02:24 GMT+0
Identified - Issues has been discovered; one of the indexes was behaving in strange ways and causing regular errors/retries. After recreating the index, cluster is now behaving as expected. Index is currently rebuilding from current data..

Jan 15, 10:18:43 GMT+0
Resolved - Elasticsearch indexes have rebuilt and search is working as intended..

Duration: 6 hours and 52 minutes

Affected Components: Website & API

Jan 3, 05:53:52 GMT+0
Investigating - We are currently investigating this incident..

Jan 3, 12:46:10 GMT+0
Resolved - Performance issues were a combination of a suspected spam attack, and some imbalance with the kubernetes nodes. Spam sources have been blocked and nodes have been rebalanced..

Duration: 15 days, 23 hours and 33 minutes

Affected Components: Background queues, Website & API

Oct 27, 10:56:45 GMT+0
Identified - The situation is now stable and services are working, but the performance is not back to the previous levels. The problem seems to be tied to the kernel version we are running, which may not have full support for the new hardware we deployed. We are working on updating our systems to a newer OS version and tuning the various system settings to ensure everything goes back to expected levels of performance..

Dec 12, 14:52:41 GMT+0
Resolved - Performance has improved after upgrading various OS packages, and upgrading kubernetes to more recent versions with performance improvements. Additional work is still ongoing to identify other areas where performance can be improved, but current performance is now better than it was..

Oct 26, 15:19:35 GMT+0
Investigating - There appears to be degraded performance after a move away from deprecated Hetzner nodes types. Currently investigating. .

Oct 26, 18:54:43 GMT+0
Investigating - Definite cause of the degradation has yet to be identified, however moving to more powerful nodes for the time being seems to have mitigated the issue. Sidekiq queues are still behind, but are no longer growing, and additional workers have been deployed to help it catch up. A ticket has been submitted with Hetzner to see if this could be an issue on their side. Waiting to hear back..

Duration: 16 minutes

Affected Components: Background queues, Website & API

Aug 8, 10:30:01 GMT+0
Identified - Maintenance is now in progress.

Aug 8, 10:30:00 GMT+0
Identified - We will be switching the database over to newer, more powerful hardware. This is a simple connection switch, so there should be minimal downtime..

Aug 8, 10:45:57 GMT+0
Completed - Database switch is finished. New hardware is working nicely..

Duration: 1 hour

Affected Components: Background queues, Website & API

Aug 5, 18:59:23 GMT+0
Investigating - We are experiencing an abnormal database load. We are currently investigating this incident..

Aug 5, 19:14:42 GMT+0
Identified - The drive array on our main database server seems to be having issues. You may experience some pages failing to load. Posts from other instances are also experiencing some delay as we are limiting their processing to get the database back at full speed..

Aug 5, 19:59:49 GMT+0
Resolved - This incident has been resolved..

Duration: 1 day

Affected Components: Website & API

Aug 3, 15:14:00 GMT+0
Resolved - This incident has been resolved..

Aug 2, 15:14:05 GMT+0
Identified - We've identified an issue with our search engine. While autocomplete still works, searching for posts currently doesn't. We're working on a solution..

Aug 2, 19:16:14 GMT+0
Monitoring - The search feature is back after a recovery procedure, but we will deploy a new infrastructure for it tomorrow to avoid any future issues..

Duration: 1 hour and 7 minutes

Affected Components: Background queues, Website & API

Jul 12, 13:52:41 GMT+0
Identified - Starting a little earlier, as the old database is having some issues anyway..

Jul 12, 15:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jul 12, 14:00:00 GMT+0
Identified - We will be performing a major database upgrade on the backend. We've created a process to minimize downtime, however there will still be some downtime during the process (est. 15-20 minutes)..

Duration: 2 hours and 10 minutes

Affected Components: Streaming API, Website & API

Jul 9, 12:24:18 GMT+0
Investigating - We are currently investigating this incident..

Jul 9, 12:30:02 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jul 9, 14:34:41 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours and 1 minute

Affected Components: Streaming API, Website & API

Jul 8, 18:13:32 GMT+0
Identified - We've identified an issue with the database server and are working on remedying it..

Jul 8, 19:07:47 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jul 8, 21:14:11 GMT+0
Resolved - This incident has been resolved..

Duration: 5 hours and 56 minutes

Affected Components: Background queues, Website & API

Jul 6, 19:03:08 GMT+0
Investigating - The database is currently experience extremely high load. We are currently investigating the cause..

Jul 6, 19:42:34 GMT+0
Identified - The increase in database load is likely the result of significantly increased user traffic over the last several days, revealing inefficiencies in some of our database background jobs. We are implementing fixes, both temporary and long-term..

Jul 6, 19:59:21 GMT+0
Monitoring - Database performance has returned to normal..

Jul 7, 00:58:54 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 45 minutes

Affected Components: Background queues, Website & API

Jul 6, 05:29:06 GMT+0
Resolved - This incident has been resolved..

Jul 6, 03:44:15 GMT+0
Investigating - The database server appears to be experiencing an issue. Active investigation is underway..

Jul 6, 03:48:10 GMT+0
Monitoring - Database is back up & running..

Duration: 1 day, 20 hours and 23 minutes

Affected Components: Background queues, Website & API

Jul 1, 19:51:01 GMT+0
Monitoring - We are experiencing a higher-than-average number of active users, and performance may be slightly degraded as a result..

Jul 2, 08:20:29 GMT+0
Investigating - Additional DB resources have been added to help ease the load. Performance should steadily return to normal as background tasks catch up. The rapid influx of new users caused us to exceed our email sending limit, and as such emails are no longer being delivered. We are currently working on extending these limits..

Jul 2, 15:28:18 GMT+0
Monitoring - We switched to a new email provider and our backlog of emails (Subscription confirmation, password reset emails…) is shrinking and should be back to normal in the coming hours. Additional resources and optimisation have been deployed to help with the web services, we are still seeing some errors but the situation should be much better now. There is still a significant delay on statuses processing, but we hope the new resources will help getting it under control..

Jul 2, 22:25:24 GMT+0
Monitoring - All pending e-mails have now been delivered and new emails (on sign-up or password reset for example) are properly delivered. We are still lagging behind on ingesting content from other Mastodon servers, but this is resorbing and we expect to be back to normal in a few hours. The load on our infrastructure is closely monitored and we are working on various optimisations to better handle this newload, as well as adding more compute power to our infrastructure..

Jul 3, 16:14:24 GMT+0
Resolved - All queues are caught up, and performance is currently stable..

Duration: 33 minutes

Affected Components: Background queues

Jun 1, 19:52:59 GMT+0
Identified - We've identified a backlog of items in our processing queue which means that some posts will appear only with a delay. We are working on resolving the issue..

Jun 1, 20:26:17 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 39 minutes

Affected Components: Media storage

Feb 3, 08:57:30 GMT+0
Identified - DNS has been switched to the new provider. We are monitoring the service..

Feb 3, 08:30:00 GMT+0
Identified - We are moving our media files (files.mastodon.social) to a new CDN provider. There should be no downtime or visible impact..

Feb 3, 10:08:42 GMT+0
Completed - No issues have been observed, the migration is now complete..