Notice history

~0.0 ms

Nov 2024

Dec 2024

Jan 2025

~0.0 s

Nov 2024

Dec 2024

Jan 2025

Nov 2024

Dec 2024

Jan 2025

Nov 2024

Dec 2024

Jan 2025

Resolved
February 10, 2025 at 1:30 PMUTC
Resolved
February 10, 2025 at 1:30 PMUTC
Data has been restored, and performance issues have been mitigated.
Identified
January 29, 2025 at 5:30 PMUTC
Identified
January 29, 2025 at 5:30 PMUTC
We've experienced an issue with our recent cache storage upgrade, causing some ephermal data loss (e.g. home feeds losing scrollback or showing the number of active users as "0"). We are working on fixing the underlying issue and restoring/regenerating the lost data.
Investigating
January 29, 2025 at 4:49 PMUTC
Investigating
January 29, 2025 at 4:49 PMUTC
We are currently investigating this incident.

Resolved
February 05, 2025 at 2:39 PMUTC
Resolved
February 05, 2025 at 2:39 PMUTC
This incident has been resolved.
Monitoring
February 04, 2025 at 5:06 PMUTC
Monitoring
February 04, 2025 at 5:06 PMUTC
The database software developer has been very responsive about this issue. They helped us troubleshoot it and provided us with a new version that solved the problem.
We are currently monitoring the system closely, but everything seems to work fine now.
Identified
January 28, 2025 at 12:55 PMUTC
Identified
January 28, 2025 at 12:55 PMUTC
Last week, we switched our Redis database to another software. This led to user reports of timelines not updating anymore with using some clients, like Ivory or Pachli.
After investigating the issue, we discovered that this new software returns invalid data in some specific cases, which are triggered by the API those Mastodon clients use.
We have reached out to the software developers to troubleshoot this and are hopeful it will get fixed soon. In the meantime, we are also working on a way to roll back to our previous software in case we can not get this fixed quickly.
We will post more details on this incident when possible.

Resolved
January 25, 2025 at 8:43 AMUTC
Resolved
January 25, 2025 at 8:43 AMUTC
This incident has been resolved.
Investigating
January 25, 2025 at 8:12 AMUTC
Investigating
January 25, 2025 at 8:12 AMUTC
We are currently investigating this incident.

Database upgrade

Completed
January 21, 2025 at 11:44 AMUTC
Completed
January 21, 2025 at 11:44 AMUTC
Database is successfully upgraded, all configuration errors have been cleaned up. Thank you to everyone for your patience.
Update
January 21, 2025 at 11:00 AMUTC
Update
January 21, 2025 at 11:00 AMUTC
The upgrade was successful and mastodon.social is now running on the newer database servers.
There is a small delay in processing due to the backlog of request accumulated during our downtime, but everything should be back to real-time soon.
We are closely monitoring the servers and will update this status once everything is back to normal.
Update
January 21, 2025 at 9:57 AMUTC
Update
January 21, 2025 at 9:57 AMUTC
We encountered a configuration issue while performing the update. We are working on a fix and the service should be back online soon.
Update
January 21, 2025 at 9:31 AMUTC
Update
January 21, 2025 at 9:31 AMUTC
Maintenance is now in progress.
Update
January 21, 2025 at 9:00 AMUTC
Update
January 21, 2025 at 9:00 AMUTC
Maintenance is now in progress
In progress
January 21, 2025 at 9:00 AMUTC
In progress
January 21, 2025 at 9:00 AMUTC
We will be upgrading the database systems to new hardware, as well as a new major version of PostgreSQL. The upgrade should be very quick, but some small downtime will still occur for around a minute.

Resolved
January 16, 2025 at 10:09 AMUTC
Resolved
January 16, 2025 at 10:09 AMUTC
No acknowledgement from Hetzner, but everything remains stable. Considering this closed.
Monitoring
January 16, 2025 at 7:18 AMUTC
Monitoring
January 16, 2025 at 7:18 AMUTC
DB connections are stable again. Current suspect is a Hetzner networking issue somewhere in the backend. Currently monitoring.
Investigating
January 16, 2025 at 7:13 AMUTC
Investigating
January 16, 2025 at 7:13 AMUTC
There appears to be an issue with our nodes connecting to the DB reliably. We are currently investigating the issue.

No notices reported this month

Resolved
November 14, 2024 at 10:48 AMUTC
Resolved
November 14, 2024 at 10:48 AMUTC
Capacity issues have been resolved, and the autoscaler has been adjusted.
Identified
November 13, 2024 at 4:17 PMUTC
Identified
November 13, 2024 at 4:17 PMUTC
There was an issue with our cluster autoscaler where it seems to have scaled down the number of nodes we have for our cluster, resulting in strained resources. We are trying to scale back up, but Hetzner is having availability issues in the region we are deployed in. We are working to resolve the issue as quickly as possible.

Nov 2024 to Jan 2025

mastodon.social - Notice history