Some Projects Unreachable following Project Restart

Updates timeline

1. Identified Apr 27, 10:54 AM UTC

   We’ve identified that a recent change to the Postgres systemd service has a dependency on a prestart script. This script is not present on some older projects, and when those projects restart, Postgres may fail to start successfully. This can result in projects remaining offline. We are working on a fix to restore service.

2. Identified Apr 27, 11:11 AM UTC

   We’ve identified that a recent change to the Postgres systemd service has a dependency on a prestart script. This script is not present on some older projects, and when those projects restart, Postgres may fail to start successfully. This can result in projects remaining offline. We are working on a fix to restore service. If you are running a version of Postgres older than 15.1.1.57, please avoid restarting your project until this issue is resolved.

3. Identified Apr 27, 11:40 AM UTC

   Our team is actively working on a fix and is manually patching impacted projects to restore service as quickly as possible. If you are running a version of Postgres older than 15.1.1.57, please avoid restarting your project until this issue is resolved.

4. Monitoring Apr 27, 01:12 PM UTC

   All impacted projects with open support tickets have now been manually patched and restored. We are currently rolling out a fix across the fleet to prevent further impact. We will continue to monitor progress and provide updates.

5. Identified Apr 27, 02:21 PM UTC

   We’ve identified additional impacted projects and are currently working to manually patch and restore them. We are also continuing to roll out a fix across the fleet to prevent further impact.

6. Identified Apr 27, 04:05 PM UTC

   The permanent fix has now been rolled out across the fleet. We are continuing to manually patch any remaining impacted projects to restore service.

7. Monitoring Apr 27, 05:31 PM UTC

   The fix has been rolled out, and we are monitoring for continued stability. Any remaining impacted projects are being manually patched.

8. Resolved Apr 27, 08:05 PM UTC

   All projects are patched and this is now resolved. Thank you for your patience while we ensured all projects were updated.