Project Lifecycle Actions Showing increased error rates across all regions

Updates timeline

1. Investigating Sep 4, 12:15 AM UTC

   We are seeing increased error rates in actions across all project lifecycle actions - Creations, Deletions, Restores, Configuration and Compute size upgrades. We are currently looking into the cause. Out of an abundance of caution, we would suggest against taking any of these actions until this incident is resolved. We will provide additional updates as we have more information

2. Identified Sep 4, 12:30 AM UTC

   We have identified the source of the errors with project lifecycle actions, and the team is currently working to resolve it. This is impacting all project-related lifecycle operations: Creates, Configuration changes, Restarts, Compute size Updates, etc. Running projects remain running, and the data APIs and direct postgres client connections are NOT affected. Out of an abundance of caution, we would suggest against taking any of these actions until this incident is resolved. We will provide additional updates as we have more information.

3. Identified Sep 4, 12:58 AM UTC

   The team has applied a fix, but error rates remain high. We are continuing to work and will take other necessary steps to resolve this as soon as possible. Running projects remain running, and the data APIs and direct postgres client connections are NOT affected. Out of an abundance of caution, we would suggest against taking any of these actions until this incident is resolved: Creates, Configuration changes, Restarts, or Compute size changes. We will provide additional updates as we have more information.

4. Identified Sep 4, 01:37 AM UTC

   The team is continuing to work on restoring project lifecycle operations. Out of caution, we have disabled lifecycle operations. This is impacting all project-related lifecycle operations: Creates, Configuration changes, Restarts, Compute size Updates, etc. Running projects remain running, and the data APIs and direct postgres client connections are NOT affected. We will provide additional updates as we have more information.

5. Identified Sep 4, 02:02 AM UTC

   We are seeing incremental improvements internally. There is a large backlog of actions that we are working through, but things are progressing. Once we make it through most of the backlog, we will re-enable lifecycle actions. This is impacting all project-related lifecycle operations: Creates, Configuration changes, Restarts, Compute size Updates, etc. Running projects remain running, and the data APIs and direct postgres client connections are NOT affected.

6. Identified Sep 4, 03:01 AM UTC

   Error rates have returned to normal, and the backlogs are draining well. We are beginning to slowly re-enable lifecycle operations for users. We will be monitoring impact, and we hope to have this re-enabled everywhere soon. We will leave this in this state until lifecycle operations have been completely restored. Running projects remain running, and the data APIs and direct postgres client connections are NOT affected.

7. Identified Sep 4, 03:41 AM UTC

   We've increased project lifecycle operations to 50% of users and things are holding steady. We're continuing to ramp this up until operations are restored for all users. This is impacting all project-related lifecycle operations: Creates, Configuration changes, Restarts, Compute size Updates, etc. Running projects remain running, and the data APIs and direct postgres client connections are NOT affected.

8. Monitoring Sep 4, 04:28 AM UTC

   Lifecycle operations have been re-enabled for 100% of users and error rates have returned to normal. We are going to monitor to ensure that things stay stable. The team is working through any remaining projects which did not automatically resolve. Some users may also notice that backups will happen a bit later than usual, but these jobs are running and completing normally.

9. Resolved Sep 4, 08:18 AM UTC

   Lifecycle operations have been re-enabled for 100% of users and error rates have returned to normal.