Project Actions Failing Across Multiple Regions

Updates timeline

1. Investigating Jul 17, 05:49 PM UTC

   Certain project actions are failing. We are actively investigating to determine the regions impacted and source of the issue.

2. Investigating Jul 17, 06:16 PM UTC

   Project actions in us-east-1, us-east-2, sa-east-1, ca-central-1 are experiencing significant increases in error rates. We are exploring the best options for restoring performance.

3. Investigating Jul 17, 06:47 PM UTC

   We have determined that there are four main project actions impacted in the identified regions: Pause/unpause failures, Restore/PITR failures, PITR configuration failures and Backup problems. We are pursuing root cause and mitigation options in parallel.

4. Investigating Jul 17, 07:20 PM UTC

   We are testing mitigation options to attempt to restore project action performance.

5. Identified Jul 17, 07:45 PM UTC

   We have identified a problem with a specific job in us-east-1, we have mitigated the job and are observing the results.

6. Monitoring Jul 17, 08:21 PM UTC

   The mitigation has returned project activity to normal performance. Project actions that failed during the outage are being requeued to run. We will continue to monitor to ensure project functionality.

7. Resolved Jul 18, 01:41 AM UTC

   Project actions have been sustained at normal performance. This incident is resolved.