Upgrade Failures

Updates timeline

1. Investigating Jun 25, 02:20 PM UTC

   We are investigating potential upgrade issues in multiple regions. We are working to confirm impact and determine next steps.

2. Investigating Jun 25, 02:41 PM UTC

   Upgrades are taking longer than usual, but generally will complete if allowed to proceed. We are investigating the cause of the delays.

3. Identified Jun 25, 03:01 PM UTC

   The upgrade delays have been caused by resource capacity limits. We are taking actions to increase the appropriate resources.

4. Identified Jun 25, 03:32 PM UTC

   We are aware that upgrades continue to be delayed. We are exploring other factors that are contributing to the slow upgrades.

5. Identified Jun 25, 04:03 PM UTC

   We are continuing to work through a variety of options for improving upgrade performance. This includes working with our upstream provider and adjusting operational parameters.

6. Identified Jun 25, 04:27 PM UTC

   We are implementing operational mitigations to try to improve upgrade performance.

7. Identified Jun 25, 05:03 PM UTC

   We are continuing our operational and investigative work to improve upgrade flow performance and mitigate further problems.

8. Identified Jun 25, 05:46 PM UTC

   We are starting to notice improvements as a result of our mitigation efforts, and expect that project upgrades should start to become less delayed. We’ll continue monitoring and will update as we observe improvements across the fleet.

9. Monitoring Jun 25, 06:08 PM UTC

   We have implemented all necessary mitigations and the project upgrade rates have returned to normal across the fleet. We will continue to monitor these flows for any irregularities or recurring degradation. If you are still experiencing delayed or blocked upgrades on your Project, please contact Support.

10. Resolved Jun 25, 06:51 PM UTC

    Through our monitoring, we have determined that project upgrade behavior has returned to normal across the fleet. This incident is resolved.