Increase in Workflow runs stuck as pending

Updates timeline

1. Investigating Jun 25, 07:45 AM UTC

   We are currently investigating an issue where some recently deployed Workflow projects are stuck as pending and not delivering queued messages as expected.

2. Identified Jun 25, 08:27 AM UTC

   The issue has been identified and a fix is being implemented.

3. Identified Jun 25, 09:25 AM UTC

   The issue has been identified. We are continuing to work on a fix for this issue.

4. Identified Jun 25, 10:42 AM UTC

   We are still working on a fix for this issue. While we fix the issue, users experiencing pending Workflow runs can trigger an instant rollback to a working deployment created before Jun 24 21:00 UTC to resume operation on new runs.

5. Identified Jun 25, 12:15 PM UTC

   We are currently rolling a fix for the issue. We are monitoring the results.

6. Monitoring Jun 25, 01:30 PM UTC

   The fix has been fully rolled out and we are monitoring the results.

7. Resolved Jun 25, 01:43 PM UTC

   This incident has been resolved.