Incident with CodeQL, Webhooks, Notifications, and Slack Integration

Updates timeline

1. Investigating May 12, 02:38 PM UTC

   We are investigating reports of degraded performance for CodeQL

2. Investigating May 12, 03:13 PM UTC

   CodeQL actions are currently experiencing delays, which may result in those actions being stuck in a pending state or having failed due to a timeout.

3. Investigating May 12, 03:42 PM UTC

   We're continuing to investigate issues with CodeQL actions workflows. We're additionally seeing delays for notifications, webhooks, and the Slack integration.

4. Investigating May 12, 03:44 PM UTC

   Webhooks is experiencing degraded performance. We are continuing to investigate.

5. Investigating May 12, 04:18 PM UTC

   We've established that most delays are related to a queuing service and are working to scale out. Early signals from the scale-out are showing signs of recovery for some services. We'll provide an update when services are fully recovered.

6. Investigating May 12, 04:28 PM UTC

   Webhooks is operating normally.

7. Investigating May 12, 04:29 PM UTC

   Webhooks have fully recovered. Continuing to work on recovery for the other services.

8. Investigating May 12, 04:59 PM UTC

   CodeQL has fully recovered. We're continuing to work on recovery for the remaining impacted services.

9. Investigating May 12, 05:43 PM UTC

   All services have fully recovered.

10. Resolved May 12, 05:43 PM UTC

    This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.