Latency issues across a number of services

Updates timeline

1. Investigating Jul 23, 07:53 AM UTC

   We are investigating reports of degraded availability for Actions, Issues and Webhooks

2. Investigating Jul 23, 08:25 AM UTC

   Pull Requests is experiencing degraded performance. We are continuing to investigate.

3. Investigating Jul 23, 08:34 AM UTC

   We're currently investigating latency across multiple services. This can show as Actions jobs taking longer to start, Issues search serving stale results, and other listed services being similarly impacted.

4. Investigating Jul 23, 09:18 AM UTC

   The degradation affecting Issues has been mitigated. We are monitoring to ensure stability.

5. Investigating Jul 23, 09:19 AM UTC

   The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.

6. Investigating Jul 23, 09:22 AM UTC

   We identified the source of latency affecting multiple services and applied a fix. Issues and Actions are recovering, and remaining affected services are seeing improvement as processing backlogs clear. We are actively monitoring recovery across all services.

7. Investigating Jul 23, 09:27 AM UTC

   The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability.

8. Investigating Jul 23, 09:35 AM UTC

   Webhooks is operating normally.

9. Monitoring Jul 23, 09:39 AM UTC

   The degradation has been mitigated. We are monitoring to ensure stability.

10. Resolved Jul 23, 09:39 AM UTC

    On July 23, 2026, between 07:08 and 09:39 UTC, several services experienced delays: 8% of actions workflow runs experienced an average run start delay of 10 minutes, 5% of webhook deliveries exceeded SLO, and code scanning, repos, notifications, issues and pull requests experienced increased latency over the life of the incident. <br /><br />The root cause of the incident was a node of our background job processing system which did not recover after entering scheduled host maintenance. The incident was mitigated by identifying the problematic shard and restoring its correct state, after which queue backlogs drained and services recovered. <br /><br />To speed mitigation, we have added monitors for nodes in this unhealthy state after maintenance operations. To prevent future recurrence, we are adapting our lifecycle automation to verify host rejoin after a scheduled reboot.