Disruption with some GitHub services

Updates timeline

1. Investigating Jul 24, 04:17 PM UTC

   We are investigating reports of degraded performance for API Requests and Issues

2. Investigating Jul 24, 04:19 PM UTC

   Actions is experiencing degraded performance. We are continuing to investigate.

3. Investigating Jul 24, 04:20 PM UTC

   Pull Requests is experiencing degraded performance. We are continuing to investigate.

4. Investigating Jul 24, 04:22 PM UTC

   We are investigating timeouts to some GitHub services

5. Investigating Jul 24, 04:26 PM UTC

   Copilot is experiencing degraded performance. We are continuing to investigate.

6. Investigating Jul 24, 04:27 PM UTC

   Pages is experiencing degraded performance. We are continuing to investigate.

7. Investigating Jul 24, 04:28 PM UTC

   Actions is experiencing degraded availability. We are continuing to investigate.

8. Investigating Jul 24, 04:40 PM UTC

   We have applied a mitigation and are monitoring for recovery

9. Investigating Jul 24, 04:41 PM UTC

   Actions is experiencing degraded performance. We are continuing to investigate.

10. Investigating Jul 24, 05:16 PM UTC

    The degradation affecting API Requests, Actions, Copilot, Issues, Pages and Pull Requests has been mitigated. We are monitoring to ensure stability.

11. Investigating Jul 24, 05:24 PM UTC

    We are seeing recovery across all services

12. Resolved Jul 24, 05:36 PM UTC

    On July 24th at 16:04 UTC, a loss of connectivity occurred in network paths in one of our three physical data center availability zones (AZs). This resulted in packet loss due to the remaining active paths becoming saturated. Our data centers use a leaf-spine switch fabric in each compute cage, and an aggregation layer interconnecting the spines from each cage within each AZ. The loss of connectivity affected links between one cage’s spine switches and the aggregation layer within that specific AZ. <br /><br />Workloads depending on compute resources in this cage became degraded due to packet loss, and exhibited intermittent errors: <br /><br />- Actions saw 10% of jobs fail during the impact window, and 5% of jobs succeeded but with delayed starts. <br />- 27% of GitHub issues interactions saw slow requests or timeouts. <br />- 4% of GitHub Copilot requests experienced errors, though most automatically retry. <br />- 4% of git push operations saw impacts during the affected window. <br />- Authentication requests saw increased latency during the affected window, but error rates, while elevated, were < 1% in all cases. <br /><br />We were able to mitigate the outage by re-routing affected connections to available fiber paths that were allocated for future capacity upgrades. Sufficient network capacity to eliminate packet loss was restored at 17:07, with most services showing full recovery by 17:16. All paths were restored and services healthy at 17:36. <br /><br />This incident affected 25% of available network interconnect capacity. Older cages utilize a 100Gbps network interface standard. To remove risk of reoccurrence, a planned upgrade to 400Gbps interfaces is being accelerated as much as possible, ensuring increased bandwidth available at all layers of the switch fabric for resiliency to path or device loss.