Incident with Webhooks

Updates timeline

1. Investigating Jun 11, 07:42 PM UTC

   We are investigating reports of degraded performance for Webhooks

2. Investigating Jun 11, 07:42 PM UTC

   We are currently experiencing delays in Web Hook delivery and are actively investigating the root cause.

3. Investigating Jun 11, 07:58 PM UTC

   We have applied a mitigation and are monitoring for recovery.

4. Monitoring Jun 11, 08:00 PM UTC

   The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability.

5. Resolved Jun 11, 10:19 PM UTC

   On June 11, 2026, between 19:28 UTC and 21:06 UTC, GitHub webhook deliveries were delayed. Average delivery latency peaked at approximately 3.4 minutes, with some deliveries delayed by as much as 62 minutes at the 99th percentile. No events were lost — delayed events were queued and delivered once processing caught up.<br /><br />This was due to a change in how webhook traffic was distributed across regions: to relieve load on one region, a portion of processing was shifted to another, where higher latency prevented our delivery workers from keeping pace with incoming volume, creating a backlog. We mitigated the incident by rebalancing webhook traffic distribution; as load returned to normal levels, processing caught up and the delivery backlog fully drained.<br /><br />We are working on improving the accuracy of the utilization metrics used to scale our delivery worker pools, and reassess how we distribute webhook traffic across regions, to reduce our time to detection and mitigation of issues like this one in the future.