Increased Error Rates

Updates timeline

1. Investigating Jun 22, 02:20 PM UTC

   Cloudflare is investigating increased error rates and latency in multiple services beginning 13:35 UTC.

2. Identified Jun 22, 02:37 PM UTC

   The issue has been identified and a fix is being implemented.

3. Identified Jun 22, 02:44 PM UTC

   We are continuing to work on a fix for this issue.

4. Identified Jun 22, 02:47 PM UTC

   We are continuing to work on a fix for this issue.

5. Identified Jun 22, 02:48 PM UTC

   We are investigating a fiber cut experienced by a third party network provider in Eastern North America. Customers connecting through North America or accessing services in Europe may see increased latencies and timeouts as Cloudflare engineers look to mitigate.

6. Identified Jun 22, 03:12 PM UTC

   We are continuing our traffic engineering work to mitigate the impact.

7. Identified Jun 22, 03:38 PM UTC

   Our network teams are continuing active traffic engineering efforts to reroute traffic away from congested links and restore full service stability. We are closely monitoring dashboards and customer reports to confirm recovery and will continue to provide status updates as we manage the remaining load.

8. Identified Jun 22, 03:40 PM UTC

   Our Engineers are continuing our traffic engineering work to mitigate the impact.

9. Identified Jun 22, 04:30 PM UTC

   Traffic engineering efforts have successfully mitigated the majority of congestion and packet drops. Services are largely stable, with only minor residual impact remaining. A small number of intermittent errors may still be observed for services with origins in North America as we work to fully clear the remaining load.

10. Monitoring Jun 22, 04:50 PM UTC

    Traffic engineering efforts have successfully mitigated the majority of congestion and packet drops.

11. Resolved Jun 22, 09:22 PM UTC

    This incident has been resolved.