Authentication issues related to API requests

Updates timeline

1. Investigating Jun 10, 03:20 PM UTC

   We are investigating reports of impacted performance for some GitHub services.

2. Investigating Jun 10, 03:23 PM UTC

   API Requests is experiencing degraded availability. We are continuing to investigate.

3. Investigating Jun 10, 03:27 PM UTC

   Issues is experiencing degraded performance. We are continuing to investigate.

4. Investigating Jun 10, 03:27 PM UTC

   We are investigating issues related to sporadic authentication failures impacting approximately 15% of API traffic. We will continue to investigate and provide updates.

5. Investigating Jun 10, 03:46 PM UTC

   We continue to investigate issues related to sporadic authentication failures, impacting approximately 15% of API traffic. Further updates will be provided as we work to mitigate.

6. Investigating Jun 10, 03:46 PM UTC

   The degradation affecting Issues has been mitigated. We are monitoring to ensure stability.

7. Investigating Jun 10, 04:21 PM UTC

   We continue to investigate issues related to sporadic authentication failures, impacting approximately 15% of API traffic. Erroneous 401 responses are causing app integrations to trigger authentication flows. We have identified a problematic component in our infrastructure and are working to mitigate.

8. Investigating Jun 10, 04:36 PM UTC

   The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability.

9. Monitoring Jun 10, 04:37 PM UTC

   The degradation has been mitigated. We are monitoring to ensure stability.

10. Resolved Jun 10, 04:39 PM UTC

    Between 15:05 UTC and 16:25 UTC, GitHub API services experienced degraded availability due to sporadic authentication failures affecting approximately 9% of requests. Customers experienced intermittent "logged out" behavior as erroneous 401 responses triggered repeated authentication flows in app integrations. Affected requests also experienced approximately 800ms of additional latency. <br /><br />A memcached proxy service rollout to our internal API infrastructure caused our authentication service to pick up an incorrect memcached host configuration, leading to intermittent authentication lookup failures. We mitigated the incident by deploying a configuration change to memcached to use the correct host. <br /><br />To prevent similar issues in the future, we plan to migrate our authentication system to the new memcached infrastructure to improve resilience and strengthen overall reliability posture.