Incident with Webhooks

Updates timeline

1. Investigating Aug 13, 02:45 PM UTC

   We are investigating reports of degraded performance for Webhooks

2. Investigating Aug 13, 02:46 PM UTC

   Pull Requests is experiencing degraded performance. We are continuing to investigate.

3. Investigating Aug 13, 02:46 PM UTC

   Issues is experiencing degraded performance. We are continuing to investigate.

4. Investigating Aug 13, 02:46 PM UTC

   Git Operations is experiencing degraded performance. We are continuing to investigate.

5. Investigating Aug 13, 02:56 PM UTC

   Packages is experiencing degraded performance. We are continuing to investigate.

6. Investigating Aug 13, 02:58 PM UTC

   We are currently investigating a brief degradation of service for Git operations (specifically pushes), issues, pull requests, package registry, and webhooks between 14:32 and 14:46 UTC. We have identified the source of the degradation and are investigating mitigation strategies to prevent recurrence.

7. Monitoring Aug 13, 03:33 PM UTC

   The degradation affecting Git Operations, Issues, Packages, Pull Requests and Webhooks has been mitigated. We are monitoring to ensure stability.

8. Monitoring Aug 13, 03:33 PM UTC

   We have temporarily disabled a background job which caused the impact. At this time the impact is fully mitigated.

9. Resolved Aug 13, 03:36 PM UTC

   This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.