Elevated errors across multiple services in IAD1 (Washington, D.C., USA)

Updates timeline

1. Identified May 8, 01:18 AM UTC

   Some Vercel functions that run in the IAD1 region are experiencing elevated invocation failures. We are investigating the issue and will share more information as it becomes available.

2. Identified May 8, 01:47 AM UTC

   Traffic to the IAD1 region has been re-routed to nearby regions. Functions configured in IAD1 will be invoked in a different region if failover regions are configured, but if no other regions are configured, the function will still be invoked in IAD1.

3. Identified May 8, 02:13 AM UTC

   New messages to Vercel Workflows and Vercel Queues are being queued, but message processing is paused. Queued messages will be processed when service is restored.

4. Identified May 8, 02:18 AM UTC

   We are continuing to work on a fix for this issue.

5. Identified May 8, 02:33 AM UTC

   New workflow runs and queue messages are being processed now. Processing the backlog of existing workflow runs and queued messages is still being investigated.

6. Monitoring May 8, 04:54 AM UTC

   We implemented a fix and are monitoring the results. Backlogs for Workflows and Queues will be processed within the next few hours.

7. Resolved May 8, 06:39 AM UTC

   We've deployed mitigations and all services are operating normally. Traffic may continue to reroute to nearby regions for the next few hours, but we will restore traffic gradually after verifying system availability in the IAD1 region.