Intermittent latency in Eastern US

Updates timeline

1. Identified Sep 29, 04:26 PM UTC

   We have identified an issue with increased latency for clients in the eastern US during spikes caused by bursty traffic. This is most noticeable during EDT working hours around the :00 and :30 hour marks. We are actively working on a fix. We will provide updates as the fix is implemented.

2. Identified Sep 29, 05:55 PM UTC

   We are continuing our work to fix this latency issue, which has been observed to impact clients connecting to Supabase Projects from the eastern United States. These clients may include servers and serverless functions located in that geographical region, regardless of the region in which the Project is hosted.

3. Identified Sep 29, 09:35 PM UTC

   A mitigation for this issue was applied earlier today, but we do not have sufficient evidence to conclude that the mitigation resulted in improvement to this eastern US latency issue. We have tied the issue to a particular PoP, and we are continuing to work with our networking partners to bring this issue to a final resolution. We will provide another update within 4 hours, if not before. We appreciate your patience.

4. Identified Sep 30, 02:14 AM UTC

   We are working with our networking partners on a permanent fix. While we do not have an estimated resolution time, we will share updates here as they become available.

5. Identified Sep 30, 03:03 PM UTC

   We're continuing to coordinate with our networking partners to deliver a permanent resolution. We will provide updates as soon as they're available.

6. Identified Sep 30, 09:26 PM UTC

   We are seeing signs of improvement and are continuing to work alongside our networking partners towards a resolution. We will provide another update as more information becomes available.

7. Identified Oct 1, 08:23 PM UTC

   Our networking partners have implemented a mitigation to the latency issue at the problematic PoP in the Eastern US, and we have since observed significant improvement in rates of high latency across the region. We are continuing to work with those partners to isolate any additional root causes of latency and improve the reliability of our networking infrastructure across the region.

8. Identified Oct 2, 09:06 PM UTC

   While we have observed significant improvement in rates of high latency, we note that some users are still experiencing issues during peak Eastern US business hours. We continue to work with our networking partners to isolate and resolve any lingering sources of high latency. We recognize that this is a longstanding issue and appreciate the patience of affected customers. We will continue to share additional updates here as soon as they are available.

9. Monitoring Oct 5, 06:45 PM UTC

   A fix for this issue has been implemented and our data shows recovery for users on projects that have previously been affected. We expect this situation to be resolved, but will keep monitoring for a while until fully resolving this incident.

10. Resolved Oct 6, 12:15 PM UTC

    A fix for this issue has been implemented and our data shows previous impacted users have recovered.