Elevated error rate across multiple models

Updates timeline

1. Investigating Jun 23, 02:19 PM UTC

   We are currently investigating this issue.

2. Identified Jun 23, 02:25 PM UTC

   The issue has been identified and a fix is being implemented.

3. Monitoring Jun 23, 02:53 PM UTC

   A fix has been implemented and we are monitoring the results.

4. Monitoring Jun 23, 03:28 PM UTC

   We are continuing to monitor for any further issues.

5. Monitoring Jun 23, 04:05 PM UTC

   We have seen error rates broadly return to normal success levels across all models. We saw elevated error rates on requests to multiple models from 7:08am PT / 14:08 UTC through 8:33am PT / 15:33 UTC.

6. Resolved Jun 23, 04:44 PM UTC

   This incident has been resolved.