HuggingFace download issues

Updates timeline

1. Investigating Jul 16, 08:31 AM UTC

   We are aware of 504s being returned by models that reach out to HuggingFace during setup. We believe this is likely related to the disruption visible at [https://status.huggingface.co/](https://status.huggingface.co/ "https://status.huggingface.co/"). We are monitoring and will update when we can identify the HuggingFace CDN is back up/

2. Monitoring Jul 16, 09:10 AM UTC

   HuggingFace is reporting recovery, which we are also seeing on our end. However, we still see minor degradation, which we believe is due to thundering herd once the main issue was resolved. We are still keeping an eye out, but believe we are recovering

3. Monitoring Jul 16, 09:28 AM UTC

   We have seen most HuggingFace calls start succeeding, except for a small subset that are being downloaded through [cas-bridge.xethub.hf.co](http://cas-bridge.xethub.hf.co "cas-bridge.xethub.hf.co"). Most models are now succeeding to spin up

4. Monitoring Jul 16, 09:46 AM UTC

   Still seeing signs of improvement, most models are able to spin up with a few still waiting their turn from HuggingFace. Models should spin up after enough tries if they are failing for HuggingFace related reasons.

5. Resolved Jul 16, 04:04 PM UTC

   We have not seen elevated rates of HuggingFace/model setup errors for a couple hours now, so we believe this incident is cleared