Railway Blocked by Google Cloud
Mirrored from Hacker News — AI on Front Page for archival readability. Support the source by reading on the original site.
We are investigating a widespread service disruption affecting Railway. Users may be experiencing errors including "no healthy upstream", "unconditional drop overload", login failures, and inability to access the dashboard. We are actively investigating and will provide updates as soon as possible.
We have identified the cause of the disruption. Access to our upstream cloud provider has been restored and we are working on a fix. The Railway dashboard and services running on our cloud infrastructure may continue to be affected until the fix is deployed.
Google Cloud has blocked our account, making some Railway services unavailable. We have escalated this directly with Google. The Railway Platform team has since confirmed access to Google Cloud and is working on restoring access to all workloads. We have access to some of our Google Cloud–hosted infrastructure and are working to restore the rest of the service. We apologize for the disruption.
We are working to restore the Google Cloud infrastructure that powers our dashboard, API, and internal network's control plane. We are in direct contact with Google Cloud's support team. We do not have an ETA at this time.
We are continuing to work on restoring access to our workloads on Google Cloud. In parallel, our infrastructure team is evaluating alternative paths to bring affected services back online. We do not have an ETA at this time and will continue posting updates as we have them.
We have recovered our compute on Google Cloud, but services are unable to start because of ongoing networking issues on Google Cloud's side. We are engaged with Google Cloud support to resolve this and will post the next update as soon as we have more to share.
We are seeing gradual recovery on Railway metal workloads. To ensure things remain stable as we ramp back up, we are temporarily throttling all non-enterprise builds to avoid overwhelming our build infrastructure.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.