The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers
Mirrored from arXiv — Machine Learning for archival readability. Support the source by reading on the original site.
Computer Science > Machine Learning
Title:The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers
Abstract:LLM routing has become a popular approach to improve the cost-quality trade-off of LLM services by dynamically selecting a model for each query. Recent work has explored a broad range of routing methods, including clustering-based routers, learned classifiers, pairwise ranking, and confidence-based approaches. Our extensive study of 21 routing methods across five benchmarks reveals a consistent phenomenon that we call the routing plateau: many methods, including kNN, achieve very similar accuracy and converge to a narrow performance range that remains far below the oracle router. Our investigation shows that the plateau is largely caused by a predictability bottleneck: current routers mainly learn global averaged model-performance trends rather than fine-grained query-specific routing signals. As a result, they solve overlapping easy queries but collectively fail on hard queries that require instance-specific routing decisions. We further study how to move beyond the plateau and find that larger training datasets, stronger encoders, and end-to-end fine-tuning can further improve routing accuracy. These findings characterize the common limits of current routing methods and provide insights and actionable directions for the community to build more effective routing systems.
| Comments: | 23 Pages, 12 Tables, 9 Figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.07587 [cs.LG] |
| (or arXiv:2606.07587v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.07587
arXiv-issued DOI via DataCite
|
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
References & Citations
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
More from arXiv — Machine Learning
-
Offline Reinforcement Learning for Plasma Control in Nuclear Fusion: Codebase and Benchmark
Jun 9
-
MedicalRec: Medical recommender system for image classification without retraining
Jun 9
-
SPIN: Decentralized Swarm Control via Tensorized Policy Coordination
Jun 9
-
Boundary Variance Inflation Causes Acquisition Bias in Gaussian Processes
Jun 9
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.