In load-balancing for LLM inference, TTFT consists of three distinct costs: network latency, prefill time, and queueing delay. While cache-aware policies aim to reuse existing KV-cache, these policies cannot balance load effectively, causing E2E latency to collapse. GORGO jointly optimizes all three routing signals using a weighted cost function. During online tuning on a long-context user workload, the GORGO policy shows a 6-30% reduction in TTFT and E2E latency over existing policies such as simple session affinity.</p>\n","updatedAt":"2026-07-07T04:41:08.801Z","author":{"_id":"67a53de25e65aa63c6fa4a63","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67a53de25e65aa63c6fa4a63/uesLQAm2TkegYri6ZFeT2.jpeg","fullname":"Alessio Toniolo","name":"alessiotoniolo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8959694504737854},"editors":["alessiotoniolo"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/67a53de25e65aa63c6fa4a63/uesLQAm2TkegYri6ZFeT2.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2602.11688","authors":[{"_id":"6a455e554f1dd35e48fb8dc2","user":{"_id":"67a53de25e65aa63c6fa4a63","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67a53de25e65aa63c6fa4a63/uesLQAm2TkegYri6ZFeT2.jpeg","isPro":false,"fullname":"Alessio Toniolo","user":"alessiotoniolo","type":"user","name":"alessiotoniolo"},"name":"Alessio Ricci Toniolo","status":"claimed_verified","statusLastChangedAt":"2026-07-05T21:14:27.388Z","hidden":false},{"_id":"6a455e554f1dd35e48fb8dc3","user":{"_id":"68642ef25d2f5330dba29f63","avatarUrl":"/avatars/f7812592fc58b37cb48fdfcfef37fdc9.svg","isPro":false,"fullname":"Rome Thorstenson","user":"Rome-1","type":"user","name":"Rome-1"},"name":"Rome Thorstenson","status":"claimed_verified","statusLastChangedAt":"2026-07-05T21:14:30.709Z","hidden":false},{"_id":"6a455e554f1dd35e48fb8dc4","name":"Abinaya Dinesh","hidden":false}],"publishedAt":"2026-06-30T00:00:00.000Z","submittedOnDailyAt":"2026-07-07T00:00:00.000Z","title":"GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving","submittedOnDailyBy":{"_id":"67a53de25e65aa63c6fa4a63","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67a53de25e65aa63c6fa4a63/uesLQAm2TkegYri6ZFeT2.jpeg","isPro":false,"fullname":"Alessio Toniolo","user":"alessiotoniolo","type":"user","name":"alessiotoniolo"},"summary":"Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improve p95 TTFT by 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baseline load-balancing policies such as simple session affinity and prefix-cache. The code and ART-Chat-2.5M dataset can be found at https://github.com/Arcadia-Research-Team/GORGO.","upvotes":4,"discussionId":"6a455e554f1dd35e48fb8dc5","githubRepo":"https://github.com/atoniolo76/GORGO","githubRepoAddedBy":"user","ai_summary":"GORGO is a proxy architecture that optimizes LLM inference load balancing by jointly considering network latency, prefill cost, and queueing delay through evolutionary strategy tuning on a new synthetic dataset.","ai_keywords":["LLM inference services","load-balancing policies","KV-cache locality","replica load","network latency","prefill cost","queueing delay","evolutionary strategies","p95 TTFT","p95 end-to-end latency","session affinity","prefix-cache"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":4},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67a53de25e65aa63c6fa4a63","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67a53de25e65aa63c6fa4a63/uesLQAm2TkegYri6ZFeT2.jpeg","isPro":false,"fullname":"Alessio Toniolo","user":"alessiotoniolo","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6a146b3bb28ec6a2ad91c059","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_Pj08NpcHNc5x6aL8ng8T.png","isPro":false,"fullname":"Scarlett King","user":"ScarlettKing","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2602/2602.11688.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2602.11688 in a model README.md to link it from this page.

Cite arxiv.org/abs/2602.11688 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.