We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at <a href=\"https://github.com/shengshu-ai/TurboServe\" rel=\"nofollow\">https://github.com/shengshu-ai/TurboServe</a>.</p>\n","updatedAt":"2026-07-02T03:12:02.835Z","author":{"_id":"66c0a08bac74db25de8427ec","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg","fullname":"Jintao Zhang","name":"jt-zhang","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":53,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8727968335151672},"editors":["jt-zhang"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.19271","authors":[{"_id":"6a45d65f4f1dd35e48fb8f3a","name":"Youhe Jiang","hidden":false},{"_id":"6a45d65f4f1dd35e48fb8f3b","name":"Haoxu Wang","hidden":false},{"_id":"6a45d65f4f1dd35e48fb8f3c","name":"Haotong Bao","hidden":false},{"_id":"6a45d65f4f1dd35e48fb8f3d","name":"Kai Jiang","hidden":false},{"_id":"6a45d65f4f1dd35e48fb8f3e","name":"Jianfei Chen","hidden":false},{"_id":"6a45d65f4f1dd35e48fb8f3f","name":"Jun Zhu","hidden":false},{"_id":"6a45d65f4f1dd35e48fb8f40","name":"Fangcheng Fu","hidden":false},{"_id":"6a45d65f4f1dd35e48fb8f41","name":"Jintao Zhang","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/66c0a08bac74db25de8427ec/KHBp7rjcUIsLCgUSwHGJ6.png"],"publishedAt":"2026-06-17T00:00:00.000Z","submittedOnDailyAt":"2026-07-02T00:00:00.000Z","title":"TurboServe: Serving Streaming Video Generation Efficiently and Economically","submittedOnDailyBy":{"_id":"66c0a08bac74db25de8427ec","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg","isPro":false,"fullname":"Jintao Zhang","user":"jt-zhang","type":"user","name":"jt-zhang"},"summary":"Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving, streaming video generation must preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods.\n We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.","upvotes":19,"discussionId":"6a45d6604f1dd35e48fb8f42","projectPage":"https://github.com/shengshu-ai/TurboServe","githubRepo":"https://github.com/shengshu-ai/TurboServe","githubRepoAddedBy":"user","ai_summary":"TurboServe is a specialized serving system for streaming video generation that addresses session state management and dynamic resource allocation challenges through integrated scheduling, autoscaling, and migration mechanisms.","ai_keywords":["streaming video generation","session state preservation","online scheduling","GPU provisioning","closed-loop scheduling","migration-aware placement","load-driven autoscaling","coalesced chunk processing","GPU-CPU offloading","NCCL-based GPU-GPU migration"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":14,"organization":{"_id":"628735cbc83a2d6ab8d14a66","name":"Tsinghua","fullname":"Tsinghua University","avatar":"https://www.gravatar.com/avatar/6c5c1441e3283e7543342e59277ea219?d=retro&size=100"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66c0a08bac74db25de8427ec","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg","isPro":false,"fullname":"Jintao Zhang","user":"jt-zhang","type":"user"},{"_id":"64f3473a8c2ba61ee2155ca8","avatarUrl":"/avatars/afbba885d0b4a8c19c0ab8f1f34e0f89.svg","isPro":false,"fullname":"Yan Ran","user":"yanonhf","type":"user"},{"_id":"67052ac55c89251d79ebef91","avatarUrl":"/avatars/d65c44b4d17a33fd6cbc8461fb402ad4.svg","isPro":false,"fullname":"tsingte","user":"thu-zzte","type":"user"},{"_id":"67be8630e7b05f9e43b21204","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/c7jrbuqdaZfWZlj1ASJoN.png","isPro":false,"fullname":"ice","user":"akjhnh","type":"user"},{"_id":"6686c943ffcabf23d278412d","avatarUrl":"/avatars/2f2254c7926b3eaaaa0103573172e5a9.svg","isPro":false,"fullname":"ht","user":"vlgunsdaddy","type":"user"},{"_id":"66ac56c4e77d184cb95bf2ec","avatarUrl":"/avatars/39dd49e2d29eb93299836d6cdc93cfa2.svg","isPro":false,"fullname":"Fangcheng Fu","user":"ccchengff","type":"user"},{"_id":"646b24e3e96a751c52507ab1","avatarUrl":"/avatars/f93ec526aa0eb0d2253d65cab9612520.svg","isPro":false,"fullname":"Shi Jiahao","user":"Shijiahao314","type":"user"},{"_id":"634fe6d217a6475e8bfd0950","avatarUrl":"/avatars/f7add01c39ca428f8ce6ac198f49a1fa.svg","isPro":false,"fullname":"jiang kai","user":"aaa-ceku7","type":"user"},{"_id":"66137d19fa38433529791c76","avatarUrl":"/avatars/7b2d1f1e3b9f2cbac614fb685b40f490.svg","isPro":false,"fullname":"zz","user":"yzzer","type":"user"},{"_id":"6790d471ccf138a92c65f7c8","avatarUrl":"/avatars/10d5c0ccf6a0366962f9748aafdd3c95.svg","isPro":false,"fullname":"travis lee","user":"travismtg","type":"user"},{"_id":"643383946c2a26ae66d79d38","avatarUrl":"/avatars/5a613598a1b318387ae02f0f76e89b08.svg","isPro":false,"fullname":"Subprime7374","user":"Subprime7374","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":2,"organization":{"_id":"628735cbc83a2d6ab8d14a66","name":"Tsinghua","fullname":"Tsinghua University","avatar":"https://www.gravatar.com/avatar/6c5c1441e3283e7543342e59277ea219?d=retro&size=100"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.19271.md","query":{}}">
TurboServe: Serving Streaming Video Generation Efficiently and Economically
Abstract
TurboServe is a specialized serving system for streaming video generation that addresses session state management and dynamic resource allocation challenges through integrated scheduling, autoscaling, and migration mechanisms.
Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving, streaming video generation must preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods.
We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.
Community
We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.19271 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.19271 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.19271 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.