Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-p routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present FVattn, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. FVattn uses Top-p routing, a Top-k safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, FVattn reduces average load imbalance from 1.34 to 1.08 and delivers a 4.41× attention speedup over FlashAttention, while achieving a 2.02--2.11× DiT inference speedup with competitive video quality.</p>\n","updatedAt":"2026-07-23T05:05:57.508Z","author":{"_id":"665c91e15b11dca02f0c5891","avatarUrl":"/avatars/49a4ee76c3edfe5b0916051a5ac4acfd.svg","fullname":"Ye Huang","name":"henry-y1","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false,"primaryOrg":{"avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/665c91e15b11dca02f0c5891/ek9KGIc02tiFfaYeDfLaU.png","fullname":"DAGroup-PKU","name":"DAGroup-PKU","type":"org","isHf":false}}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8018568158149719},"editors":["henry-y1"],"editorAvatarUrls":["/avatars/49a4ee76c3edfe5b0916051a5ac4acfd.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.16190","authors":[{"_id":"6a5d923f6a69ce099f4d6e0e","name":"Hao Liu","hidden":false},{"_id":"6a5d923f6a69ce099f4d6e0f","name":"Chenghuan Huang","hidden":false},{"_id":"6a5d923f6a69ce099f4d6e10","name":"Ye Huang","hidden":false},{"_id":"6a5d923f6a69ce099f4d6e11","name":"Zhiying Wen","hidden":false},{"_id":"6a5d923f6a69ce099f4d6e12","name":"Hao Liu","hidden":false},{"_id":"6a5d923f6a69ce099f4d6e13","name":"Mohan Zhang","hidden":false},{"_id":"6a5d923f6a69ce099f4d6e14","name":"Chen Li","hidden":false},{"_id":"6a5d923f6a69ce099f4d6e15","name":"Ziyang Ma","hidden":false},{"_id":"6a5d923f6a69ce099f4d6e16","name":"Jing Lyu","hidden":false},{"_id":"6a5d923f6a69ce099f4d6e17","name":"Jiangsu Du","hidden":false}],"publishedAt":"2026-07-17T00:00:00.000Z","submittedOnDailyAt":"2026-07-23T00:00:00.000Z","title":"FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation","submittedOnDailyBy":{"_id":"665c91e15b11dca02f0c5891","avatarUrl":"/avatars/49a4ee76c3edfe5b0916051a5ac4acfd.svg","isPro":false,"fullname":"Ye Huang","user":"henry-y1","type":"user","name":"henry-y1"},"summary":"Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-p routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present , a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. uses Top-p routing, a Top-k safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, reduces average load imbalance from 1.34 to 1.08 and delivers a 4.41times attention speedup over FlashAttention, while achieving a 2.02--2.11times DiT inference speedup with competitive video quality.","upvotes":4,"discussionId":"6a5d923f6a69ce099f4d6e18"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"665c91e15b11dca02f0c5891","avatarUrl":"/avatars/49a4ee76c3edfe5b0916051a5ac4acfd.svg","isPro":false,"fullname":"Ye Huang","user":"henry-y1","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6915acbb907428dbc99519fe","avatarUrl":"/avatars/860a461ffa45b8ec8c969fb2428f9fee.svg","isPro":false,"fullname":"aaa","user":"yhrbb02","type":"user"},{"_id":"680350a7cd4782848c1ae70c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ysmIY9FK3q58rPMwJMdiU.jpeg","isPro":false,"fullname":"GarvinRay","user":"GarvinRay","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.16190.md","query":{}}">
FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
Abstract
Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-p routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present , a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. uses Top-p routing, a Top-k safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, reduces average load imbalance from 1.34 to 1.08 and delivers a 4.41times attention speedup over FlashAttention, while achieving a 2.02--2.11times DiT inference speedup with competitive video quality.
Community
Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-p routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present FVattn, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. FVattn uses Top-p routing, a Top-k safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, FVattn reduces average load imbalance from 1.34 to 1.08 and delivers a 4.41× attention speedup over FlashAttention, while achieving a 2.02--2.11× DiT inference speedup with competitive video quality.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.16190 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.16190 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.16190 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.