Pipeline parallelism is essential for training large neural networks, but existing schedules trade off throughput, memory, and optimization consistency. Synchronous pipelines preserve forward/backward weight consistency but suffer from bubbles; asynchronous pipelines remove bubbles but introduce weight-version mismatch, typically requiring weight stashing, prediction, or correction mechanisms. We introduce PACI (Pipeline Asynchronous training with Controlled Inconsistency), a bubble-free asynchronous pipeline method that bounds forward/backward version drift without weight stashing, prediction, additional parameter copies, or global synchronization. The key idea is to use local gradient accumulation as a version-control mechanism: by slowing parameter-version evolution relative to pipeline delay, PACI limits the number of optimizer updates crossed by any micro-batch while preserving steady-state utilization. In GPT-style language-model pretraining, PACI matches the stability and final perplexity of synchronous 1F1B-flush, retains the same peak memory footprint, achieves fully utilized pipeline throughput, and improves training time-to-accuracy by up to 1.69x over the fastest flush baseline. These results show that forward/backward inconsistency need not be eliminated: when explicitly bounded, it can be safely traded for substantial efficiency gains.</p>\n","updatedAt":"2026-06-11T09:01:22.609Z","author":{"_id":"64ea60c85ba66cfe777e84fc","avatarUrl":"/avatars/75c721b2a45e4f64d29d35f9cc7f7643.svg","fullname":"itay elam","name":"ItayElam","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8339810967445374},"editors":["ItayElam"],"editorAvatarUrls":["/avatars/75c721b2a45e4f64d29d35f9cc7f7643.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.07881","authors":[{"_id":"6a282fa3e7d78ea7587e5072","user":{"_id":"64ea60c85ba66cfe777e84fc","avatarUrl":"/avatars/75c721b2a45e4f64d29d35f9cc7f7643.svg","isPro":false,"fullname":"itay elam","user":"ItayElam","type":"user","name":"ItayElam"},"name":"Itay Elam","status":"claimed_verified","statusLastChangedAt":"2026-06-11T08:46:24.816Z","hidden":false},{"_id":"6a282fa3e7d78ea7587e5073","user":{"_id":"67b9e1ba9734000097f005e5","avatarUrl":"/avatars/5c9969abe1ae7818a30c3936e9c662e8.svg","isPro":false,"fullname":"Eliron Rahimi","user":"ElironRahimi","type":"user","name":"ElironRahimi"},"name":"Eliron Rahimi","status":"claimed_verified","statusLastChangedAt":"2026-06-11T08:46:21.518Z","hidden":false},{"_id":"6a282fa3e7d78ea7587e5074","name":"Avi Mendelson","hidden":false},{"_id":"6a282fa3e7d78ea7587e5075","name":"Chaim Baskin","hidden":false}],"publishedAt":"2026-06-05T00:00:00.000Z","submittedOnDailyAt":"2026-06-11T00:00:00.000Z","title":"Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency","submittedOnDailyBy":{"_id":"64ea60c85ba66cfe777e84fc","avatarUrl":"/avatars/75c721b2a45e4f64d29d35f9cc7f7643.svg","isPro":false,"fullname":"itay elam","user":"ItayElam","type":"user","name":"ItayElam"},"summary":"Pipeline parallelism is essential for training large neural networks, but existing schedules trade off throughput, memory, and optimization consistency. Synchronous pipelines preserve forward/backward weight consistency but suffer from bubbles; asynchronous pipelines remove bubbles but introduce weight-version mismatch, typically requiring weight stashing, prediction, or correction mechanisms. We introduce PACI (Pipeline Asynchronous training with Controlled Inconsistency), a bubble-free asynchronous pipeline method that bounds forward/backward version drift without weight stashing, prediction, additional parameter copies, or global synchronization. The key idea is to use local gradient accumulation as a version-control mechanism: by slowing parameter-version evolution relative to pipeline delay, PACI limits the number of optimizer updates crossed by any micro-batch while preserving steady-state utilization. In GPT-style language-model pretraining, PACI matches the stability and final perplexity of synchronous 1F1B-flush, retains the same peak memory footprint, achieves fully utilized pipeline throughput, and improves training time-to-accuracy by up to 1.69times over the fastest flush baseline. These results show that forward/backward inconsistency need not be eliminated: when explicitly bounded, it can be safely traded for substantial efficiency gains.","upvotes":6,"discussionId":"6a282fa3e7d78ea7587e5076","githubRepo":"https://github.com/ItayElam/PACI","githubRepoAddedBy":"user","ai_summary":"PACI enables efficient asynchronous pipeline training by controlling forward/backward weight inconsistency through local gradient accumulation, achieving higher throughput and faster training time-to-accuracy without sacrificing stability or memory usage.","ai_keywords":["pipeline parallelism","synchronous pipelines","asynchronous pipelines","weight stashing","gradient accumulation","optimizer updates","micro-batches","GPT-style language-model pretraining","1F1B-flush","peak memory footprint","steady-state utilization"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":2,"organization":{"_id":"6393322be2364bc1eea56e45","name":"Technion","fullname":"Technion Israel institute of technology","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1670591001944-63926124526c29d5b5011374.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67b9e1ba9734000097f005e5","avatarUrl":"/avatars/5c9969abe1ae7818a30c3936e9c662e8.svg","isPro":false,"fullname":"Eliron Rahimi","user":"ElironRahimi","type":"user"},{"_id":"672f93f448b31810a26a760d","avatarUrl":"/avatars/dc816759d0693b832e116c1a6ce8e215.svg","isPro":false,"fullname":"Avi fishman","user":"Avi05","type":"user"},{"_id":"64ea60c85ba66cfe777e84fc","avatarUrl":"/avatars/75c721b2a45e4f64d29d35f9cc7f7643.svg","isPro":false,"fullname":"itay elam","user":"ItayElam","type":"user"},{"_id":"6a2abffe3d55934730586b3c","avatarUrl":"/avatars/a912242b0bcdf5fbbc8a68b12352ac0f.svg","isPro":false,"fullname":"Ofek A","user":"Vexrzz","type":"user"},{"_id":"66800dc895989c5c95a6cbb1","avatarUrl":"/avatars/4016b5d3b269271f67bc4bd936ac5161.svg","isPro":false,"fullname":"Eliezer Shlomi","user":"EliezerS","type":"user"},{"_id":"6a2ae6c2e36bc84d91b6e7cc","avatarUrl":"/avatars/abf4b4c0020f9332b6827952cc53163e.svg","isPro":false,"fullname":"mmgood","user":"mmgood","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6393322be2364bc1eea56e45","name":"Technion","fullname":"Technion Israel institute of technology","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1670591001944-63926124526c29d5b5011374.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.07881.md"}">
Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency
Abstract
PACI enables efficient asynchronous pipeline training by controlling forward/backward weight inconsistency through local gradient accumulation, achieving higher throughput and faster training time-to-accuracy without sacrificing stability or memory usage.
Pipeline parallelism is essential for training large neural networks, but existing schedules trade off throughput, memory, and optimization consistency. Synchronous pipelines preserve forward/backward weight consistency but suffer from bubbles; asynchronous pipelines remove bubbles but introduce weight-version mismatch, typically requiring weight stashing, prediction, or correction mechanisms. We introduce PACI (Pipeline Asynchronous training with Controlled Inconsistency), a bubble-free asynchronous pipeline method that bounds forward/backward version drift without weight stashing, prediction, additional parameter copies, or global synchronization. The key idea is to use local gradient accumulation as a version-control mechanism: by slowing parameter-version evolution relative to pipeline delay, PACI limits the number of optimizer updates crossed by any micro-batch while preserving steady-state utilization. In GPT-style language-model pretraining, PACI matches the stability and final perplexity of synchronous 1F1B-flush, retains the same peak memory footprint, achieves fully utilized pipeline throughput, and improves training time-to-accuracy by up to 1.69times over the fastest flush baseline. These results show that forward/backward inconsistency need not be eliminated: when explicitly bounded, it can be safely traded for substantial efficiency gains.
Community
Pipeline parallelism is essential for training large neural networks, but existing schedules trade off throughput, memory, and optimization consistency. Synchronous pipelines preserve forward/backward weight consistency but suffer from bubbles; asynchronous pipelines remove bubbles but introduce weight-version mismatch, typically requiring weight stashing, prediction, or correction mechanisms. We introduce PACI (Pipeline Asynchronous training with Controlled Inconsistency), a bubble-free asynchronous pipeline method that bounds forward/backward version drift without weight stashing, prediction, additional parameter copies, or global synchronization. The key idea is to use local gradient accumulation as a version-control mechanism: by slowing parameter-version evolution relative to pipeline delay, PACI limits the number of optimizer updates crossed by any micro-batch while preserving steady-state utilization. In GPT-style language-model pretraining, PACI matches the stability and final perplexity of synchronous 1F1B-flush, retains the same peak memory footprint, achieves fully utilized pipeline throughput, and improves training time-to-accuracy by up to 1.69x over the fastest flush baseline. These results show that forward/backward inconsistency need not be eliminated: when explicitly bounded, it can be safely traded for substantial efficiency gains.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.07881 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.07881 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.07881 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.