awesome</p>\n","updatedAt":"2026-06-25T07:41:34.936Z","author":{"_id":"6752cc1a10576e69f9bdc542","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6752cc1a10576e69f9bdc542/WkFgo6vx07H6IVLRZmFO_.jpeg","fullname":"Chanyoung Kim","name":"chanyoungkim","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.4128333032131195},"editors":["chanyoungkim"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6752cc1a10576e69f9bdc542/WkFgo6vx07H6IVLRZmFO_.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.25331","authors":[{"_id":"6a3c9112f3facdb67e9ff0a4","name":"Shen Nie","hidden":false},{"_id":"6a3c9112f3facdb67e9ff0a5","name":"Qiyang Min","hidden":false},{"_id":"6a3c9112f3facdb67e9ff0a6","name":"Shaoxuan Xu","hidden":false},{"_id":"6a3c9112f3facdb67e9ff0a7","name":"Zihao Huang","hidden":false},{"_id":"6a3c9112f3facdb67e9ff0a8","name":"Yuxuan Song","hidden":false},{"_id":"6a3c9112f3facdb67e9ff0a9","name":"Yong Shan","hidden":false},{"_id":"6a3c9112f3facdb67e9ff0aa","name":"Yankai Lin","hidden":false},{"_id":"6a3c9112f3facdb67e9ff0ab","name":"Wayne Xin Zhao","hidden":false},{"_id":"6a3c9112f3facdb67e9ff0ac","name":"Chongxuan Li","hidden":false},{"_id":"6a3c9112f3facdb67e9ff0ad","name":"Ji-Rong Wen","hidden":false}],"publishedAt":"2026-06-24T00:00:00.000Z","submittedOnDailyAt":"2026-06-25T00:00:00.000Z","title":"Improved Large Language Diffusion Models","submittedOnDailyBy":{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user","name":"taesiri"},"summary":"Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present iLLaDA, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.","upvotes":21,"discussionId":"6a3c9113f3facdb67e9ff0ae","ai_summary":"Masked diffusion language models with fully bidirectional attention outperform autoregressive counterparts on various benchmarks while maintaining competitiveness with established models.","ai_keywords":["masked diffusion language model","bidirectional attention","autoregressive factorization","causal attention","supervised fine-tuning","variable-length generation","confidence-based scoring","BBH","ARC-Challenge","MATH","HumanEval"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"67d1140985ea0644e2f14b99","name":"ByteDance-Seed","fullname":"ByteDance Seed","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/flkDUqd_YEuFsjeNET3r-.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"665ebae8bcbb98f60db0b4b1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/665ebae8bcbb98f60db0b4b1/YTKM4qTZXh_2SeU8U7BfB.webp","isPro":false,"fullname":"Jiale Zhao","user":"Heisenburger2000","type":"user"},{"_id":"63958b4414513eaf9029ebf1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png","isPro":false,"fullname":"Xiangtai Li","user":"LXT","type":"user"},{"_id":"67962e3d7d631a6694b605d3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/hxdT83LPL6g7H7Ke79AkP.png","isPro":false,"fullname":"RuiyangSi","user":"RuiyangSi","type":"user"},{"_id":"67344c08ba8aa259654bd819","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67344c08ba8aa259654bd819/OzFuQ9ebdOlX8hCq0Yrm1.jpeg","isPro":false,"fullname":"Yueyi Sun","user":"bitersun","type":"user"},{"_id":"66f960de7b28c82e56d97829","avatarUrl":"/avatars/965d116d95b82adaf1ff3812bd7f1161.svg","isPro":false,"fullname":"张桓炜","user":"2044154891cz","type":"user"},{"_id":"631c386bc73939ffc0716a37","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1662793811119-noauth.jpeg","isPro":false,"fullname":"SeongWan Kim","user":"idgmatrix","type":"user"},{"_id":"65c4eb7cd1dcbd30d86febec","avatarUrl":"/avatars/001c8f02e8ce794b2c21883628b2da72.svg","isPro":false,"fullname":"free-bit","user":"free-bit","type":"user"},{"_id":"640dd9a5fdeaae13908208a7","avatarUrl":"/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg","isPro":false,"fullname":"nieshen","user":"nieshen","type":"user"},{"_id":"666bb205926c2e7e7837f2c6","avatarUrl":"/avatars/6bd696cbd1a9abf6a0d83f0e907da697.svg","isPro":false,"fullname":"HanzhongGuo","user":"Alllann","type":"user"},{"_id":"67be0888267276f5a2f9ce71","avatarUrl":"/avatars/c27dc0003351d2e59d7361064d572e55.svg","isPro":false,"fullname":"Jia-Nan Li","user":"JinaLeejnl","type":"user"},{"_id":"624f909eac5dd186b01ac3f5","avatarUrl":"/avatars/0aafdb1cbb492fda52a0303031cc6c14.svg","isPro":false,"fullname":"Zebin You","user":"yyyou","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"67d1140985ea0644e2f14b99","name":"ByteDance-Seed","fullname":"ByteDance Seed","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/flkDUqd_YEuFsjeNET3r-.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.25331.md","query":{}}">
Improved Large Language Diffusion Models
Authors: ,
,
,
,
,
,
,
,
,
Abstract
Masked diffusion language models with fully bidirectional attention outperform autoregressive counterparts on various benchmarks while maintaining competitiveness with established models.
Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present iLLaDA, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.25331 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.25331 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.25331 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.