Sumi: Open Uniform Diffusion Language Model from Scratch
Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.
Sumi: Open Uniform Diffusion Language Model from Scratch
Abstract
A large-scale uniform diffusion language model pretrained from scratch demonstrates competitive performance on knowledge and reasoning tasks while highlighting differences in commonsense reasoning compared to autoregressive models.
Diffusion models have become a promising alternative to autoregressive models. Among these, uniform diffusion language models (UDLMs) permit any token to be updated at any step, in principle enabling more flexible generation. However, no UDLM has yet been pretrained from scratch at both large parameter scale and large token budget. Both autoregressive modeling and masked diffusion modeling already have capable models at scale that the community can study and build on; uniform diffusion has none. A scratch-pretrained UDLM at scale would provide a clean reference point for studying scaling behavior, generation dynamics, controllability, and trade-offs against established autoregressive and masked diffusion models. To this end, we introduce Sumi ("ink" in Japanese), a fully open 7B uniform diffusion language model pretrained from scratch on 1.5T tokens. Sumi performs competitively with autoregressive models trained at comparable token budgets on knowledge, reasoning, and coding benchmarks, while under-performing on commonsense benchmarks, where our education-heavy data mixture is a likely contributor. We release our model weights, checkpoints, and full training recipe, including a complete specification of the data mixture over publicly available corpora. We hope this release enables the community to study native uniform diffusion at scale and catalyzes work on its as-yet poorly understood aspects.
Get this paper in your agent:
hf papers read 2606.19005 curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 1
More from Hugging Face Daily Papers
-
COrigami: An AI Pipeline for Co-Designing Flat-Foldable Visually Recognisable Origami
Jun 27
-
Fast LeWorldModel
Jun 27
-
ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation
Jun 27
-
Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
Jun 26
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.