Title:Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models

Abstract:Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales
quadratically with context length and becomes prohibitive for long-context scenarios. We propose Prefilling-dLLM, a training-free
prefill-decode disaggregation framework for dLLMs that partitions the prefix into N chunks, caches their KV representations once,
and selects the top-K most relevant chunks with intra-chunk token sparsity for decoding, showing that sparse prefilling can
outperform dense attention while reducing per-step complexity from quadratic in the full sequence length to quadratic only in the
decode length. On LongBench and InfiniteBench, Prefilling-dLLM achieves state-of-the-art quality among dLLM acceleration methods,
and an attention kernel that parallelizes decoding over the non-contiguously cached chunk KV yields 9.1--28.0x speedup at 8K--32K
contexts. We further show that beginning-of-sequence tokens prepended to each chunk act as periodic attention anchors that eliminate
the lost-in-the-middle phenomenon. Code is available at this https URL.

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools