Reducing drift in interactive world-model rollouts: a mixed bidirectional/autoregressive attention mask + distillation over long self-rollouts[R]

Read through the method behind an open-weights interactive world model whose weights just went public. The backbone is a causal DiT generating frames live, conditioned on user input. To stop it from over-relying on its own recent frames, the usual source of drift, they use a MoBA attention mask that mixes bidirectional and autoregressive attention, with dynamic KV-cache scheduling so long rollouts stay tractable. Camera control is Plücker embeddings plus AdaLN. The part that stands out is the post-training: consistency distillation and distribution-matching distillation computed over long self-rollout trajectories, not just teacher-forced frames, which is what they credit for LingBot World staying stable across long interactive sessions. Their own stress test is a single continuous 60-minute rollout with no visible decay; no independent reproductions exist yet given how new this is. Honest caveat from their limitations section: persistence is in appearance, not identity, so a region that leaves the context window is regenerated on revisit, not recalled. Weights are open but CC-BY-NC-SA, so noncommercial. The paper and weights are under lingbot-world-v2 for anyone who wants to poke at it. Curious whether the long-rollout stability holds up once people start running it.