Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory
Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.
Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory
Abstract
Matrix-Game 3.5 improves interactive world generation via geometry-aware memory, static-dynamic disentanglement, and progressive distillation for long-horizon real-time simulation.
Interactive world models extend video generation from offline clip synthesis toward persistent simulation of interactive virtual worlds, enabling applications in games, robotics, embodied agents, and XR. Achieving stable long-horizon interactive generation, however, remains challenging, as the model must simultaneously preserve scene geometry, dynamic consistency, and camera control while supporting real-time autoregressive generation. Building upon Matrix-Game 3.0, we present Matrix-Game 3.5, as shown in Figure 1, which advances real-time interactive world generation toward geometry-aware and long-horizon consistent simulation through three key improvements. First, we propose a unified geometry-aware memory framework, whose patch-memory and tiled-PRoPE components introduce no additional learnable parameters, combining explicit 3D patch retrieval with projective camera conditioning to enable geometry-consistent camera control and faithful long-horizon scene recall. Second, we introduce a static-dynamic disentangled world representation that separately models static scene geometry and dynamic subjects, preserving both geometric consistency and subject identity throughout long-horizon generation. Third, we develop a two-stage progressive real-time distillation framework that converts a bidirectional diffusion model into a few-step causal generator through Perceptual Flow Matching and curriculum based Self-Rollout DMD, enabling minute-long real-time interactive generation. Extensive experiments demonstrate that, with a unified training corpus spanning Unreal simulation environments, open-world games, and internet videos, MatrixGame 3.5 achieves strong performance in long-horizon scene recall, precise camera control, subject consistency, prompt-driven world generation, and stable real-time open-world interaction.
Models citing this paper
No model linking this paper
Datasets citing this paper
No dataset linking this paper
Spaces citing this paper
No Space linking this paper
Collections including this paper
No Collection including this paper
More from Hugging Face Daily Papers
-
CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents
Sep 17
-
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Sep 17
-
In-Context Robot Learning with VLM Agents
Sep 17
-
Assessing nnU-Net Generalization across Brain Tumor Populations in BraTS-GoAT 2026
Sep 17
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.