FactorJEPA restructures V-JEPA’s monolithic future-latent predictor into explicitly supervised <strong>layout, agent, and interaction subspaces</strong>, adding visibility-aware entity aggregation and cross-channel separation to substantially improve future prediction, intervention sensitivity, and robustness under occlusion in dense urban scenes. </p>\n<p>➡️ 𝐊𝐞𝐲 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬 𝐨𝐟 𝐅𝐚𝐜𝐭𝐨𝐫𝐉𝐄𝐏𝐀:</p>\n<p>🌏 <strong>𝑫𝑬𝑵𝑺𝑬𝑾𝑶𝑹𝑳𝑫: 𝑨 𝑵𝒆𝒘 𝑾𝒐𝒓𝒍𝒅-𝑴𝒐𝒅𝒆𝒍𝒊𝒏𝒈 𝑹𝒆𝒈𝒊𝒎𝒆:</strong> Introduces <strong>DENSEWORLD-115k</strong>, built from ~1,000 hours of drive-through, walk-through, and aerial footage across 22 Indian cities, specifically targeting high agent density, heterogeneous mixed traffic, persistent occlusion, soft spatial boundaries, and rapid multi-agent negotiation. The authors show that frozen JEPA-family representations struggle in this regime—V-JEPA 2.1 reaches only <strong>44.4% action top-1</strong>—motivating world models whose latent structure explicitly represents agents and their interactions rather than relying on correlated scene-level shortcuts. </p>\n<p>🧩 <strong>𝑭𝒂𝒄𝒕𝒐𝒓𝑱𝑬𝑷𝑨: 𝑳𝒂𝒚𝒐𝒖𝒕–𝑨𝒈𝒆𝒏𝒕–𝑰𝒏𝒕𝒆𝒓𝒂𝒄𝒕𝒊𝒐𝒏 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒐𝒓 𝑺𝒖𝒓𝒈𝒆𝒓𝒚:</strong> Replaces V-JEPA’s monolithic predictor with three structured predictive coordinates, $C=[C_L,C_A,C_I]$, synthesized back into the future embedding as $$\\hat{Y}=C_LA_L^\\top+C_AA_A^\\top+C_IA_I^\\top$$. Agents are aggregated through a <strong>soft visibility gate</strong> so occluded entities are downweighted rather than discarded, while interactions use sparse, weighted pairwise relational embeddings. DINOv2-derived structural targets semantically anchor each channel, and covariance + nonlinear RBF dependence penalties suppress cross-factor leakage. Training preserves the pretrained JEPA machinery and performs targeted “predictor surgery,” progressively emphasizing <strong>layout → agents → interactions</strong> while only unfreezing the top encoder blocks. </p>\n<p>📈 <strong>𝑺𝒕𝒓𝒖𝒄𝒕𝒖𝒓𝒆𝒅 𝑭𝒖𝒕𝒖𝒓𝒆 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒊𝒐𝒏 𝑩𝒆𝒂𝒕𝒔 𝑮𝒆𝒏𝒆𝒓𝒊𝒄 𝑭𝒊𝒏𝒆-𝑻𝒖𝒏𝒊𝒏𝒈:</strong> Against LoRA, DoRA, Auto-RGN, full fine-tuning, continual SSL and other adaptations, FactorJEPA improves the metrics most directly tied to world modeling: <strong>Future-frame L1, Causal L1, and Mask-ratio robustness</strong>. With the full 115k corpus on the 1B V-JEPA 2.1 backbone, its advantage over the strongest competitor reaches <strong>33.2×, 13.9×, and 43.3× paired-CI widths</strong>, respectively, while Motion cosine also becomes a <strong>20.0×-CI win</strong>. Method rankings replicate strongly between 1B and 2B backbones $$(\\rho=0.895–0.979)$$ across the four primary diagnostics), suggesting that explicitly structuring <em>how</em> a JEPA represents the future—not merely adapting more parameters—is the key contribution.</p>\n","updatedAt":"2026-08-07T21:08:50.238Z","author":{"_id":"63a4754927f1f64ed7238dac","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg","fullname":"Aman Chadha","name":"amanchadha","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":9,"isUserFollowing":false}},"numEdits":2,"identifiedLanguage":{"language":"en","probability":0.7825747728347778},"editors":["amanchadha"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg"],"reactions":[],"isReport":false}},{"id":"6a7687b30cbc2793b0bee1f0","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":378,"isUserFollowing":false},"createdAt":"2026-08-08T01:34:43.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Diffusion Transformer World-Action Model for AV Scene Prediction](https://huggingface.co/papers/2606.12987) (2026)\n* [Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving](https://huggingface.co/papers/2607.29031) (2026)\n* [OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation](https://huggingface.co/papers/2606.17536) (2026)\n* [ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadow](https://huggingface.co/papers/2607.28362) (2026)\n* [Current World Models Lack a Persistent State Core](https://huggingface.co/papers/2606.20545) (2026)\n* [Temporally Centered SIGReg Improves Multi-Task LeWorldModel Learning: From Analysis to Method](https://huggingface.co/papers/2607.26924) (2026)\n* [The JEPA Paradox in Language: The Geometry of Linguistic Alternatives](https://huggingface.co/papers/2607.23531) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"<p>This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. </p>\n<p>The following papers were recommended by the Semantic Scholar API </p>\n<ul>\n<li><a href=\"https://huggingface.co/papers/2606.12987\">Diffusion Transformer World-Action Model for AV Scene Prediction</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.29031\">Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.17536\">OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.28362\">ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadow</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.20545\">Current World Models Lack a Persistent State Core</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.26924\">Temporally Centered SIGReg Improves Multi-Task LeWorldModel Learning: From Analysis to Method</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.23531\">The JEPA Paradox in Language: The Geometry of Linguistic Alternatives</a> (2026)</li>\n</ul>\n<p> Please give a thumbs up to this comment if you found it helpful!</p>\n<p> If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space</p>\n<p> You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code>@librarian-bot recommend</code></p>\n","updatedAt":"2026-08-08T01:34:43.785Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":378,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7385011911392212},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.01049","authors":[{"_id":"6a7647558e9301703eaa58a4","name":"Kapil Wanaskar","hidden":false},{"_id":"6a7647558e9301703eaa58a5","name":"Gaytri Jena","hidden":false},{"_id":"6a7647558e9301703eaa58a6","user":{"_id":"63a4754927f1f64ed7238dac","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg","isPro":false,"fullname":"Aman Chadha","user":"amanchadha","type":"user","name":"amanchadha"},"name":"Aman Chadha","status":"claimed_verified","statusLastChangedAt":"2026-08-08T00:45:04.557Z","hidden":false},{"_id":"6a7647558e9301703eaa58a7","name":"Vinija Jain","hidden":false},{"_id":"6a7647558e9301703eaa58a8","name":"Vasu Sharma","hidden":false},{"_id":"6a7647558e9301703eaa58a9","name":"Amitava Das","hidden":false}],"publishedAt":"2026-08-02T00:00:00.000Z","submittedOnDailyAt":"2026-08-07T00:00:00.000Z","title":"FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds","submittedOnDailyBy":{"_id":"63a4754927f1f64ed7238dac","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg","isPro":false,"fullname":"Aman Chadha","user":"amanchadha","type":"user","name":"amanchadha"},"summary":"World models have attracted significant attention for their ability to capture and predict the structure and dynamics of the physical world. In this emerging landscape, Joint Embedding Predictive Architectures (JEPA) offer a particularly compelling direction.\n We study a largely unexplored regime: populous, crowded, and chaotic Global South urban environments, which we call DENSEWORLD. Unlike the lower-density, lane-structured settings that dominate existing evaluations, these scenes exhibit soft spatial boundaries, extreme agent heterogeneity, persistent occlusion, and rapid social negotiation under mixed traffic. We introduce the first large-scale dataset for this regime: 1,000 hours of drive-through, walk-through, and aerial video across 22 cities. Existing JEPA formulations struggle to preserve dense interaction dynamics under heterogeneity and partial observability.\n We introduce FactorJEPA, which makes world structure a first-class predictive primitive. Rather than encoding the future in a monolithic latent, it composes layout, entities, and interactions, using a visibility gate and separated subspaces to preserve partially observed agents and discourage cross-factor shortcuts. FactorJEPA improves (i) future-latent accuracy (Future-frame L1), (ii) intervention-sensitive prediction (Causal L1), and (iii) robustness to reduced visual evidence (Mask-ratio slope), while exposing (iv) a reproducible motion-information trade-off (Motion cosine). Method rankings replicate across 2B and 1B V-JEPA 2.1 backbones, with rho = 0.895 to 0.978.\n We publicly release the DENSEWORLD-115k dataset (https://huggingface.co/datasets/anonymousML123/denseworld-115k) and the surgery-trained FactorJEPA checkpoints (https://huggingface.co/datasets/anonymousML123/factorjepa-outputs/tree/main/outputs/full/vjepa_2_1_vitg_1B/train/m09c_surgery_3stage_DI_diheavy_encoder).","upvotes":5,"discussionId":"6a7647568e9301703eaa58aa"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6a742a683d438d7c49c8eaa2","avatarUrl":"/avatars/f7828c23fad2988df910b983e5f466b2.svg","isPro":false,"fullname":"Benjamin Nathaniel Hathorne","user":"Entmarch","type":"user"},{"_id":"6a6a92f51b3822fa45f64b3b","avatarUrl":"/avatars/8d2e9df93e7cdbe7a98b8511f0fa2efe.svg","isPro":false,"fullname":"Kevin Perez","user":"k-perez","type":"user"},{"_id":"6a6c894d5a0fa9cc28ac665d","avatarUrl":"/avatars/7c9c2321c82d6c16af04817930b4f1c6.svg","isPro":false,"fullname":"Karen Williams","user":"Karen-Williams","type":"user"},{"_id":"6a6dea1d067f0e2726f81e06","avatarUrl":"/avatars/02afbceec3ca6047937af545cccef437.svg","isPro":false,"fullname":"Edward Lee","user":"Azure-Leo","type":"user"},{"_id":"6a6def87cc72fa11589cab9d","avatarUrl":"/avatars/14eda435fb8fefecf1b3fc923fe7f7b2.svg","isPro":false,"fullname":"Robert Lee","user":"FrostRobert","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.01049.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2608.01049 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.01049 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2608.01049 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.