LychSim is a highly controllable, interactive simulation framework built on Unreal Engine 5, designed to lower the technical barrier of using a modern game engine for computer vision research.</p>\n","updatedAt":"2026-05-13T01:57:54.063Z","author":{"_id":"625f81afe1994410eef1c36a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1650426282769-noauth.jpeg","fullname":"Wufei Ma","name":"wufeim","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8692672848701477},"editors":["wufeim"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1650426282769-noauth.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.12449","authors":[{"_id":"6a03da3b86b054ce2fa40d25","name":"Wufei Ma","hidden":false},{"_id":"6a03da3b86b054ce2fa40d26","name":"Chloe Wang","hidden":false},{"_id":"6a03da3b86b054ce2fa40d27","name":"Siyi Chen","hidden":false},{"_id":"6a03da3b86b054ce2fa40d28","name":"Jiawei Peng","hidden":false},{"_id":"6a03da3b86b054ce2fa40d29","name":"Patrick Li","hidden":false},{"_id":"6a03da3b86b054ce2fa40d2a","name":"Alan Yuille","hidden":false}],"publishedAt":"2026-05-12T00:00:00.000Z","submittedOnDailyAt":"2026-05-13T00:00:00.000Z","title":"LychSim: A Controllable and Interactive Simulation Framework for Vision Research","submittedOnDailyBy":{"_id":"625f81afe1994410eef1c36a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1650426282769-noauth.jpeg","isPro":true,"fullname":"Wufei Ma","user":"wufeim","type":"user","name":"wufeim"},"summary":"While self-supervised pretraining has reduced vision systems' reliance on synthetic data, simulation remains an indispensable tool for closed-loop optimization and rigorous out-of-distribution (OOD) evaluation. However, modern simulation platforms often present steep technical barriers, requiring extensive expertise in computer graphics and game development. In this work, we present LychSim, a highly controllable and interactive simulation framework built upon Unreal Engine 5 to bridge this gap. LychSim is built around three key designs: (1) a streamlined Python API that abstracts away underlying engine complexities; (2) a procedural data pipeline capable of generating diverse, high-fidelity environments with varying out-of-distribution (OOD) visual challenges, paired with rich 2D and 3D ground truths; and (3) a native integration of the Model Context Protocol (MCP) that transforms the simulator into a dynamic, closed-loop playground for reasoning agentic LLMs. We further annotate scene-level procedural rules and object-level pose alignments to enable semantically aligned 3D ground truths and automated scene modification. We demonstrate LychSim's capability across multiple downstream applications, including serving as a synthetic data engine, powering reinforcement learning-based adversarial examiners, and facilitating interactive, language-driven scene layout generation. To benefit the broader vision community, LychSim will be made publicly available, including full source code and various data annotations.","upvotes":3,"discussionId":"6a03da3b86b054ce2fa40d2b","projectPage":"https://lychsim.github.io/","githubRepo":"https://github.com/wufeim/LychSim","githubRepoAddedBy":"user","ai_summary":"A simulation framework called LychSim is introduced, featuring a Python API, procedural data pipeline, and MCP integration to enable controllable and interactive environments for vision system development and evaluation.","ai_keywords":["simulation framework","Unreal Engine 5","Python API","procedural data pipeline","out-of-distribution evaluation","Model Context Protocol","reinforcement learning","synthetic data engine","language-driven scene layout generation"],"githubStars":5,"organization":{"_id":"653945b47ba797097a7b4eab","name":"JohnsHopkins","fullname":"Johns Hopkins University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/653944e58e687a41625a4694/qqHzBOarppVrUuZbbjqwh.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"625f81afe1994410eef1c36a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1650426282769-noauth.jpeg","isPro":true,"fullname":"Wufei Ma","user":"wufeim","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"6a03e73cf6af1506edc21796","avatarUrl":"/avatars/e4e556dd6cd841bf437634e5d33e58b7.svg","isPro":false,"fullname":"Lizhi Ma","user":"dafeigou","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"653945b47ba797097a7b4eab","name":"JohnsHopkins","fullname":"Johns Hopkins University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/653944e58e687a41625a4694/qqHzBOarppVrUuZbbjqwh.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.12449.md"}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2605.12449 in a model README.md to link it from this page.

Cite arxiv.org/abs/2605.12449 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2605.12449 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.