Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at <a href=\"https://github.com/Within-yao/EnvACE\" rel=\"nofollow\">https://github.com/Within-yao/EnvACE</a>.</p>\n","updatedAt":"2026-08-07T03:02:26.641Z","author":{"_id":"65afbba21edab235a1323ad0","avatarUrl":"/avatars/bd047a559821e2bc802d66a073b994df.svg","fullname":"3","name":"xuzishan","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.917829155921936},"editors":["xuzishan"],"editorAvatarUrls":["/avatars/bd047a559821e2bc802d66a073b994df.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.06197","authors":[{"_id":"6a75423ae1228e04b32381ab","name":"Zishan Xu","hidden":false},{"_id":"6a75423ae1228e04b32381ac","name":"Zhiyuan Yao","hidden":false},{"_id":"6a75423ae1228e04b32381ad","name":"Yuxin Chen","hidden":false},{"_id":"6a75423ae1228e04b32381ae","user":{"_id":"6603e7b679ed63106e974434","avatarUrl":"/avatars/4a7de974260da697ebab1136ee070c9f.svg","isPro":false,"fullname":"Yifu Guo","user":"Eric1019","type":"user","name":"Eric1019"},"name":"Yifu Guo","status":"claimed_verified","statusLastChangedAt":"2026-08-07T08:45:04.536Z","hidden":false},{"_id":"6a75423ae1228e04b32381af","user":{"_id":"676127cf11b19ea602bb202a","avatarUrl":"/avatars/dfd802a24bd63e509728159ebb1769f6.svg","isPro":false,"fullname":"Zhengxi Lu","user":"LZXzju","type":"user","name":"LZXzju"},"name":"Zhengxi Lu","status":"claimed_verified","statusLastChangedAt":"2026-08-07T08:45:04.541Z","hidden":false},{"_id":"6a75423ae1228e04b32381b0","name":"Yuquan Lu","hidden":false},{"_id":"6a75423ae1228e04b32381b1","name":"Jinyang Huang","hidden":false},{"_id":"6a75423ae1228e04b32381b2","name":"Yan Xu","hidden":false},{"_id":"6a75423ae1228e04b32381b3","name":"Yasheng Wang","hidden":false},{"_id":"6a75423ae1228e04b32381b4","name":"Weinan Zhang","hidden":false},{"_id":"6a75423ae1228e04b32381b5","name":"Xingshan Zeng","hidden":false},{"_id":"6a75423ae1228e04b32381b6","name":"Weiwen Liu","hidden":false}],"publishedAt":"2026-08-06T00:00:00.000Z","submittedOnDailyAt":"2026-08-07T00:00:00.000Z","title":"EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning","submittedOnDailyBy":{"_id":"65afbba21edab235a1323ad0","avatarUrl":"/avatars/bd047a559821e2bc802d66a073b994df.svg","isPro":true,"fullname":"3","user":"xuzishan","type":"user","name":"xuzishan"},"summary":"Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.","upvotes":28,"discussionId":"6a75423be1228e04b32381b7","projectPage":"https://within-yao.github.io/EnvACE/","githubRepo":"https://github.com/Within-yao/EnvACE","githubRepoAddedBy":"user","githubStars":13,"organization":{"_id":"66543b6e420092799d2f625c","name":"tencent","fullname":"Tencent","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/Lp3m-XLpjQGwBItlvn69q.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6603e7b679ed63106e974434","avatarUrl":"/avatars/4a7de974260da697ebab1136ee070c9f.svg","isPro":false,"fullname":"Yifu Guo","user":"Eric1019","type":"user"},{"_id":"6708edcae69f6e30a816af9f","avatarUrl":"/avatars/c4daa9b0cb2f4bb2a7db0e78b22034cb.svg","isPro":false,"fullname":"Yao","user":"distant-yuan","type":"user"},{"_id":"664d68a44be557d4bf53af93","avatarUrl":"/avatars/804a7f01bbc3f337d549b8be925e908c.svg","isPro":false,"fullname":"Xiinye Wang","user":"xinyewang","type":"user"},{"_id":"6565aa15459394ff71996b54","avatarUrl":"/avatars/7b969804b810bb7438faabab95e52098.svg","isPro":false,"fullname":"Yuxin Liu","user":"sod1010","type":"user"},{"_id":"65afbba21edab235a1323ad0","avatarUrl":"/avatars/bd047a559821e2bc802d66a073b994df.svg","isPro":true,"fullname":"3","user":"xuzishan","type":"user"},{"_id":"68a2e1c81a9e1d6d27688a09","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/68a2e1c81a9e1d6d27688a09/WctOz1NhnALEw9fAmtG_k.jpeg","isPro":false,"fullname":"Weiwen","user":"vivienlau","type":"user"},{"_id":"6747de57f8cab58c22ec94a2","avatarUrl":"/avatars/5bae0341862fac24564781c0fa32aac5.svg","isPro":false,"fullname":"Jinyang Wu","user":"Jinyang23","type":"user"},{"_id":"6858f7393794a7a08a254364","avatarUrl":"/avatars/aa2870bed47f7ef25356a4e9ff54ce38.svg","isPro":false,"fullname":"kexu Cheng*","user":"sediment1024","type":"user"},{"_id":"696db9060aefda1f4520816e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5WcvTplXGNUn7Y9vvC_Us.jpeg","isPro":false,"fullname":"Zhaoyang Liu","user":"AngelinaTheLime","type":"user"},{"_id":"68ad19ea5c1a878ddb3d8a04","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Tg90gBaSYj5YFVvDLnWBg.png","isPro":false,"fullname":"Ruitong Zhang","user":"DongdongZ","type":"user"},{"_id":"656c3503e0ff1cebe95e5730","avatarUrl":"/avatars/e9e6ecde1ca79833bfe23df51ee0dc68.svg","isPro":false,"fullname":"Tsuki","user":"Tsukihjy","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"66543b6e420092799d2f625c","name":"tencent","fullname":"Tencent","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/Lp3m-XLpjQGwBItlvn69q.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.06197.md","query":{}}">
EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning
Published on Aug 6
· Submitted by 3 on Aug 7 Abstract
Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.
Community
Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.06197 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.06197 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.06197 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.