Role-playing agents (RPAs) have become one of the most important consumer applications of large language models. Users engage in multi-turn conversations with RPAs for experiences such as emotional comfort, making reliable evaluation essential for measuring capability, comparing systems, and guiding further improvement. Existing benchmarks, however, typically require an RPA to continue a fixed dialogue history and then evaluate the continuation using a fixed rubric detached from the user. We identify and empirically demonstrate two limitations of this design. First, an RPA's output is shaped by the preceding dialogue history, preventing a scientifically grounded assessment of its role-playing ability in real multi-turn settings. Second, user experience varies substantially across individuals, and conventional fixed rubrics need not align with user satisfaction. We therefore introduce PALATE (Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation), a scalable RPA benchmark built on user simulators. PALATE is accompanied by a pool of 300 character profiles. Its main evaluation trains five per-user simulators and lets them engage candidate RPAs in free-form, multi-turn conversations over a pre-frozen panel of character profiles. Alongside a general quality rubric, we construct personalized rubrics to measure user satisfaction; on held-out annotated data, the personalized rubrics show higher agreement with human judgments than the general rubric. In the main evaluation of 16 candidates, PALATE separately characterizes generic turn quality, long-horizon session capability, and per-user experience on multi-turn trajectories co-constructed by each candidate. It thereby produces interpretable evaluations of specific user-RPA pairs rather than compressing systems into a single user-independent ranking.</p>\n","updatedAt":"2026-07-31T03:44:07.450Z","author":{"_id":"646dbba74ad7f907279dd486","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/XGHJMFEIpWeDlh0cn9Slu.png","fullname":"Mingxuan Du","name":"Ayanami0730","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":13,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9058874845504761},"editors":["Ayanami0730"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/XGHJMFEIpWeDlh0cn9Slu.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.27816","authors":[{"_id":"6a6c11d1202e2d9e3ffdb74a","user":{"_id":"6a6824d5a2bbdde86f7cad76","avatarUrl":"/avatars/6ff679759fc9be5956f1b12799a3b874.svg","isPro":false,"fullname":"Yuhang Zhu","user":"Yuki-Kitayama","type":"user","name":"Yuki-Kitayama"},"name":"Yuhang Zhu","status":"claimed_verified","statusLastChangedAt":"2026-07-31T08:45:05.579Z","hidden":false},{"_id":"6a6c11d1202e2d9e3ffdb74b","user":{"_id":"646dbba74ad7f907279dd486","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/XGHJMFEIpWeDlh0cn9Slu.png","isPro":false,"fullname":"Mingxuan Du","user":"Ayanami0730","type":"user","name":"Ayanami0730"},"name":"Mingxuan Du","status":"claimed_verified","statusLastChangedAt":"2026-07-31T08:45:05.586Z","hidden":false},{"_id":"6a6c11d1202e2d9e3ffdb74c","name":"Benfeng Xu","hidden":false},{"_id":"6a6c11d1202e2d9e3ffdb74d","name":"Jie Gao","hidden":false},{"_id":"6a6c11d1202e2d9e3ffdb74e","name":"Lingyun Yu","hidden":false},{"_id":"6a6c11d1202e2d9e3ffdb74f","name":"Hongtao Xie","hidden":false}],"publishedAt":"2026-07-30T00:00:00.000Z","submittedOnDailyAt":"2026-07-31T00:00:00.000Z","title":"Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation","submittedOnDailyBy":{"_id":"646dbba74ad7f907279dd486","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/XGHJMFEIpWeDlh0cn9Slu.png","isPro":false,"fullname":"Mingxuan Du","user":"Ayanami0730","type":"user","name":"Ayanami0730"},"summary":"Role-playing agents (RPAs) have become one of the most important consumer applications of large language models. Users engage in multi-turn conversations with RPAs for experiences such as emotional comfort, making reliable evaluation essential for measuring capability, comparing systems, and guiding further improvement. Existing benchmarks, however, typically require an RPA to continue a fixed dialogue history and then evaluate the continuation using a fixed rubric detached from the user. We identify and empirically demonstrate two limitations of this design. First, an RPA's output is shaped by the preceding dialogue history, preventing a scientifically grounded assessment of its role-playing ability in real multi-turn settings. Second, user experience varies substantially across individuals, and conventional fixed rubrics need not align with user satisfaction. We therefore introduce PALATE (Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation), a scalable RPA benchmark built on user simulators. PALATE is accompanied by a pool of 300 character profiles. Its main evaluation trains five per-user simulators and lets them engage candidate RPAs in free-form, multi-turn conversations over a pre-frozen panel of character profiles. Alongside a general quality rubric, we construct personalized rubrics to measure user satisfaction; on held-out annotated data, the personalized rubrics show higher agreement with human judgments than the general rubric. In the main evaluation of 16 candidates, PALATE separately characterizes generic turn quality, long-horizon session capability, and per-user experience on multi-turn trajectories co-constructed by each candidate. It thereby produces interpretable evaluations of specific user-RPA pairs rather than compressing systems into a single user-independent ranking.","upvotes":28,"discussionId":"6a6c11d2202e2d9e3ffdb750","githubRepo":"https://github.com/Zhuyh1139/PALATE","githubRepoAddedBy":"user","githubStars":3,"organization":{"_id":"6912993f45f02a20f4c50b8a","name":"muset-ai","fullname":"muset.ai","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/C5j0HEiqEmmxshM61WRON.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6a6824d5a2bbdde86f7cad76","avatarUrl":"/avatars/6ff679759fc9be5956f1b12799a3b874.svg","isPro":false,"fullname":"Yuhang Zhu","user":"Yuki-Kitayama","type":"user"},{"_id":"646dbba74ad7f907279dd486","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/XGHJMFEIpWeDlh0cn9Slu.png","isPro":false,"fullname":"Mingxuan Du","user":"Ayanami0730","type":"user"},{"_id":"68959bf9950f311270f0e010","avatarUrl":"/avatars/96f5ac05139954579a97a80e0f29b528.svg","isPro":false,"fullname":"Ruizhe Li","user":"imlrz","type":"user"},{"_id":"6a2a342b086133dad7d61c92","avatarUrl":"/avatars/db700b4fc2f762f072e4a22b2f48b757.svg","isPro":false,"fullname":"annlin","user":"annl092723","type":"user"},{"_id":"62d56f063bf5e059f7cac515","avatarUrl":"/avatars/6bd5cfdc21506ace6176d00a2973d8e5.svg","isPro":false,"fullname":"BenfengXu","user":"SpiketheCowboy","type":"user"},{"_id":"68cbf13fd9c1b531d9564757","avatarUrl":"/avatars/e1e26dc251102c5c18ce5fa172e6a8fa.svg","isPro":false,"fullname":"Zhou","user":"Tuowen","type":"user"},{"_id":"68c8ccff2de8edcf8da2fe51","avatarUrl":"/avatars/e55e509d11049691fefe3750a4bed62a.svg","isPro":false,"fullname":"wangpengyu","user":"Bstwpy","type":"user"},{"_id":"6440d49b7663594a126716f2","avatarUrl":"/avatars/bb04af9d1ae9c5bc058ebfbf08f4ebc8.svg","isPro":false,"fullname":"shaohanwang","user":"WShao","type":"user"},{"_id":"6a0fc58cc8676ad292e8a15d","avatarUrl":"/avatars/2649c5c1d64cd72b77265747b89e0cea.svg","isPro":false,"fullname":"Ruizhe Li","user":"imlrz01","type":"user"},{"_id":"675eebb86205e9e0d9a0feab","avatarUrl":"/avatars/8204df1b850e556094f51c4ad019fc7c.svg","isPro":false,"fullname":"ccrrkk","user":"ccrrkk","type":"user"},{"_id":"62c926c71eba56b9f6212922","avatarUrl":"/avatars/e78d2bceae9b9f2e5295408f7094c4c8.svg","isPro":false,"fullname":"Hal","user":"farawayxxx","type":"user"},{"_id":"64ffd28c4b7e0cd1b7c05fd1","avatarUrl":"/avatars/684ad1ed6d251f519488f1b93bf135e7.svg","isPro":false,"fullname":"Gu","user":"Infinitegu","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6912993f45f02a20f4c50b8a","name":"muset-ai","fullname":"muset.ai","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/C5j0HEiqEmmxshM61WRON.png"},"query":{}}">
Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
Abstract
Role-playing agents (RPAs) have become one of the most important consumer applications of large language models. Users engage in multi-turn conversations with RPAs for experiences such as emotional comfort, making reliable evaluation essential for measuring capability, comparing systems, and guiding further improvement. Existing benchmarks, however, typically require an RPA to continue a fixed dialogue history and then evaluate the continuation using a fixed rubric detached from the user. We identify and empirically demonstrate two limitations of this design. First, an RPA's output is shaped by the preceding dialogue history, preventing a scientifically grounded assessment of its role-playing ability in real multi-turn settings. Second, user experience varies substantially across individuals, and conventional fixed rubrics need not align with user satisfaction. We therefore introduce PALATE (Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation), a scalable RPA benchmark built on user simulators. PALATE is accompanied by a pool of 300 character profiles. Its main evaluation trains five per-user simulators and lets them engage candidate RPAs in free-form, multi-turn conversations over a pre-frozen panel of character profiles. Alongside a general quality rubric, we construct personalized rubrics to measure user satisfaction; on held-out annotated data, the personalized rubrics show higher agreement with human judgments than the general rubric. In the main evaluation of 16 candidates, PALATE separately characterizes generic turn quality, long-horizon session capability, and per-user experience on multi-turn trajectories co-constructed by each candidate. It thereby produces interpretable evaluations of specific user-RPA pairs rather than compressing systems into a single user-independent ranking.
Community
Role-playing agents (RPAs) have become one of the most important consumer applications of large language models. Users engage in multi-turn conversations with RPAs for experiences such as emotional comfort, making reliable evaluation essential for measuring capability, comparing systems, and guiding further improvement. Existing benchmarks, however, typically require an RPA to continue a fixed dialogue history and then evaluate the continuation using a fixed rubric detached from the user. We identify and empirically demonstrate two limitations of this design. First, an RPA's output is shaped by the preceding dialogue history, preventing a scientifically grounded assessment of its role-playing ability in real multi-turn settings. Second, user experience varies substantially across individuals, and conventional fixed rubrics need not align with user satisfaction. We therefore introduce PALATE (Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation), a scalable RPA benchmark built on user simulators. PALATE is accompanied by a pool of 300 character profiles. Its main evaluation trains five per-user simulators and lets them engage candidate RPAs in free-form, multi-turn conversations over a pre-frozen panel of character profiles. Alongside a general quality rubric, we construct personalized rubrics to measure user satisfaction; on held-out annotated data, the personalized rubrics show higher agreement with human judgments than the general rubric. In the main evaluation of 16 candidates, PALATE separately characterizes generic turn quality, long-horizon session capability, and per-user experience on multi-turn trajectories co-constructed by each candidate. It thereby produces interpretable evaluations of specific user-RPA pairs rather than compressing systems into a single user-independent ranking.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.27816 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.27816 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.27816 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.