Personalization is becoming a core promise of LLM systems: chatbots remember your job, interests, preferences, and past conversations to tailor responses. But “personalized” does not always mean helpful — it can also feel uncomfortable, offensive, or just unnecessary.</p>\n<p>This raises a basic but surprisingly under-examined question: 𝗪𝗵𝗼 𝗱𝗲𝗰𝗶𝗱𝗲𝘀 𝘄𝗵𝗲𝘁𝗵𝗲𝗿 𝗽𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗵𝗲𝗹𝗽𝗳𝘂𝗹 — 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹, 𝗼𝗿 𝘁𝗵𝗲 𝗽𝗲𝗿𝘀𝗼𝗻 𝗯𝗲𝗶𝗻𝗴 𝗽𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘇𝗲𝗱 𝗳𝗼𝗿?</p>\n<p>Most existing benchmarks rely heavily on synthetic personas, simulated conversations, and LLM judges. In this work, we put 𝗿𝗲𝗮𝗹 𝗵𝘂𝗺𝗮𝗻𝘀 back into the loop.</p>\n<p>We study personalization as a three-stage pipeline:</p>\n<p>🧠 𝗔𝘁𝘁𝗿𝗶𝗯𝘂𝘁𝗲 𝗲𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — what should the system infer from conversation history?<br>🎯 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗺𝗮𝘁𝗰𝗵𝗶𝗻𝗴 — which attributes actually matter for the current request?<br>✍️ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 — does personalization improve the user experience?</p>\n<p>Using 550 real user conversations and nearly 19,000 human judgments, we find systematic 𝗵𝘂𝗺𝗮𝗻–𝗟𝗟𝗠 𝗴𝗮𝗽𝘀 at every stage:</p>\n<p>• Models extract noisy and overgeneralized attributes from real conversations. Synthetic data underestimates this difficulty.<br>• LLMs and humans disagree on which attributes should be used in a new question(κ=0.30), but each agree well within their own group (κ=0.60 and 0.43).<br>• LLMs select 𝟮–𝟯× 𝗺𝗼𝗿𝗲 𝗮𝘁𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝘀 as relevant than humans do, suggesting a tendency to over-personalize.<br>• Even with human-selected relevant attributes, 𝟱𝟰.𝟲% of personalized responses are judged 𝗻𝗼 𝗯𝗲𝘁𝘁𝗲𝗿 𝘁𝗵𝗮𝗻 𝗴𝗲𝗻𝗲𝗿𝗶𝗰 ones by humans.<br>• LLM judges often overestimate personalization quality, sometimes rewarding surface-level attribute mentions that humans do not find useful.</p>\n<p>We also find that lightweight training improves attribute verification and relevance matching substantially. But response-level personalization remains much harder, likely because “good personalization” is inherently individual.</p>\n<p>𝗢𝘂𝗿 𝘁𝗮𝗸𝗲𝗮𝘄𝗮𝘆 𝗶𝘀 𝘀𝗶𝗺𝗽𝗹𝗲: Synthetic users and LLM judges aren't enough to capture the complex nature of human preferences. We highlight the importance of 𝗵𝘂𝗺𝗮𝗻 𝗱𝗮𝘁𝗮 and call for 𝗿𝗲-𝗰𝗲𝗻𝘁𝗲𝗿𝗶𝗻𝗴 𝗵𝘂𝗺𝗮𝗻𝘀 in LLM personalization.</p>\n","updatedAt":"2026-06-18T22:38:29.047Z","author":{"_id":"6360a4c4c579d1b2c66d4e95","avatarUrl":"/avatars/ba9fee700a4d7d42c073a9dce7d703bc.svg","fullname":"Lechen Zhang","name":"leczhang","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8681259155273438},"editors":["leczhang"],"editorAvatarUrls":["/avatars/ba9fee700a4d7d42c073a9dce7d703bc.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.06614","authors":[{"_id":"6a3473064c5c5e0d69bf1b37","name":"Lechen Zhang","hidden":false},{"_id":"6a3473064c5c5e0d69bf1b38","name":"Jiarui Liu","hidden":false},{"_id":"6a3473064c5c5e0d69bf1b39","name":"Tal August","hidden":false}],"publishedAt":"2026-06-04T00:00:00.000Z","submittedOnDailyAt":"2026-06-18T00:00:00.000Z","title":"Re-Centering Humans in LLM Personalization","submittedOnDailyBy":{"_id":"6360a4c4c579d1b2c66d4e95","avatarUrl":"/avatars/ba9fee700a4d7d42c073a9dce7d703bc.svg","isPro":false,"fullname":"Lechen Zhang","user":"leczhang","type":"user","name":"leczhang"},"summary":"Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.","upvotes":0,"discussionId":"6a3473074c5c5e0d69bf1b3a","projectPage":"https://languageinteraction.github.io/2026/06/15/recenter-personalization.html","githubRepo":"https://github.com/orange0629/recenter-personalization","githubRepoAddedBy":"user","ai_summary":"Human-centered evaluation reveals significant gaps between synthetic and real-world LLM personalization performance, with models struggling to extract user attributes and generate truly personalized responses that match human quality judgments.","ai_keywords":["large language models","personalization","synthetic data","human data","user attributes","automated personalization","reward models","human-aligned personalization"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.06614.md","query":{}}">
Re-Centering Humans in LLM Personalization
Abstract
Human-centered evaluation reveals significant gaps between synthetic and real-world LLM personalization performance, with models struggling to extract user attributes and generate truly personalized responses that match human quality judgments.
Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.
Community
Personalization is becoming a core promise of LLM systems: chatbots remember your job, interests, preferences, and past conversations to tailor responses. But “personalized” does not always mean helpful — it can also feel uncomfortable, offensive, or just unnecessary.
This raises a basic but surprisingly under-examined question: 𝗪𝗵𝗼 𝗱𝗲𝗰𝗶𝗱𝗲𝘀 𝘄𝗵𝗲𝘁𝗵𝗲𝗿 𝗽𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗵𝗲𝗹𝗽𝗳𝘂𝗹 — 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹, 𝗼𝗿 𝘁𝗵𝗲 𝗽𝗲𝗿𝘀𝗼𝗻 𝗯𝗲𝗶𝗻𝗴 𝗽𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘇𝗲𝗱 𝗳𝗼𝗿?
Most existing benchmarks rely heavily on synthetic personas, simulated conversations, and LLM judges. In this work, we put 𝗿𝗲𝗮𝗹 𝗵𝘂𝗺𝗮𝗻𝘀 back into the loop.
We study personalization as a three-stage pipeline:
🧠 𝗔𝘁𝘁𝗿𝗶𝗯𝘂𝘁𝗲 𝗲𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — what should the system infer from conversation history?
🎯 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗺𝗮𝘁𝗰𝗵𝗶𝗻𝗴 — which attributes actually matter for the current request?
✍️ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 — does personalization improve the user experience?
Using 550 real user conversations and nearly 19,000 human judgments, we find systematic 𝗵𝘂𝗺𝗮𝗻–𝗟𝗟𝗠 𝗴𝗮𝗽𝘀 at every stage:
• Models extract noisy and overgeneralized attributes from real conversations. Synthetic data underestimates this difficulty.
• LLMs and humans disagree on which attributes should be used in a new question(κ=0.30), but each agree well within their own group (κ=0.60 and 0.43).
• LLMs select 𝟮–𝟯× 𝗺𝗼𝗿𝗲 𝗮𝘁𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝘀 as relevant than humans do, suggesting a tendency to over-personalize.
• Even with human-selected relevant attributes, 𝟱𝟰.𝟲% of personalized responses are judged 𝗻𝗼 𝗯𝗲𝘁𝘁𝗲𝗿 𝘁𝗵𝗮𝗻 𝗴𝗲𝗻𝗲𝗿𝗶𝗰 ones by humans.
• LLM judges often overestimate personalization quality, sometimes rewarding surface-level attribute mentions that humans do not find useful.
We also find that lightweight training improves attribute verification and relevance matching substantially. But response-level personalization remains much harder, likely because “good personalization” is inherently individual.
𝗢𝘂𝗿 𝘁𝗮𝗸𝗲𝗮𝘄𝗮𝘆 𝗶𝘀 𝘀𝗶𝗺𝗽𝗹𝗲: Synthetic users and LLM judges aren't enough to capture the complex nature of human preferences. We highlight the importance of 𝗵𝘂𝗺𝗮𝗻 𝗱𝗮𝘁𝗮 and call for 𝗿𝗲-𝗰𝗲𝗻𝘁𝗲𝗿𝗶𝗻𝗴 𝗵𝘂𝗺𝗮𝗻𝘀 in LLM personalization.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.06614 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.06614 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.06614 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.