\nOverview of RevPropBench. The upper half shows the benchmark construction process, while the lower half shows the evaluation process. Samples are generated via LLM-based synthetic sampling and human annotation.\n</p>\n\nLinks:\narXiv: https://arxiv.org/abs/2609.03254\nalphaXiv: https://www.alphaxiv.org/abs/2609.03254\nCode & dataset: https://github.com/ntt-dkiku/llm-revision-propagation","html":"<p>In this paper, we propose RevPropBench, a benchmark for evaluating the ability of LLMs to propagate revisions across dependent elements in conversationally generated JSON artifacts when users specify only a local change. We also explore cost-effective test-time compute for this setting.</p>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/6528f3a590f06dd8a8e97572/JDCVSn3OHwIBQk1IWY1Fm.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/6528f3a590f06dd8a8e97572/JDCVSn3OHwIBQk1IWY1Fm.png\" alt=\"revpropbench-overview\"></a></p>\n<p>\nOverview of RevPropBench. The upper half shows the benchmark construction process, while the lower half shows the evaluation process. Samples are generated via LLM-based synthetic sampling and human annotation.\n</p>\n\n<p>Links:<br>arXiv: <a href=\"https://arxiv.org/abs/2609.03254\" rel=\"nofollow\">https://arxiv.org/abs/2609.03254</a><br>alphaXiv: <a href=\"https://www.alphaxiv.org/abs/2609.03254\" rel=\"nofollow\">https://www.alphaxiv.org/abs/2609.03254</a><br>Code &amp; dataset: <a href=\"https://github.com/ntt-dkiku/llm-revision-propagation\" rel=\"nofollow\">https://github.com/ntt-dkiku/llm-revision-propagation</a></p>\n","updatedAt":"2026-09-08T02:44:12.773Z","author":{"_id":"6528f3a590f06dd8a8e97572","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6528f3a590f06dd8a8e97572/_-mnMzoN6ouN7B_qzb3ab.png","fullname":"Daisuke Kikuta","name":"oookiku","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7212276458740234},"editors":["oookiku"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6528f3a590f06dd8a8e97572/_-mnMzoN6ouN7B_qzb3ab.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.03254","authors":[{"_id":"6a9f75476c8e10537d563b16","name":"Daisuke Kikuta","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/6528f3a590f06dd8a8e97572/gx2RwSMBk1036GnnBCjAn.png","https://cdn-uploads.huggingface.co/production/uploads/6528f3a590f06dd8a8e97572/i9BTLlFl2wLIW4e8VTC5F.png","https://cdn-uploads.huggingface.co/production/uploads/6528f3a590f06dd8a8e97572/5CV7adsjJ-MnKaE29fyJi.png"],"publishedAt":"2026-09-03T00:00:00.000Z","submittedOnDailyAt":"2026-09-08T00:00:00.000Z","title":"What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation","submittedOnDailyBy":{"_id":"6528f3a590f06dd8a8e97572","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6528f3a590f06dd8a8e97572/_-mnMzoN6ouN7B_qzb3ab.png","isPro":false,"fullname":"Daisuke Kikuta","user":"oookiku","type":"user","name":"oookiku"},"summary":"Large Language Models (LLMs) often help users generate artifacts through iterative cycles of generation and revision in conversation. A challenge here is that, when users specify only a local change during revision, LLMs must instead identify the relevant dependencies and propagate the revision to all affected parts of the artifact. This paper studies this ability of LLMs on conversationally generated artifacts, where the artifact context and its dependencies may be embedded in the conversation history. Toward practical use, we also explore cost-effective test-time compute for this new setting. Specifically, we introduce a new benchmark for this setting, and evaluate nine revision methods, including sequential reflection and parallel sampling variants, using gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b on the benchmark. The results show that baselines achieve accuracies of 68.3--93%, and the most cost-effective method is selecting from three parallel samples using either LLM-based or medoid selection, which improves accuracy by 2.2--9.7%. Our code and dataset are available at https://github.com/ntt-dkiku/llm-revision-propagation.","upvotes":1,"discussionId":"6a9f75486c8e10537d563b17","githubRepo":"https://github.com/ntt-dkiku/llm-revision-propagation","githubRepoAddedBy":"user","githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6528f3a590f06dd8a8e97572","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6528f3a590f06dd8a8e97572/_-mnMzoN6ouN7B_qzb3ab.png","isPro":false,"fullname":"Daisuke Kikuta","user":"oookiku","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.03254.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2609.03254 in a model README.md to link it from this page.

Cite arxiv.org/abs/2609.03254 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2609.03254 in a Space README.md to link it from this page.