Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.</p>\n","updatedAt":"2026-09-17T05:20:44.844Z","author":{"_id":"678323cb4bd851a06acb936f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/amD4z7Yr6mCh-mQ0_Rnb1.png","fullname":"Peter L. Chen","name":"PeterLauLukCh","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.870055615901947},"editors":["PeterLauLukCh"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/amD4z7Yr6mCh-mQ0_Rnb1.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.19144","authors":[{"_id":"6aab78771d9cc4dec796253d","user":{"_id":"678323cb4bd851a06acb936f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/amD4z7Yr6mCh-mQ0_Rnb1.png","isPro":false,"fullname":"Peter L. Chen","user":"PeterLauLukCh","type":"user","name":"PeterLauLukCh"},"name":"Peter Chen","status":"claimed_verified","statusLastChangedAt":"2026-09-17T08:45:04.522Z","hidden":false},{"_id":"6aab78771d9cc4dec796253e","name":"Xi Chen","hidden":false},{"_id":"6aab78771d9cc4dec796253f","name":"Wotao Yin","hidden":false},{"_id":"6aab78771d9cc4dec7962540","name":"Tianyi Lin","hidden":false}],"publishedAt":"2026-09-16T00:00:00.000Z","submittedOnDailyAt":"2026-09-17T00:00:00.000Z","title":"A Zeroth-Order Paradigm for LLM Preference Alignment","submittedOnDailyBy":{"_id":"678323cb4bd851a06acb936f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/amD4z7Yr6mCh-mQ0_Rnb1.png","isPro":false,"fullname":"Peter L. Chen","user":"PeterLauLukCh","type":"user","name":"PeterLauLukCh"},"summary":"Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.","upvotes":22,"discussionId":"6aab78771d9cc4dec7962541","organization":{"_id":"61f20a9ce108f2cba2dc0730","name":"Berkeley","fullname":"UC Berkeley","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/61ac8f8a00d01045fca0ad2f/0FjsTg2txEZZ4dEgmMnQL.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"678323cb4bd851a06acb936f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/amD4z7Yr6mCh-mQ0_Rnb1.png","isPro":false,"fullname":"Peter L. Chen","user":"PeterLauLukCh","type":"user"},{"_id":"6944e2acd5b791cc5c6d13d1","avatarUrl":"/avatars/d6c5846df0b16effbd295c43a3d6be27.svg","isPro":false,"fullname":"Jessica Hsu","user":"JessicaHsuPF","type":"user"},{"_id":"6a6a81ef4c287dbb8e7ea111","avatarUrl":"/avatars/b360809862f7e66d9662afd87c2e488a.svg","isPro":false,"fullname":"Patricia Smith","user":"patricia-smith","type":"user"},{"_id":"6a6aa1c6557526be22f38a09","avatarUrl":"/avatars/8677177172aaef08cbc5b35680dce73a.svg","isPro":false,"fullname":"Steven Hernandez","user":"orbitglade","type":"user"},{"_id":"6a6c7bd216dedbf22d783e7d","avatarUrl":"/avatars/617d078df22f35ba26a20acebe23ecea.svg","isPro":false,"fullname":"Linda Wilson","user":"Linda-Wilson","type":"user"},{"_id":"6a6c87bb626b1eb992a0b3d9","avatarUrl":"/avatars/4d91bcdf74525cd0d1a5f083e12a6175.svg","isPro":false,"fullname":"Susan Harris","user":"frostglade","type":"user"},{"_id":"6a6c8bcfa63d5d5e50328012","avatarUrl":"/avatars/b60feffe43971725b2da9ae9508f4896.svg","isPro":false,"fullname":"Linda Garcia","user":"Granite-Linda","type":"user"},{"_id":"6a6da95b346df769d90cc578","avatarUrl":"/avatars/d8fb17b7c3265f6df50052a620fa774f.svg","isPro":false,"fullname":"Daniel Clark","user":"Nova-Noah","type":"user"},{"_id":"6a6dc6d49caa2ff7db786e2a","avatarUrl":"/avatars/efdc260d9bb75fcf331f574b1d1efbe6.svg","isPro":false,"fullname":"Jessica Gonzalez","user":"Azure-Rune","type":"user"},{"_id":"6a6dea9b6d51d01100d751c5","avatarUrl":"/avatars/547b0eedadc3d76f7cf1450e527f09b9.svg","isPro":false,"fullname":"Patricia Davis","user":"Patricia-Davis","type":"user"},{"_id":"6a701e86e8606187f1abfa0e","avatarUrl":"/avatars/498bf3741a4a33321515181ac52b57c5.svg","isPro":false,"fullname":"Richard Lee","user":"indigokite","type":"user"},{"_id":"6a826277ddd7c82d59300c83","avatarUrl":"/avatars/9172975ed7e5a1f6bfa291397b8575c9.svg","isPro":false,"fullname":"William Lewis","user":"William-Lewis","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"61f20a9ce108f2cba2dc0730","name":"Berkeley","fullname":"UC Berkeley","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/61ac8f8a00d01045fca0ad2f/0FjsTg2txEZZ4dEgmMnQL.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.19144.md","query":{}}">
A Zeroth-Order Paradigm for LLM Preference Alignment
Abstract
Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.
Community
Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2609.19144 in a model README.md to link it from this page.
Cite arxiv.org/abs/2609.19144 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2609.19144 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.