The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-lingual Ranking Preference Optimization~(CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment. Our code is available at <a href=\"https://github.com/dltmddbs100/CRPO\" rel=\"nofollow\">https://github.com/dltmddbs100/CRPO</a>.</p>\n","updatedAt":"2026-08-31T09:24:05.993Z","author":{"_id":"65d74465aa55e4dc9a3555bd","avatarUrl":"/avatars/75b14f7141d04fc438e63ebc1a0cc14b.svg","fullname":"Seungyoon Lee","name":"nobrand","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8816365003585815},"editors":["nobrand"],"editorAvatarUrls":["/avatars/75b14f7141d04fc438e63ebc1a0cc14b.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.23149","authors":[{"_id":"6a9547f7073195fee5157397","name":"Seungyoon Lee","hidden":false},{"_id":"6a9547f7073195fee5157398","name":"Minhyuk Kim","hidden":false},{"_id":"6a9547f7073195fee5157399","name":"Jungseob Lee","hidden":false},{"_id":"6a9547f7073195fee515739a","name":"Heuiseok Lim","hidden":false}],"publishedAt":"2026-08-24T11:54:36.000Z","submittedOnDailyAt":"2026-08-31T00:00:00.000Z","title":"Language Chain in Alignment: Cross-lingual Ranking Preference Optimization","submittedOnDailyBy":{"_id":"65d74465aa55e4dc9a3555bd","avatarUrl":"/avatars/75b14f7141d04fc438e63ebc1a0cc14b.svg","isPro":false,"fullname":"Seungyoon Lee","user":"nobrand","type":"user","name":"nobrand"},"summary":"The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-lingual Ranking Preference Optimization~(CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment. Our code is available at https://github.com/dltmddbs100/CRPO.","upvotes":3,"discussionId":"6a9547f7073195fee515739b","githubRepo":"https://github.com/dltmddbs100/CRPO","githubRepoAddedBy":"user","ai_summary":"Cross-lingual Ranking Preference Optimization improves multilingual alignment by transferring English preference knowledge to target languages through hierarchical ranking optimization.","ai_keywords":["Cross-lingual Ranking Preference Optimization","LambdaLoss","intra-lingual preferences","inter-lingual preferences","reward margins","preference manifold","cross-lingual alignment"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":0,"organization":{"_id":"647733b8fe51edaf1fc1382f","name":"nlpai-lab","fullname":"NLP & AI - Korea University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/647731f32573014e9544a93c/daQvS1iSNYvaW34B0lp3z.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"65d74465aa55e4dc9a3555bd","avatarUrl":"/avatars/75b14f7141d04fc438e63ebc1a0cc14b.svg","isPro":false,"fullname":"Seungyoon Lee","user":"nobrand","type":"user"},{"_id":"631c386bc73939ffc0716a37","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1662793811119-noauth.jpeg","isPro":false,"fullname":"SeongWan Kim","user":"idgmatrix","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"647733b8fe51edaf1fc1382f","name":"nlpai-lab","fullname":"NLP & AI - Korea University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/647731f32573014e9544a93c/daQvS1iSNYvaW34B0lp3z.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.23149.md","query":{}}">
Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
Abstract
Cross-lingual Ranking Preference Optimization improves multilingual alignment by transferring English preference knowledge to target languages through hierarchical ranking optimization.
The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-lingual Ranking Preference Optimization~(CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment. Our code is available at https://github.com/dltmddbs100/CRPO.
Community
The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-lingual Ranking Preference Optimization~(CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment. Our code is available at https://github.com/dltmddbs100/CRPO.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.23149 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.23149 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.23149 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.