We just released the Last Translation Benchmark paper. In a massive crowdsourcing effort we collected 3456 unique hard-to-translate examples that break state-of-the-art translation models, and which can be used for more reliable evaluation. Machine translation doesn't break on just figurative language as one would expect. In fact, for the next generation of models, we may need to invest heavily into multilingual (&amp; cultural) reasoning. Last Translation Benchmark is a live paper+dataset and you can still join. :-)</p>\n","updatedAt":"2026-09-04T14:48:29.330Z","author":{"_id":"6304ece07424d937fa35fb98","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6304ece07424d937fa35fb98/Ct4fi1SpLUwhrGDYsAcVB.jpeg","fullname":"Vilém Zouhar","name":"zouhar","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":44,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9266148209571838},"editors":["zouhar"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6304ece07424d937fa35fb98/Ct4fi1SpLUwhrGDYsAcVB.jpeg"],"reactions":[{"reaction":"🔥","users":["zetrozky","hannayukhymenko"],"count":2}],"isReport":false}},{"id":"6a9b6e8940c0f05036f2193c","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":379,"isUserFollowing":false},"createdAt":"2026-09-05T01:21:13.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Source-Free MT Evaluation Is Not MT Evaluation](https://huggingface.co/papers/2608.20925) (2026)\n* [Augmenting Text to Increase Translation Difficulty](https://huggingface.co/papers/2608.15932) (2026)\n* [TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation](https://huggingface.co/papers/2608.02975) (2026)\n* [Beyond Blind Compliance: Benchmarking Task Verification in OCR Reasoning](https://huggingface.co/papers/2609.00232) (2026)\n* [Do Evaluation Metrics Detect Errors in Classical Chinese to English Translations?](https://huggingface.co/papers/2608.08283) (2026)\n* [RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists](https://huggingface.co/papers/2608.04783) (2026)\n* [AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification](https://huggingface.co/papers/2607.11849) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"<p>This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. </p>\n<p>The following papers were recommended by the Semantic Scholar API </p>\n<ul>\n<li><a href=\"https://huggingface.co/papers/2608.20925\">Source-Free MT Evaluation Is Not MT Evaluation</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.15932\">Augmenting Text to Increase Translation Difficulty</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.02975\">TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2609.00232\">Beyond Blind Compliance: Benchmarking Task Verification in OCR Reasoning</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.08283\">Do Evaluation Metrics Detect Errors in Classical Chinese to English Translations?</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.04783\">RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.11849\">AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification</a> (2026)</li>\n</ul>\n<p> Please give a thumbs up to this comment if you found it helpful!</p>\n<p> If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space</p>\n<p> You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code>@librarian-bot recommend</code></p>\n","updatedAt":"2026-09-05T01:21:13.392Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":379,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7309603691101074},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.04173","authors":[{"_id":"6a9a50c58f7c3b755723956a","user":{"_id":"6304ece07424d937fa35fb98","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6304ece07424d937fa35fb98/Ct4fi1SpLUwhrGDYsAcVB.jpeg","isPro":false,"fullname":"Vilém Zouhar","user":"zouhar","type":"user","name":"zouhar"},"name":"Vilém Zouhar","status":"claimed_verified","statusLastChangedAt":"2026-09-04T08:45:04.305Z","hidden":false},{"_id":"6a9a50c58f7c3b755723956b","name":"Niyati Bafna","hidden":false},{"_id":"6a9a50c58f7c3b755723956c","name":"Mukund Choudhary","hidden":false},{"_id":"6a9a50c58f7c3b755723956d","name":"Maike Züfle","hidden":false},{"_id":"6a9a50c58f7c3b755723956e","name":"Sara Rajaee","hidden":false},{"_id":"6a9a50c58f7c3b755723956f","user":{"_id":"641248d400634c4fe98589a8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/641248d400634c4fe98589a8/edYgAz2j_3F4B1h-0Ma6o.png","isPro":false,"fullname":"Pinzhen Chen","user":"pinzhenchen","type":"user","name":"pinzhenchen"},"name":"Pinzhen Chen","status":"claimed_verified","statusLastChangedAt":"2026-09-05T00:45:04.085Z","hidden":false},{"_id":"6a9a50c58f7c3b7557239570","name":"Jannis Vamvas","hidden":false},{"_id":"6a9a50c58f7c3b7557239571","name":"Sara Papi","hidden":false},{"_id":"6a9a50c58f7c3b7557239572","name":"Ona de Gibert","hidden":false},{"_id":"6a9a50c58f7c3b7557239573","name":"Bhavitvya Malik","hidden":false},{"_id":"6a9a50c58f7c3b7557239574","name":"Eliya Habba","hidden":false},{"_id":"6a9a50c58f7c3b7557239575","name":"Orfeas Menis Mastromichalakis","hidden":false},{"_id":"6a9a50c58f7c3b7557239576","name":"Patrícia Schmidtová","hidden":false},{"_id":"6a9a50c58f7c3b7557239577","name":"Michelle Wastl","hidden":false},{"_id":"6a9a50c58f7c3b7557239578","name":"Sheriff Issaka","hidden":false},{"_id":"6a9a50c58f7c3b7557239579","name":"Leshem Choshen","hidden":false},{"_id":"6a9a50c58f7c3b755723957a","name":"Stella Biderman","hidden":false},{"_id":"6a9a50c58f7c3b755723957b","name":"Antonis Anastasopoulos","hidden":false},{"_id":"6a9a50c58f7c3b755723957c","name":"Jan Niehues","hidden":false},{"_id":"6a9a50c58f7c3b755723957d","name":"Rico Sennrich","hidden":false},{"_id":"6a9a50c58f7c3b755723957e","name":"Mrinmaya Sachan","hidden":false},{"_id":"6a9a50c58f7c3b755723957f","name":"Ondřej Bojar","hidden":false},{"_id":"6a9a50c58f7c3b7557239580","name":"Kenton Murray","hidden":false},{"_id":"6a9a50c58f7c3b7557239581","name":"Jörg Tiedemann","hidden":false},{"_id":"6a9a50c58f7c3b7557239582","name":"Alham Fikri Aji","hidden":false},{"_id":"6a9a50c58f7c3b7557239583","name":"Philipp Koehn","hidden":false},{"_id":"6a9a50c58f7c3b7557239584","name":"Christof Monz","hidden":false},{"_id":"6a9a50c58f7c3b7557239585","name":"Alexandra Birch","hidden":false},{"_id":"6a9a50c58f7c3b7557239586","name":"Sowmya Vajjala","hidden":false},{"_id":"6a9a50c58f7c3b7557239587","name":"Chalamalasetti Kranti","hidden":false},{"_id":"6a9a50c58f7c3b7557239588","name":"Cristina España-Bonet","hidden":false},{"_id":"6a9a50c58f7c3b7557239589","name":"Nobin Sarwar","hidden":false},{"_id":"6a9a50c58f7c3b755723958a","name":"David Kaczér","hidden":false},{"_id":"6a9a50c58f7c3b755723958b","name":"Shunta Asano","hidden":false},{"_id":"6a9a50c58f7c3b755723958c","name":"Malik Marmonier","hidden":false},{"_id":"6a9a50c58f7c3b755723958d","name":"Daban Q. Jaff","hidden":false},{"_id":"6a9a50c58f7c3b755723958e","name":"Vaisakhi Mishra","hidden":false},{"_id":"6a9a50c58f7c3b755723958f","name":"Hend Al- Khalifa","hidden":false},{"_id":"6a9a50c58f7c3b7557239590","name":"Gabriele Sarti","hidden":false},{"_id":"6a9a50c58f7c3b7557239591","name":"Sourajit Saha","hidden":false},{"_id":"6a9a50c58f7c3b7557239592","name":"Nils Rehlinger","hidden":false},{"_id":"6a9a50c58f7c3b7557239593","name":"Juan Daniel Cuervo Villa","hidden":false},{"_id":"6a9a50c58f7c3b7557239594","name":"Jonathan Tonglet","hidden":false},{"_id":"6a9a50c58f7c3b7557239595","name":"Saugata Purkayastha","hidden":false},{"_id":"6a9a50c58f7c3b7557239596","name":"Dominik Macháček","hidden":false},{"_id":"6a9a50c58f7c3b7557239597","name":"Jagannathan Ramanujam","hidden":false},{"_id":"6a9a50c58f7c3b7557239598","name":"Heejin Do","hidden":false},{"_id":"6a9a50c58f7c3b7557239599","name":"Zuzana Nadova","hidden":false},{"_id":"6a9a50c58f7c3b755723959a","name":"Fred Philippy","hidden":false},{"_id":"6a9a50c58f7c3b755723959b","name":"Fabian Retkowski","hidden":false},{"_id":"6a9a50c58f7c3b755723959c","name":"Maria Lymperaiou","hidden":false},{"_id":"6a9a50c58f7c3b755723959d","name":"Silvia Casola","hidden":false},{"_id":"6a9a50c58f7c3b755723959e","user":{"_id":"650ed7adf141bc34f91a12ae","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/rZDwRiBcqeqlUQ7mThoyU.jpeg","isPro":true,"fullname":"Hanna Yukhymenko","user":"hannayukhymenko","type":"user","name":"hannayukhymenko"},"name":"Hanna Yukhymenko","status":"claimed_verified","statusLastChangedAt":"2026-09-04T08:45:04.322Z","hidden":false},{"_id":"6a9a50c58f7c3b755723959f","name":"Shubhashis Roy Dipta","hidden":false},{"_id":"6a9a50c58f7c3b75572395a0","name":"Sangwon Ryu","hidden":false},{"_id":"6a9a50c58f7c3b75572395a1","name":"Andrés Jerez","hidden":false},{"_id":"6a9a50c58f7c3b75572395a2","name":"Ron Keinan","hidden":false},{"_id":"6a9a50c58f7c3b75572395a3","name":"Shuaib Shuaib Yusuf","hidden":false},{"_id":"6a9a50c58f7c3b75572395a4","user":{"_id":"6a9a52ae2ad6e182820b4b25","avatarUrl":"/avatars/01c874903fc4af46b3b9bf57595bb288.svg","isPro":false,"fullname":"Avantica Vempati","user":"zetrozky","type":"user","name":"zetrozky"},"name":"Avantica Vempati","status":"claimed_verified","statusLastChangedAt":"2026-09-04T08:45:04.311Z","hidden":false},{"_id":"6a9a50c58f7c3b75572395a5","name":"Maria Carmen Staiano","hidden":false},{"_id":"6a9a50c58f7c3b75572395a6","name":"Sukannya Purkayastha","hidden":false},{"_id":"6a9a50c58f7c3b75572395a7","name":"Adrian Cosma","hidden":false},{"_id":"6a9a50c58f7c3b75572395a8","name":"Vitalii Babenko","hidden":false},{"_id":"6a9a50c58f7c3b75572395a9","name":"Erivan Inan","hidden":false},{"_id":"6a9a50c58f7c3b75572395aa","name":"Aviral Nigam","hidden":false},{"_id":"6a9a50c58f7c3b75572395ab","name":"Wafa Aissa","hidden":false},{"_id":"6a9a50c58f7c3b75572395ac","name":"Fatima Haouari","hidden":false},{"_id":"6a9a50c58f7c3b75572395ad","name":"Venkata Prasanth Kumar Gummadi","hidden":false},{"_id":"6a9a50c58f7c3b75572395ae","name":"Mehdi Jafarzadeh","hidden":false},{"_id":"6a9a50c58f7c3b75572395af","name":"Valentin Scourneau","hidden":false},{"_id":"6a9a50c58f7c3b75572395b0","name":"Lukas Edman","hidden":false},{"_id":"6a9a50c58f7c3b75572395b1","name":"Kaiser Sun","hidden":false},{"_id":"6a9a50c58f7c3b75572395b2","name":"Shaomu Tan","hidden":false},{"_id":"6a9a50c58f7c3b75572395b3","name":"Mohammad Sadegh Gholizadeh","hidden":false},{"_id":"6a9a50c58f7c3b75572395b4","name":"Johannes-Rudolf David","hidden":false},{"_id":"6a9a50c58f7c3b75572395b5","name":"Dipankar Srirag","hidden":false},{"_id":"6a9a50c58f7c3b75572395b6","name":"Javier García Gilabert","hidden":false},{"_id":"6a9a50c58f7c3b75572395b7","name":"Ruta Binkyte","hidden":false},{"_id":"6a9a50c58f7c3b75572395b8","name":"Manar Ali","hidden":false},{"_id":"6a9a50c58f7c3b75572395b9","name":"Ana-Maria Bucur","hidden":false},{"_id":"6a9a50c58f7c3b75572395ba","name":"Sabry E. Farrag","hidden":false},{"_id":"6a9a50c58f7c3b75572395bb","name":"Youssef Saber","hidden":false},{"_id":"6a9a50c58f7c3b75572395bc","name":"Yihong Liu","hidden":false},{"_id":"6a9a50c58f7c3b75572395bd","name":"Jean Maillard","hidden":false},{"_id":"6a9a50c58f7c3b75572395be","name":"Cojocaru Nicoleta","hidden":false},{"_id":"6a9a50c58f7c3b75572395bf","name":"Xiaochuang Yuan","hidden":false},{"_id":"6a9a50c58f7c3b75572395c0","name":"Sina Ahmadi","hidden":false},{"_id":"6a9a50c58f7c3b75572395c1","name":"Philipp Mondorf","hidden":false},{"_id":"6a9a50c58f7c3b75572395c2","name":"Kaustubh Dhole","hidden":false},{"_id":"6a9a50c58f7c3b75572395c3","name":"Roman Wixinger","hidden":false},{"_id":"6a9a50c58f7c3b75572395c4","name":"Shenbin Qian","hidden":false},{"_id":"6a9a50c58f7c3b75572395c5","name":"Manuel Tuor","hidden":false},{"_id":"6a9a50c58f7c3b75572395c6","name":"Sergey Troshin","hidden":false},{"_id":"6a9a50c58f7c3b75572395c7","name":"Jonathan Yahav","hidden":false},{"_id":"6a9a50c58f7c3b75572395c8","name":"Fida Mohammad Thoker","hidden":false},{"_id":"6a9a50c58f7c3b75572395c9","name":"Amir Arsalan Rezapour","hidden":false},{"_id":"6a9a50c58f7c3b75572395ca","name":"Lance Calvin Lim Gamboa","hidden":false},{"_id":"6a9a50c58f7c3b75572395cb","name":"Manon Reusens","hidden":false},{"_id":"6a9a50c58f7c3b75572395cc","name":"Kätriin Kukk","hidden":false},{"_id":"6a9a50c58f7c3b75572395cd","name":"Koel Dutta Chowdhury","hidden":false},{"_id":"6a9a50c58f7c3b75572395ce","name":"Giuseppe Gallipoli","hidden":false},{"_id":"6a9a50c58f7c3b75572395cf","name":"Christian Hoang","hidden":false},{"_id":"6a9a50c58f7c3b75572395d0","name":"Shaswati Saha","hidden":false},{"_id":"6a9a50c58f7c3b75572395d1","name":"Seth Aycock","hidden":false},{"_id":"6a9a50c58f7c3b75572395d2","name":"Jan Kocoń","hidden":false},{"_id":"6a9a50c58f7c3b75572395d3","name":"Bo Chen","hidden":false},{"_id":"6a9a50c58f7c3b75572395d4","name":"Linh Vu","hidden":false},{"_id":"6a9a50c58f7c3b75572395d5","name":"Vatsal Venkatkrishna","hidden":false},{"_id":"6a9a50c58f7c3b75572395d6","name":"Arafat Ahsan","hidden":false},{"_id":"6a9a50c58f7c3b75572395d7","user":{"_id":"60b870571e5fdaee9dd52193","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60b870571e5fdaee9dd52193/VvwQQfL-FWzoXVwLKl6GZ.jpeg","isPro":false,"fullname":"Luan Thanh Nguyen","user":"tarudesu","type":"user","name":"tarudesu"},"name":"Luan Thanh Nguyen","status":"claimed_verified","statusLastChangedAt":"2026-09-05T00:45:04.091Z","hidden":false},{"_id":"6a9a50c58f7c3b75572395d8","name":"Hassan Soliman","hidden":false},{"_id":"6a9a50c58f7c3b75572395d9","name":"Daryna Dementieva","hidden":false},{"_id":"6a9a50c58f7c3b75572395da","user":{"_id":"66fa381d94df3bc9dd23a991","avatarUrl":"/avatars/dbbf67f27593a510177dff06b00bf098.svg","isPro":false,"fullname":"TVR","user":"theresiavr","type":"user","name":"theresiavr"},"name":"Theresia Veronika Rampisela","status":"claimed_verified","statusLastChangedAt":"2026-09-04T12:56:39.291Z","hidden":false},{"_id":"6a9a50c58f7c3b75572395db","name":"Ngoc Quynh Tram Do","hidden":false},{"_id":"6a9a50c58f7c3b75572395dc","name":"Marius Huber","hidden":false},{"_id":"6a9a50c58f7c3b75572395dd","name":"Kazuki Egashira","hidden":false},{"_id":"6a9a50c58f7c3b75572395de","user":{"_id":"627608ec624e6576d693c05c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1677077880239-627608ec624e6576d693c05c.png","isPro":false,"fullname":"Azmine Toushik Wasi","user":"azminetoushikwasi","type":"user","name":"azminetoushikwasi"},"name":"Azmine Toushik Wasi","status":"claimed_verified","statusLastChangedAt":"2026-09-04T16:45:04.048Z","hidden":false},{"_id":"6a9a50c58f7c3b75572395df","name":"Vladislav Poritski","hidden":false},{"_id":"6a9a50c58f7c3b75572395e0","user":{"_id":"60d33fbbd7b174177faabd4f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60d33fbbd7b174177faabd4f/pfyv_xj2B2m2N4F4sT9zJ.jpeg","isPro":true,"fullname":"Mike Zhang","user":"jjzha","type":"user","name":"jjzha"},"name":"Mike Zhang","status":"claimed_verified","statusLastChangedAt":"2026-09-04T08:45:04.334Z","hidden":false},{"_id":"6a9a50c58f7c3b75572395e1","name":"Deep Shah","hidden":false},{"_id":"6a9a50c58f7c3b75572395e2","name":"Paul Gavrikov","hidden":false},{"_id":"6a9a50c58f7c3b75572395e3","name":"Luis Frentzen Salim","hidden":false},{"_id":"6a9a50c58f7c3b75572395e4","name":"David Africa","hidden":false},{"_id":"6a9a50c58f7c3b75572395e5","name":"R. Damanhuri","hidden":false},{"_id":"6a9a50c58f7c3b75572395e6","name":"Bello Umar Bello","hidden":false},{"_id":"6a9a50c58f7c3b75572395e7","name":"Anumit Garg","hidden":false},{"_id":"6a9a50c58f7c3b75572395e8","name":"Gengyu Rao","hidden":false},{"_id":"6a9a50c58f7c3b75572395e9","name":"Pawan Sasanka Ammanamanchi","hidden":false},{"_id":"6a9a50c58f7c3b75572395ea","name":"Kamile Dementaviciute","hidden":false},{"_id":"6a9a50c58f7c3b75572395eb","name":"Andrianos Michail","hidden":false},{"_id":"6a9a50c58f7c3b75572395ec","name":"L D M S Sai Teja","hidden":false},{"_id":"6a9a50c58f7c3b75572395ed","name":"Dawei Zhu","hidden":false},{"_id":"6a9a50c58f7c3b75572395ee","name":"Yi Fan","hidden":false},{"_id":"6a9a50c58f7c3b75572395ef","user":{"_id":"67a62f8e47fc008de27889ea","avatarUrl":"/avatars/0b3cba784863962b1cc10d99de810d59.svg","isPro":false,"fullname":"Wei Liu","user":"willie1206","type":"user","name":"willie1206"},"name":"Wei Liu","status":"claimed_verified","statusLastChangedAt":"2026-09-04T08:45:04.317Z","hidden":false},{"_id":"6a9a50c58f7c3b75572395f0","name":"Farhan Farsi","hidden":false},{"_id":"6a9a50c58f7c3b75572395f1","name":"Elias Herranen","hidden":false},{"_id":"6a9a50c58f7c3b75572395f2","name":"Sankalan Pal Chowdhury","hidden":false},{"_id":"6a9a50c58f7c3b75572395f3","name":"Karen Sanchez","hidden":false},{"_id":"6a9a50c58f7c3b75572395f4","name":"Farzad Shami","hidden":false},{"_id":"6a9a50c58f7c3b75572395f5","name":"Ashok Urlana","hidden":false},{"_id":"6a9a50c58f7c3b75572395f6","name":"Zimu Wang","hidden":false},{"_id":"6a9a50c58f7c3b75572395f7","name":"Tomasz Limisiewicz","hidden":false},{"_id":"6a9a50c58f7c3b75572395f8","name":"Priyaranjan Pattnayak","hidden":false},{"_id":"6a9a50c58f7c3b75572395f9","name":"Marii Ojastu","hidden":false},{"_id":"6a9a50c58f7c3b75572395fa","name":"Hongbin Na","hidden":false},{"_id":"6a9a50c58f7c3b75572395fb","name":"Emilian Radoi","hidden":false},{"_id":"6a9a50c58f7c3b75572395fc","name":"Chenyi Zhao","hidden":false},{"_id":"6a9a50c58f7c3b75572395fd","name":"Carlos Hinojosa","hidden":false},{"_id":"6a9a50c58f7c3b75572395fe","name":"Andrea Gregor de Varda","hidden":false},{"_id":"6a9a50c58f7c3b75572395ff","name":"Zaid Alyafeai","hidden":false},{"_id":"6a9a50c58f7c3b7557239600","name":"Reem Alzahrani","hidden":false},{"_id":"6a9a50c58f7c3b7557239601","name":"Nehal Kathrotia","hidden":false},{"_id":"6a9a50c58f7c3b7557239602","name":"Alex Flückiger","hidden":false},{"_id":"6a9a50c58f7c3b7557239603","name":"Ulysses Sekai Tully Carr","hidden":false},{"_id":"6a9a50c58f7c3b7557239604","name":"Jimson Paulo Layacan","hidden":false},{"_id":"6a9a50c58f7c3b7557239605","name":"Guy Kaplan","hidden":false},{"_id":"6a9a50c58f7c3b7557239606","name":"Ritwik Tiwari","hidden":false},{"_id":"6a9a50c58f7c3b7557239607","name":"Rishit Dagli","hidden":false},{"_id":"6a9a50c58f7c3b7557239608","name":"Oksana Volchek","hidden":false},{"_id":"6a9a50c58f7c3b7557239609","name":"Isaac R Caswell","hidden":false},{"_id":"6a9a50c58f7c3b755723960a","name":"Bowen Yi","hidden":false},{"_id":"6a9a50c58f7c3b755723960b","name":"Blanka Kövér","hidden":false},{"_id":"6a9a50c58f7c3b755723960c","name":"Amir Hossein Yari","hidden":false},{"_id":"6a9a50c58f7c3b755723960d","name":"Aicha Chorana","hidden":false},{"_id":"6a9a50c58f7c3b755723960e","name":"Zhengxiang Wang","hidden":false},{"_id":"6a9a50c58f7c3b755723960f","name":"Selja Keränen","hidden":false},{"_id":"6a9a50c58f7c3b7557239610","name":"Samuel Simko","hidden":false},{"_id":"6a9a50c58f7c3b7557239611","name":"Joy Olusanya","hidden":false},{"_id":"6a9a50c58f7c3b7557239612","name":"Jenny Chim","hidden":false},{"_id":"6a9a50c58f7c3b7557239613","name":"Enzo Doyen","hidden":false},{"_id":"6a9a50c58f7c3b7557239614","name":"Vivek Harsha Lakkamaneni","hidden":false},{"_id":"6a9a50c58f7c3b7557239615","name":"Sophia Conrad","hidden":false},{"_id":"6a9a50c58f7c3b7557239616","name":"Pouya Sadeghi","hidden":false},{"_id":"6a9a50c58f7c3b7557239617","name":"Panayiotis Panayiotou","hidden":false},{"_id":"6a9a50c58f7c3b7557239618","name":"Luis Lara","hidden":false},{"_id":"6a9a50c58f7c3b7557239619","name":"Jannatul Nayem","hidden":false},{"_id":"6a9a50c58f7c3b755723961a","name":"Eran Yahav","hidden":false},{"_id":"6a9a50c58f7c3b755723961b","name":"Debanshu Das","hidden":false},{"_id":"6a9a50c58f7c3b755723961c","name":"Antonia Karamolegkou","hidden":false},{"_id":"6a9a50c58f7c3b755723961d","name":"Anmol Goel","hidden":false},{"_id":"6a9a50c58f7c3b755723961e","name":"Aishik Mandal","hidden":false},{"_id":"6a9a50c58f7c3b755723961f","user":{"_id":"65044901221035b9ee187178","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65044901221035b9ee187178/o2LTLLIoT3T17Y4ZZZy_x.jpeg","isPro":false,"fullname":"Tommaso Cerruti","user":"Cerru02","type":"user","name":"Cerru02"},"name":"Tommaso Cerruti","status":"claimed_verified","statusLastChangedAt":"2026-09-04T08:45:04.327Z","hidden":false},{"_id":"6a9a50c58f7c3b7557239620","name":"Raoyuan Zhao","hidden":false},{"_id":"6a9a50c58f7c3b7557239621","name":"Mykola Haltiuk","hidden":false},{"_id":"6a9a50c58f7c3b7557239622","name":"Thura Aung","hidden":false},{"_id":"6a9a50c58f7c3b7557239623","name":"Naser Almousa","hidden":false},{"_id":"6a9a50c58f7c3b7557239624","name":"Amir Hossein Kargaran","hidden":false},{"_id":"6a9a50c58f7c3b7557239625","name":"Rachel Bawden","hidden":false},{"_id":"6a9a50c58f7c3b7557239626","name":"Qiaoyuan Zheng","hidden":false},{"_id":"6a9a50c58f7c3b7557239627","name":"Mateusz Lango","hidden":false},{"_id":"6a9a50c58f7c3b7557239628","name":"Beni Egressy","hidden":false},{"_id":"6a9a50c58f7c3b7557239629","name":"Fidel Rodríguez Velásquez","hidden":false},{"_id":"6a9a50c58f7c3b755723962a","name":"Natchapon Jongwiriyanurak","hidden":false},{"_id":"6a9a50c58f7c3b755723962b","name":"Minh Ngoc Do","hidden":false},{"_id":"6a9a50c58f7c3b755723962c","name":"Marco Gaido","hidden":false},{"_id":"6a9a50c58f7c3b755723962d","name":"Lena Libon","hidden":false},{"_id":"6a9a50c58f7c3b755723962e","name":"Dzmitry Kuzmin","hidden":false},{"_id":"6a9a50c58f7c3b755723962f","name":"Badal Nyalang","hidden":false},{"_id":"6a9a50c58f7c3b7557239630","name":"Antoine Taroni","hidden":false},{"_id":"6a9a50c58f7c3b7557239631","name":"Andrei Niculae","hidden":false},{"_id":"6a9a50c58f7c3b7557239632","name":"Abdulaziz Nura Kani","hidden":false},{"_id":"6a9a50c58f7c3b7557239633","name":"Rushikesh Zawar","hidden":false},{"_id":"6a9a50c58f7c3b7557239634","name":"Marek Šuppa","hidden":false},{"_id":"6a9a50c58f7c3b7557239635","name":"Beatrice Savoldi","hidden":false},{"_id":"6a9a50c58f7c3b7557239636","name":"Andreas Simons","hidden":false},{"_id":"6a9a50c58f7c3b7557239637","name":"Rayyan Merchant","hidden":false},{"_id":"6a9a50c58f7c3b7557239638","name":"Ilai Yaron Levy","hidden":false},{"_id":"6a9a50c58f7c3b7557239639","name":"Francesco Pinto","hidden":false},{"_id":"6a9a50c58f7c3b755723963a","name":"Ziyi Yang","hidden":false},{"_id":"6a9a50c58f7c3b755723963b","name":"Yolanda Xavier","hidden":false},{"_id":"6a9a50c58f7c3b755723963c","name":"Samuel Frontull","hidden":false},{"_id":"6a9a50c58f7c3b755723963d","name":"Muhammad Ravi Shulthan Habibi","hidden":false},{"_id":"6a9a50c58f7c3b755723963e","name":"Kenneth Enevoldsen","hidden":false},{"_id":"6a9a50c58f7c3b755723963f","name":"Harris Abdul Majid","hidden":false},{"_id":"6a9a50c58f7c3b7557239640","name":"Francesca Padovani","hidden":false},{"_id":"6a9a50c58f7c3b7557239641","name":"Tim Graf","hidden":false},{"_id":"6a9a50c58f7c3b7557239642","name":"Tatiana Bielakova","hidden":false},{"_id":"6a9a50c58f7c3b7557239643","name":"Sharifa Djurabaeva","hidden":false},{"_id":"6a9a50c58f7c3b7557239644","name":"Shaoxiong Ji","hidden":false},{"_id":"6a9a50c58f7c3b7557239645","name":"Raia Abu Ahmad","hidden":false},{"_id":"6a9a50c58f7c3b7557239646","name":"Pavel Stepachev","hidden":false},{"_id":"6a9a50c58f7c3b7557239647","name":"Jirui Qi","hidden":false},{"_id":"6a9a50c58f7c3b7557239648","name":"Ayush Sunil Munot","hidden":false},{"_id":"6a9a50c58f7c3b7557239649","name":"Alireza Pakniat","hidden":false},{"_id":"6a9a50c58f7c3b755723964a","name":"Ayla Rigouts Terryn","hidden":false},{"_id":"6a9a50c58f7c3b755723964b","name":"Yuxing Lu","hidden":false},{"_id":"6a9a50c58f7c3b755723964c","name":"Yurii Paniv","hidden":false},{"_id":"6a9a50c58f7c3b755723964d","name":"Xiyan Fu","hidden":false},{"_id":"6a9a50c58f7c3b755723964e","name":"Tosin Adewumi","hidden":false},{"_id":"6a9a50c58f7c3b755723964f","name":"Sunisth Kumar","hidden":false},{"_id":"6a9a50c58f7c3b7557239650","name":"Stéphane J. P. S. Thunus","hidden":false},{"_id":"6a9a50c58f7c3b7557239651","name":"Shree Harsha Bokkahalli Satish","hidden":false},{"_id":"6a9a50c58f7c3b7557239652","name":"Shayan Bali","hidden":false},{"_id":"6a9a50c58f7c3b7557239653","name":"Prakhar Gupta","hidden":false},{"_id":"6a9a50c58f7c3b7557239654","name":"Papa Abdou Karim Karou Diallo","hidden":false},{"_id":"6a9a50c58f7c3b7557239655","name":"Matija Akrap","hidden":false},{"_id":"6a9a50c58f7c3b7557239656","name":"Marko Culjak","hidden":false},{"_id":"6a9a50c58f7c3b7557239657","name":"Kristýna Onderková","hidden":false},{"_id":"6a9a50c58f7c3b7557239658","name":"Joseph Attieh","hidden":false},{"_id":"6a9a50c58f7c3b7557239659","name":"Esrael Teferi Tensay","hidden":false},{"_id":"6a9a50c58f7c3b755723965a","name":"Elisabeth Fittschen","hidden":false},{"_id":"6a9a50c58f7c3b755723965b","name":"Benoît Sagot","hidden":false},{"_id":"6a9a50c58f7c3b755723965c","name":"Jingwei Ni","hidden":false},{"_id":"6a9a50c58f7c3b755723965d","name":"Yu Fan","hidden":false}],"publishedAt":"2026-09-03T00:00:00.000Z","submittedOnDailyAt":"2026-09-04T00:00:00.000Z","title":"Last Translation Benchmark","submittedOnDailyBy":{"_id":"6304ece07424d937fa35fb98","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6304ece07424d937fa35fb98/Ct4fi1SpLUwhrGDYsAcVB.jpeg","isPro":false,"fullname":"Vilém Zouhar","user":"zouhar","type":"user","name":"zouhar"},"summary":"For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, vulnerable to reward-hacking, and provide unactionable assessments. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking objective progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. The Last Translation Benchmark is a live dataset that accepts ongoing contributions. The latest version is LTBv1, containing accepted contributions prior to September 1st 2026, with future releases planned as new data is continuously collected.","upvotes":25,"discussionId":"6a9a50c68f7c3b755723965e","projectPage":"https://last-translation-benchmark.vilda.net/","githubRepo":"https://github.com/zouharvi/last-translation-benchmark","githubRepoAddedBy":"user","ai_summary":"The Last Translation Benchmark introduces peer-reviewed, multimodal examples that break leading translation models alongside handcrafted verification rules for reliable, actionable evaluation.","ai_keywords":["machine translation","Last Translation Benchmark","verification rules","multimodal evaluation","reward-hacking","automatic translation metrics"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":50},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6304ece07424d937fa35fb98","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6304ece07424d937fa35fb98/Ct4fi1SpLUwhrGDYsAcVB.jpeg","isPro":false,"fullname":"Vilém Zouhar","user":"zouhar","type":"user"},{"_id":"650ed7adf141bc34f91a12ae","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/rZDwRiBcqeqlUQ7mThoyU.jpeg","isPro":true,"fullname":"Hanna Yukhymenko","user":"hannayukhymenko","type":"user"},{"_id":"65044901221035b9ee187178","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65044901221035b9ee187178/o2LTLLIoT3T17Y4ZZZy_x.jpeg","isPro":false,"fullname":"Tommaso Cerruti","user":"Cerru02","type":"user"},{"_id":"60d33fbbd7b174177faabd4f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60d33fbbd7b174177faabd4f/pfyv_xj2B2m2N4F4sT9zJ.jpeg","isPro":true,"fullname":"Mike Zhang","user":"jjzha","type":"user"},{"_id":"6a9a52ae2ad6e182820b4b25","avatarUrl":"/avatars/01c874903fc4af46b3b9bf57595bb288.svg","isPro":false,"fullname":"Avantica Vempati","user":"zetrozky","type":"user"},{"_id":"66fa381d94df3bc9dd23a991","avatarUrl":"/avatars/dbbf67f27593a510177dff06b00bf098.svg","isPro":false,"fullname":"TVR","user":"theresiavr","type":"user"},{"_id":"627608ec624e6576d693c05c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1677077880239-627608ec624e6576d693c05c.png","isPro":false,"fullname":"Azmine Toushik Wasi","user":"azminetoushikwasi","type":"user"},{"_id":"6a6aa1f73550efadfe67147d","avatarUrl":"/avatars/783cd172a79ae72d02835468572bff74.svg","isPro":false,"fullname":"Anthony Williams","user":"Echo-Kai","type":"user"},{"_id":"6a6c7fd7a7b7e4e7d009eb87","avatarUrl":"/avatars/ac1c66e18985f8fbe74b587c2a7e60bf.svg","isPro":false,"fullname":"Timothy Clark","user":"velvetbeacon","type":"user"},{"_id":"6a6da7e008e6705013bc9e63","avatarUrl":"/avatars/c3d8bc73df06931ba549417b8af48e6b.svg","isPro":false,"fullname":"Linda Gonzalez","user":"NimbusLens","type":"user"},{"_id":"6a6dcb13097052be15667902","avatarUrl":"/avatars/be85cdc648b1d02daaa28bafba165ba9.svg","isPro":false,"fullname":"Joseph Williams","user":"Joseph-Williams","type":"user"},{"_id":"6a82544f787690a843abddfe","avatarUrl":"/avatars/86ae71b46fb61fa95d49f162271aa6d5.svg","isPro":false,"fullname":"Xinyi Li","user":"Drift-XinyiLi","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"query":{}}">

Cite arxiv.org/abs/2609.04173 in a model README.md to link it from this page.

Cite arxiv.org/abs/2609.04173 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.