submitted to EMNLP 2026</p>\n","updatedAt":"2026-06-04T03:45:26.245Z","author":{"_id":"671914b500bffab72acac309","avatarUrl":"/avatars/67a283ff4baa86eb43840b48479c380f.svg","fullname":"赵宇翔","name":"zhaoyx39","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9245067238807678},"editors":["zhaoyx39"],"editorAvatarUrls":["/avatars/67a283ff4baa86eb43840b48479c380f.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.30792","authors":[{"_id":"6a202b7a15100c5272a842fc","name":"Yanjie An","hidden":false},{"_id":"6a202b7a15100c5272a842fd","name":"Yuxiang Zhao","hidden":false},{"_id":"6a202b7a15100c5272a842fe","name":"Yichi Zhang","hidden":false},{"_id":"6a202b7a15100c5272a842ff","name":"Qixi Zheng","hidden":false},{"_id":"6a202b7a15100c5272a84300","name":"Yujie Tu","hidden":false},{"_id":"6a202b7a15100c5272a84301","name":"Keqi Deng","hidden":false},{"_id":"6a202b7a15100c5272a84302","name":"Kai Yu","hidden":false},{"_id":"6a202b7a15100c5272a84303","name":"Xie Chen","hidden":false}],"publishedAt":"2026-05-29T00:00:00.000Z","submittedOnDailyAt":"2026-06-04T00:00:00.000Z","title":"OpenSTBench: Beyond Semantic Evaluation for Speech Translation","submittedOnDailyBy":{"_id":"671914b500bffab72acac309","avatarUrl":"/avatars/67a283ff4baa86eb43840b48479c380f.svg","isPro":false,"fullname":"赵宇翔","user":"zhaoyx39","type":"user","name":"zhaoyx39"},"summary":"Speech translation systems increasingly span speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline translation, and streaming generation, producing outputs that differ in modality, speech realization, and timing behavior. Existing evaluation practices assess important aspects such as translation quality, speech quality, and temporal quality, but these aspects are often evaluated under separate protocols, making it difficult to compare heterogeneous systems comprehensively. To address this gap, we present OpenSTBench, a unified multidimensional evaluation framework that organizes heterogeneous speech translation outputs into a shared evaluation format. OpenSTBench supports both S2TT and S2ST systems in offline and streaming settings, and jointly evaluates translation quality, speech quality, speaker preservation, emotion and paralinguistic fidelity, temporal consistency, and latency. Through experiments on representative speech translation systems, we show that systems with strong translation quality can still differ substantially in speech quality, as well as in temporal quality. OpenSTBench provides a reproducible protocol for analyzing these cross-dimensional differences and supporting application-oriented comparison of speech translation systems. The code and datasets are available at https://github.com/sjtuayj/OpenSTBench.","upvotes":3,"discussionId":"6a202b7a15100c5272a84304","githubRepo":"https://github.com/sjtuayj/OpenSTBench","githubRepoAddedBy":"user","ai_summary":"OpenSTBench presents a unified evaluation framework for speech translation systems that assesses multiple dimensions including translation quality, speech quality, and temporal consistency across different modalities and settings.","ai_keywords":["speech-to-text translation","speech-to-speech translation","offline translation","streaming generation","multidimensional evaluation","temporal quality","speech quality","translation quality","speaker preservation","emotion fidelity","paralinguistic fidelity","temporal consistency","latency"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":9},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"671914b500bffab72acac309","avatarUrl":"/avatars/67a283ff4baa86eb43840b48479c380f.svg","isPro":false,"fullname":"赵宇翔","user":"zhaoyx39","type":"user"},{"_id":"66935bdc5489e4f73c76bc7b","avatarUrl":"/avatars/129d1e86bbaf764b507501f4feb177db.svg","isPro":false,"fullname":"Abidoye Aanuoluwapo","user":"Aanuoluwapo65","type":"user"},{"_id":"6a146e21ee253327b6bfe6db","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/apQXy1XQgVSyma27JjBJZ.jpeg","isPro":false,"fullname":"White Lily","user":"lilywhite52","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0}">

Cite arxiv.org/abs/2605.30792 in a model README.md to link it from this page.

Cite arxiv.org/abs/2605.30792 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.