Hugging Face Daily Papers · June 8, 2026 · 4 min read

MMAE: A Massive Multitask Audio Editing Benchmark

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

MMAE--A Massive Multitask Audio Editing Benchmark, is the first comprehensive evaluation benchmark for speech and audio \"Banana🍌\"</p>\n","updatedAt":"2026-06-08T05:09:07.396Z","author":{"_id":"630388b0d14428368d1616c5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630388b0d14428368d1616c5/Z8O82fDVlB5qkM8Jmq65_.jpeg","fullname":"Ziyang Ma","name":"BoJack","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7966682314872742},"editors":["BoJack"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/630388b0d14428368d1616c5/Z8O82fDVlB5qkM8Jmq65_.jpeg"],"reactions":[{"reaction":"🚀","users":["Honggao"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.07229","authors":[{"_id":"6a264ccee4c258a029492124","name":"Ziyang Ma","hidden":false},{"_id":"6a264ccee4c258a029492125","name":"Ruiqi Yan","hidden":false},{"_id":"6a264ccee4c258a029492126","name":"Ruiyang Xu","hidden":false},{"_id":"6a264ccee4c258a029492127","name":"Jie Fang","hidden":false},{"_id":"6a264ccee4c258a029492128","name":"Zhikang Niu","hidden":false},{"_id":"6a264ccee4c258a029492129","name":"Yi-Wen Chao","hidden":false},{"_id":"6a264ccee4c258a02949212a","name":"Wenming Tu","hidden":false},{"_id":"6a264ccee4c258a02949212b","name":"Tianrui Wang","hidden":false},{"_id":"6a264ccee4c258a02949212c","name":"Auden","hidden":false},{"_id":"6a264ccee4c258a02949212d","name":"Qi Chen","hidden":false},{"_id":"6a264ccee4c258a02949212e","name":"Wenxi Chen","hidden":false},{"_id":"6a264ccee4c258a02949212f","name":"Jiaying Chi","hidden":false},{"_id":"6a264ccee4c258a029492130","name":"Yanru Huo","hidden":false},{"_id":"6a264ccee4c258a029492131","name":"Zixuan Jiang","hidden":false},{"_id":"6a264ccee4c258a029492132","name":"Xiquan Li","hidden":false},{"_id":"6a264ccee4c258a029492133","name":"Yalin Li","hidden":false},{"_id":"6a264ccee4c258a029492134","name":"Junxi Liu","hidden":false},{"_id":"6a264ccee4c258a029492135","name":"Minghao Liu","hidden":false},{"_id":"6a264ccee4c258a029492136","name":"Binghao Qiang","hidden":false},{"_id":"6a264ccee4c258a029492137","name":"Yijia Shan","hidden":false},{"_id":"6a264ccee4c258a029492138","name":"Zheshu Song","hidden":false},{"_id":"6a264ccee4c258a029492139","name":"Tian Tan","hidden":false},{"_id":"6a264ccee4c258a02949213a","name":"Zixiang Wang","hidden":false},{"_id":"6a264ccee4c258a02949213b","name":"Zeyu Xie","hidden":false},{"_id":"6a264ccee4c258a02949213c","name":"Zhifei Xie","hidden":false},{"_id":"6a264ccee4c258a02949213d","name":"Xiaoyu Xing","hidden":false},{"_id":"6a264ccee4c258a02949213e","name":"Qixiang Xu","hidden":false},{"_id":"6a264ccee4c258a02949213f","name":"Chen Yang","hidden":false},{"_id":"6a264ccee4c258a029492140","name":"Guanrou Yang","hidden":false},{"_id":"6a264ccee4c258a029492141","name":"Shan Yang","hidden":false},{"_id":"6a264ccee4c258a029492142","name":"Yifan Yang","hidden":false},{"_id":"6a264ccee4c258a029492143","name":"Steve Yves","hidden":false},{"_id":"6a264ccee4c258a029492144","name":"Haotian Zhang","hidden":false},{"_id":"6a264ccee4c258a029492145","name":"Haina Zhu","hidden":false},{"_id":"6a264ccee4c258a029492146","name":"Kai Yu","hidden":false},{"_id":"6a264ccee4c258a029492147","name":"Liefeng Bo","hidden":false},{"_id":"6a264ccee4c258a029492148","name":"Eng-Siong Chng","hidden":false},{"_id":"6a264ccee4c258a029492149","name":"Xie Chen","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/630388b0d14428368d1616c5/WG6Gpu-P_9dHByFX11W3p.mp4","https://cdn-uploads.huggingface.co/production/uploads/630388b0d14428368d1616c5/Rhd_ENrfHKFCRQjf-x8NB.png","https://cdn-uploads.huggingface.co/production/uploads/630388b0d14428368d1616c5/uP2wOmeeAAVpkz4gt_1x6.png","https://cdn-uploads.huggingface.co/production/uploads/630388b0d14428368d1616c5/bmX-K6SS2uwsZaZgDx2XF.png"],"publishedAt":"2026-06-05T00:00:00.000Z","submittedOnDailyAt":"2026-06-08T00:00:00.000Z","title":"MMAE: A Massive Multitask Audio Editing Benchmark","submittedOnDailyBy":{"_id":"630388b0d14428368d1616c5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630388b0d14428368d1616c5/Z8O82fDVlB5qkM8Jmq65_.jpeg","isPro":false,"fullname":"Ziyang Ma","user":"BoJack","type":"user","name":"BoJack"},"summary":"We introduce MMAE, a Massive Multitask Audio Editing benchmark, serving as the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing. Spurred by the shift toward intelligent creation, interactive editing has rapidly expanded from visual domains, pioneered by models like Nano-banana 2 for images and Gemini-Omni for video, into audio. However, the current evaluation infrastructure lags severely, remaining highly fragmented and restricted to specific subdomains or basic operations. Unlike existing benchmarks that are limited in scope, MMAE extends to a broad spectrum of real-world scenarios, encompassing 7 distinct audio modalities, including sound, speech, music, and their mixtures. Furthermore, we establish a comprehensive taxonomy spanning 6 levels of task complexity, from basic modifications to multi-hop reasoning and multi-round editing, 2 levels of granularity, and 8 distinct operation types. Meticulously curated through human-agent collaboration, MMAE comprises 2,000 high-fidelity samples paired with a pioneering rubric-based evaluation framework. By decomposing free-form tasks into 17,741 verifiable criteria, this robust rubric-based paradigm enables a precise, multi-dimensional assessment of both instruction following and context consistency. Our extensive evaluation of leading models reveals that current systems remain far from achieving reliable edits. Strikingly, the Exact Match Rate (EMR) consistently falls below 5% and plummets to an absolute 0% in complex, mixed-modality tasks, exposing critical bottlenecks in precise execution and structural robustness. We hope MMAE will serve as a catalyst for future advances in the intelligent creation community, providing a clear diagnostic roadmap and establishing a standardized, long-lasting evaluation paradigm for next-generation audio editing systems.","upvotes":33,"discussionId":"6a264ccee4c258a02949214a","githubRepo":"https://github.com/ddlBoJack/MMAE","githubRepoAddedBy":"user","ai_summary":"MMAE presents a comprehensive benchmark for instruction-based audio editing across multiple modalities and complexity levels, revealing significant gaps in current model capabilities.","ai_keywords":["Multitask Audio Editing","instruction-based audio editing","audio modalities","task complexity","operation types","rubric-based evaluation","Exact Match Rate","multi-hop reasoning","multi-round editing"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":27},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"630388b0d14428368d1616c5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630388b0d14428368d1616c5/Z8O82fDVlB5qkM8Jmq65_.jpeg","isPro":false,"fullname":"Ziyang Ma","user":"BoJack","type":"user"},{"_id":"6a18f79852019fb6c46ec119","avatarUrl":"/avatars/9f4107332f41e19c729f23e499031316.svg","isPro":false,"fullname":"Qixiang Xu","user":"Rickman5566","type":"user"},{"_id":"6440656b757aa3c2ad86a67a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6440656b757aa3c2ad86a67a/oVSwa0QfQYJjTBUPwXhO-.jpeg","isPro":false,"fullname":"Wenming Tu","user":"tutu0604","type":"user"},{"_id":"6447d332ab5c7251886d6fd1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6447d332ab5c7251886d6fd1/bm5nwIp5CA_HosO8wXFvI.jpeg","isPro":false,"fullname":"ZhikangNiu-SII","user":"zkniu","type":"user"},{"_id":"6786007928014d20f37fb228","avatarUrl":"/avatars/08fa31dc953362bb263127c00aae922d.svg","isPro":false,"fullname":"Yanru Huo","user":"iHateTheWorld555","type":"user"},{"_id":"692cfa3bb14ceb7582037c90","avatarUrl":"/avatars/59f78031fac1ae894042e45e8619b3a6.svg","isPro":false,"fullname":"az","user":"masaz14","type":"user"},{"_id":"63999368636597492d720400","avatarUrl":"/avatars/c109a62c73f8a964b97f3278a2ea0ac5.svg","isPro":false,"fullname":"Yifan Yang","user":"yfyeung","type":"user"},{"_id":"69e5d357f8405c2e79118e03","avatarUrl":"/avatars/8869f618a7442c42e6dc96686ef03ecf.svg","isPro":false,"fullname":"Emilia","user":"DrEmilia","type":"user"},{"_id":"656c3503e0ff1cebe95e5730","avatarUrl":"/avatars/e9e6ecde1ca79833bfe23df51ee0dc68.svg","isPro":false,"fullname":"Tsuki","user":"Tsukihjy","type":"user"},{"_id":"6864e44244dc36f3a7b896eb","avatarUrl":"/avatars/21f5dfd9858e3a9042c3a0cbc9fbcc69.svg","isPro":false,"fullname":"Li","user":"yayll","type":"user"},{"_id":"67fdfbb2428f53b24a3349ad","avatarUrl":"/avatars/0f4fa5d1373e17637b7cfef154ac8dbb.svg","isPro":false,"fullname":"jianan_chen","user":"Nathan1chen","type":"user"},{"_id":"657a9de87115d9f2649772b0","avatarUrl":"/avatars/8d0e67be4a9ef60b159a6bb594eb7f66.svg","isPro":false,"fullname":"TRWang","user":"ScorpioMiku","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":2}">

Papers

arxiv:2606.07229

MMAE: A Massive Multitask Audio Editing Benchmark

Published on Jun 5

· Submitted by

Ziyang Ma on Jun 8

#2 Paper of the day

Upvote

Authors:

Abstract

MMAE presents a comprehensive benchmark for instruction-based audio editing across multiple modalities and complexity levels, revealing significant gaps in current model capabilities.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

We introduce MMAE, a Massive Multitask Audio Editing benchmark, serving as the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing. Spurred by the shift toward intelligent creation, interactive editing has rapidly expanded from visual domains, pioneered by models like Nano-banana 2 for images and Gemini-Omni for video, into audio. However, the current evaluation infrastructure lags severely, remaining highly fragmented and restricted to specific subdomains or basic operations. Unlike existing benchmarks that are limited in scope, MMAE extends to a broad spectrum of real-world scenarios, encompassing 7 distinct audio modalities, including sound, speech, music, and their mixtures. Furthermore, we establish a comprehensive taxonomy spanning 6 levels of task complexity, from basic modifications to multi-hop reasoning and multi-round editing, 2 levels of granularity, and 8 distinct operation types. Meticulously curated through human-agent collaboration, MMAE comprises 2,000 high-fidelity samples paired with a pioneering rubric-based evaluation framework. By decomposing free-form tasks into 17,741 verifiable criteria, this robust rubric-based paradigm enables a precise, multi-dimensional assessment of both instruction following and context consistency. Our extensive evaluation of leading models reveals that current systems remain far from achieving reliable edits. Strikingly, the Exact Match Rate (EMR) consistently falls below 5% and plummets to an absolute 0% in complex, mixed-modality tasks, exposing critical bottlenecks in precise execution and structural robustness. We hope MMAE will serve as a catalyst for future advances in the intelligent creation community, providing a clear diagnostic roadmap and establishing a standardized, long-lasting evaluation paradigm for next-generation audio editing systems.