How do you actually know which genomic foundation model is best? Right now - you mostly can't.</p>\n<p>Every paper uses its own tasks, baselines, and evaluation protocols, so \"state-of-the-art\" claims rarely transfer. GENEB fixes this with one unified protocol: 40 genomic foundation models, 100 classification tasks, 13 functional categories, evaluated under full-data, 10-shot, and 1-shot regimes via linear probing on frozen embeddings.</p>\n<p>What we found is the interesting part:</p>\n<ul>\n<li>Aggregate leaderboards can be misleading - a model that's best on average is often far from best on the task you care about.</li>\n<li>Few-shot rankings ≠ full-data rankings. The \"best\" model flips depending on how much data you have.</li>\n<li>Bigger isn't better. Scale alone does not predict downstream performance.</li>\n</ul>\n<p>Full leaderboard and all 100 tasks are public, and you can submit your own model. If you work on DNA models, this is built for you 🧬</p>\n<p>ICML 2026.</p>\n","updatedAt":"2026-06-08T11:21:04.932Z","author":{"_id":"654172a2982a653159aa20b3","avatarUrl":"/avatars/dec7e055c5c7cfd114b9dddd9b17e7c1.svg","fullname":"Daria Ledneva","name":"darlednik","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.9131057262420654},"editors":["darlednik"],"editorAvatarUrls":["/avatars/dec7e055c5c7cfd114b9dddd9b17e7c1.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.04525","authors":[{"_id":"6a22af73047f837f98677934","user":{"_id":"654172a2982a653159aa20b3","avatarUrl":"/avatars/dec7e055c5c7cfd114b9dddd9b17e7c1.svg","isPro":false,"fullname":"Daria Ledneva","user":"darlednik","type":"user","name":"darlednik"},"name":"Daria Ledneva","status":"claimed_verified","statusLastChangedAt":"2026-06-08T09:47:23.044Z","hidden":false},{"_id":"6a22af73047f837f98677935","name":"Mikhail Nuridinov","hidden":false},{"_id":"6a22af73047f837f98677936","name":"Denis Kuznetsov","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/654172a2982a653159aa20b3/JOqIYumT3jrpq1zkhRaTw.png","https://cdn-uploads.huggingface.co/production/uploads/654172a2982a653159aa20b3/Q1fM8yIvDMECSdpSMFt9w.png","https://cdn-uploads.huggingface.co/production/uploads/654172a2982a653159aa20b3/Y-F-L4KLi49FhmhoENkxc.png","https://cdn-uploads.huggingface.co/production/uploads/654172a2982a653159aa20b3/ZsNuVjhgl0amWsMalg2mX.png","https://cdn-uploads.huggingface.co/production/uploads/654172a2982a653159aa20b3/oqByuqGDnXHmIYZIfbfvW.png"],"publishedAt":"2026-06-03T00:00:00.000Z","submittedOnDailyAt":"2026-06-08T00:00:00.000Z","title":"GENEB: Why Genomic Models Are Hard to Compare","submittedOnDailyBy":{"_id":"654172a2982a653159aa20b3","avatarUrl":"/avatars/dec7e055c5c7cfd114b9dddd9b17e7c1.svg","isPro":false,"fullname":"Daria Ledneva","user":"darlednik","type":"user","name":"darlednik"},"summary":"Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reporting. As a result, claims of superiority or generality across models are often not directly comparable. We introduce GENEB, a large-scale diagnostic benchmark that evaluates frozen representations from 40 genomic foundation models across 100 tasks spanning 13 functional categories under a unified probing-based protocol, including few-shot regimes. GENEB enables controlled comparison across model scale, architecture, tokenization, and pretraining data while explicitly exposing task-level trade-offs. Our analysis shows that aggregate leaderboards are unstable: model rankings vary sharply across task categories, scale provides only modest and inconsistent gains, and architectural and pretraining alignment frequently outweigh parameter count. These results highlight limitations of current evaluation practices and position GENEB as a reference framework for principled comparison and category-aware model selection in genomic machine learning.","upvotes":42,"discussionId":"6a22af73047f837f98677937","projectPage":"https://huggingface.co/spaces/darlednik/geneb-leaderboard","githubRepo":"https://github.com/darlednik/GENEB","githubRepoAddedBy":"user","ai_summary":"GENEB presents a comprehensive benchmark for evaluating genomic foundation models across diverse tasks and architectures under a unified protocol.","ai_keywords":["genomic foundation models","diagnostic benchmark","frozen representations","probing-based protocol","few-shot regimes","task categories","model scale","architecture","tokenization","pretraining data","aggregate leaderboards","model rankings","architectural alignment","pretraining alignment"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":10},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64d403f7d0cfd876d93392d7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/TKJXT6GZ5Wv14gcuBAIpQ.jpeg","isPro":false,"fullname":"Ilya Alexeev","user":"voorhs","type":"user"},{"_id":"6451539667703e98a3cba7c6","avatarUrl":"/avatars/399de2e7d7741226e63e20172b518163.svg","isPro":false,"fullname":"Mikhail Nuridinov","user":"aiMihail","type":"user"},{"_id":"6797b93c67fbbe18032f07aa","avatarUrl":"/avatars/f64c7f3f07fc20dbcaca76aee741cd79.svg","isPro":false,"fullname":"Gorbunov Dmitriy","user":"DPGorbunov","type":"user"},{"_id":"654172a2982a653159aa20b3","avatarUrl":"/avatars/dec7e055c5c7cfd114b9dddd9b17e7c1.svg","isPro":false,"fullname":"Daria Ledneva","user":"darlednik","type":"user"},{"_id":"6a26aa3e41d82ed47a45606d","avatarUrl":"/avatars/657871a0d283868003e91ec53a305f65.svg","isPro":false,"fullname":"-","user":"AlexLedi","type":"user"},{"_id":"676fe31a9949628b6893b325","avatarUrl":"/avatars/05088d7aad857bfbb115b936d4608c73.svg","isPro":false,"fullname":"Ilya Orekhovsky","user":"orekhovsky","type":"user"},{"_id":"68ec97a31dfe9ec7bd34b879","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/j87DmpOIi3TcS_aZ-NjP_.jpeg","isPro":false,"fullname":"Igor Pivnev","user":"qw1zzard","type":"user"},{"_id":"672cadf5b8df8337b7df67d2","avatarUrl":"/avatars/572d0c3efd6b69d4e8b61e992f3325d1.svg","isPro":false,"fullname":"minure","user":"leksminure","type":"user"},{"_id":"636024e588e41d249ecd1d05","avatarUrl":"/avatars/e622567780d8b5b5c20277bea035bc48.svg","isPro":false,"fullname":"Maxim Gordienko","user":"AeroDancer","type":"user"},{"_id":"649d7f58fb6517e323f22391","avatarUrl":"/avatars/68b7d534a7e39ea34578a5d4c3b78571.svg","isPro":false,"fullname":"Vladislav Kulikov","user":"slewie","type":"user"},{"_id":"65ecebc9cae0146ec5158006","avatarUrl":"/avatars/25ef6169252b285895e450cfe8781622.svg","isPro":false,"fullname":"Danis Zaripov","user":"dzaripov","type":"user"},{"_id":"681cb5b1edf34a77aa796101","avatarUrl":"/avatars/f32a3ca6d7b501afc3d0f19fffbd68d7.svg","isPro":false,"fullname":"Fanis Nigamadyanov","user":"FanisNgv","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":3,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.04525.md"}">
GENEB: Why Genomic Models Are Hard to Compare
Abstract
GENEB presents a comprehensive benchmark for evaluating genomic foundation models across diverse tasks and architectures under a unified protocol.
Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reporting. As a result, claims of superiority or generality across models are often not directly comparable. We introduce GENEB, a large-scale diagnostic benchmark that evaluates frozen representations from 40 genomic foundation models across 100 tasks spanning 13 functional categories under a unified probing-based protocol, including few-shot regimes. GENEB enables controlled comparison across model scale, architecture, tokenization, and pretraining data while explicitly exposing task-level trade-offs. Our analysis shows that aggregate leaderboards are unstable: model rankings vary sharply across task categories, scale provides only modest and inconsistent gains, and architectural and pretraining alignment frequently outweigh parameter count. These results highlight limitations of current evaluation practices and position GENEB as a reference framework for principled comparison and category-aware model selection in genomic machine learning.
Community
How do you actually know which genomic foundation model is best? Right now - you mostly can't.
Every paper uses its own tasks, baselines, and evaluation protocols, so "state-of-the-art" claims rarely transfer. GENEB fixes this with one unified protocol: 40 genomic foundation models, 100 classification tasks, 13 functional categories, evaluated under full-data, 10-shot, and 1-shot regimes via linear probing on frozen embeddings.
What we found is the interesting part:
- Aggregate leaderboards can be misleading - a model that's best on average is often far from best on the task you care about.
- Few-shot rankings ≠ full-data rankings. The "best" model flips depending on how much data you have.
- Bigger isn't better. Scale alone does not predict downstream performance.
Full leaderboard and all 100 tasks are public, and you can submit your own model. If you work on DNA models, this is built for you 🧬
ICML 2026.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.04525 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.04525 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.04525 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.