Website: <a href=\"https://kyutai.org/fid-lottery/\" rel=\"nofollow\">https://kyutai.org/fid-lottery/</a></p>\n","updatedAt":"2026-06-19T14:56:07.081Z","author":{"_id":"630652803aed65d34e98eee3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630652803aed65d34e98eee3/XG_PuVFA6ziGQZd3UUZSF.jpeg","fullname":"Nicolas Dufour","name":"nicolas-dufour","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":25,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7208777666091919},"editors":["nicolas-dufour"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/630652803aed65d34e98eee3/XG_PuVFA6ziGQZd3UUZSF.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.20536","authors":[{"_id":"6a355854db23715e9da12c0b","name":"Nicolas Dufour","hidden":false},{"_id":"6a355854db23715e9da12c0c","name":"Alexei A. Efros","hidden":false},{"_id":"6a355854db23715e9da12c0d","name":"Patrick Pérez","hidden":false}],"publishedAt":"2026-06-18T00:00:00.000Z","submittedOnDailyAt":"2026-06-19T00:00:00.000Z","title":"The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation","submittedOnDailyBy":{"_id":"630652803aed65d34e98eee3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630652803aed65d34e98eee3/XG_PuVFA6ziGQZd3UUZSF.jpeg","isPro":false,"fullname":"Nicolas Dufour","user":"nicolas-dufour","type":"user","name":"nicolas-dufour"},"summary":"The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256. We report surprising findings: (a) Retraining the model using the same recipe with a different seed moves FID 3.2x more (in Inception feature space) than redrawing samples from a fixed network. (b) That gap is driven by three factors: random initialisation, data ordering, and the per-step Gaussian noise of the flow-matching loss. (c) Increasing compute or model size barely tightens the spread, holding the FID coefficient of variation (CoV) inside a 1-2% band. (d) Per-cell classifier-free-guidance tuning halves the spread but reshuffles which seeds work best, and a lucky training seed reaches the same FID with up to 2x less compute than an unlucky one. Based on these findings, we recommend a new FID evaluation protocol: evaluate under per-cell optimal guidance, treat any FID gap below the empirically measured ~1.3% CoV as inconclusive, and report an error bar over several training seeds rather than a single FID number.","upvotes":2,"discussionId":"6a355855db23715e9da12c0e","projectPage":"https://kyutai.org/fid-lottery/","ai_summary":"Analysis of FID variance across different training and sampling seeds reveals significant reproducibility issues in image generation evaluation, with retraining causing larger fluctuations than resampling, and recommends updated evaluation protocols with error bars and optimal guidance tuning.","ai_keywords":["Frechet Inception Distance","FID","training seeds","generation seeds","classifier-free-guidance","flow-matching loss","inception feature space","coefficient of variation","reproducibility","image generation evaluation"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"6683d6350b54a28aff6645fe","name":"kyutai","fullname":"Kyutai","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6355a3c1805be5a8f30fea49/8xGdIOlfkopZfhbMitw_k.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"630652803aed65d34e98eee3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630652803aed65d34e98eee3/XG_PuVFA6ziGQZd3UUZSF.jpeg","isPro":false,"fullname":"Nicolas Dufour","user":"nicolas-dufour","type":"user"},{"_id":"66ea9f89a43597a36208be6c","avatarUrl":"/avatars/7024721892d8171923a8d4dced143d09.svg","isPro":false,"fullname":"Hippolyte Pilchen","user":"HippolyteP","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6683d6350b54a28aff6645fe","name":"kyutai","fullname":"Kyutai","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6355a3c1805be5a8f30fea49/8xGdIOlfkopZfhbMitw_k.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.20536.md","query":{}}">
The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
Abstract
Analysis of FID variance across different training and sampling seeds reveals significant reproducibility issues in image generation evaluation, with retraining causing larger fluctuations than resampling, and recommends updated evaluation protocols with error bars and optimal guidance tuning.
The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256. We report surprising findings: (a) Retraining the model using the same recipe with a different seed moves FID 3.2x more (in Inception feature space) than redrawing samples from a fixed network. (b) That gap is driven by three factors: random initialisation, data ordering, and the per-step Gaussian noise of the flow-matching loss. (c) Increasing compute or model size barely tightens the spread, holding the FID coefficient of variation (CoV) inside a 1-2% band. (d) Per-cell classifier-free-guidance tuning halves the spread but reshuffles which seeds work best, and a lucky training seed reaches the same FID with up to 2x less compute than an unlucky one. Based on these findings, we recommend a new FID evaluation protocol: evaluate under per-cell optimal guidance, treat any FID gap below the empirically measured ~1.3% CoV as inconclusive, and report an error bar over several training seeds rather than a single FID number.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.20536 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.20536 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.20536 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.