Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization.</p>\n","updatedAt":"2026-08-07T05:35:09.225Z","author":{"_id":"66a3ae59f33ff23e1c027ccd","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a3ae59f33ff23e1c027ccd/tZzpESNPnmhty62xhHszF.jpeg","fullname":"Vladan Stojnic","name":"stojnvla","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8797089457511902},"editors":["stojnvla"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/66a3ae59f33ff23e1c027ccd/tZzpESNPnmhty62xhHszF.jpeg"],"reactions":[],"isReport":false}},{"id":"6a75cb74e6a24fb4a23f8222","author":{"_id":"658412f93a84a40185adaf37","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/658412f93a84a40185adaf37/FKXH7e1jj09KO1v-B5sER.jpeg","fullname":"Aamer Mihaysi","name":"O96a","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-08-07T12:11:32.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Does this mean every model I've trained on scraped web images has been quietly memorizing camera fingerprints, not just semantics? If so, the fix isn't just better augmentation — it's asking whether the shortcut survives when you strip EXIF and re-encode. I'd love to see how much of the effect persists through JPEG recompression and resizing, because that's what actually happens to images before they hit most training pipelines. If it survives that, this is a much bigger deal than a dataset hygiene footnote.","html":"<p>Does this mean every model I've trained on scraped web images has been quietly memorizing camera fingerprints, not just semantics? If so, the fix isn't just better augmentation — it's asking whether the shortcut survives when you strip EXIF and re-encode. I'd love to see how much of the effect persists through JPEG recompression and resizing, because that's what actually happens to images before they hit most training pipelines. If it survives that, this is a much bigger deal than a dataset hygiene footnote.</p>\n","updatedAt":"2026-08-07T12:11:32.411Z","author":{"_id":"658412f93a84a40185adaf37","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/658412f93a84a40185adaf37/FKXH7e1jj09KO1v-B5sER.jpeg","fullname":"Aamer Mihaysi","name":"O96a","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9475628137588501},"editors":["O96a"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/658412f93a84a40185adaf37/FKXH7e1jj09KO1v-B5sER.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.05424","authors":[{"_id":"6a75566be1228e04b323820c","user":{"_id":"66a3ae59f33ff23e1c027ccd","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a3ae59f33ff23e1c027ccd/tZzpESNPnmhty62xhHszF.jpeg","isPro":true,"fullname":"Vladan Stojnic","user":"stojnvla","type":"user","name":"stojnvla"},"name":"Vladan Stojnić","status":"claimed_verified","statusLastChangedAt":"2026-08-07T08:45:04.555Z","hidden":false},{"_id":"6a75566be1228e04b323820d","name":"Ryan Ramos","hidden":false},{"_id":"6a75566be1228e04b323820e","name":"Giorgos Kordopatis-Zilos","hidden":false},{"_id":"6a75566be1228e04b323820f","name":"Noa Garcia","hidden":false},{"_id":"6a75566be1228e04b3238210","name":"Giorgos Tolias","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/66a3ae59f33ff23e1c027ccd/P3kmyfQMvgTHpayuNkczM.png"],"publishedAt":"2026-08-05T00:00:00.000Z","submittedOnDailyAt":"2026-08-07T00:00:00.000Z","title":"Invisible Shortcuts: Why Vision Encoders Know Your Camera","submittedOnDailyBy":{"_id":"66a3ae59f33ff23e1c027ccd","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a3ae59f33ff23e1c027ccd/tZzpESNPnmhty62xhHszF.jpeg","isPro":true,"fullname":"Vladan Stojnic","user":"stojnvla","type":"user","name":"stojnvla"},"summary":"Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces","upvotes":4,"discussionId":"6a75566be1228e04b3238211"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66a3ae59f33ff23e1c027ccd","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a3ae59f33ff23e1c027ccd/tZzpESNPnmhty62xhHszF.jpeg","isPro":true,"fullname":"Vladan Stojnic","user":"stojnvla","type":"user"},{"_id":"65c4eb7cd1dcbd30d86febec","avatarUrl":"/avatars/001c8f02e8ce794b2c21883628b2da72.svg","isPro":false,"fullname":"free-bit","user":"free-bit","type":"user"},{"_id":"66f2c5aa9731b434eed18401","avatarUrl":"/avatars/23a70932e7fb64f4a0bda726bbeb7a98.svg","isPro":false,"fullname":"Giorgos Tolias","user":"gtolias","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"query":{}}">
Invisible Shortcuts: Why Vision Encoders Know Your Camera
Abstract
Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces
Community
Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization.
Does this mean every model I've trained on scraped web images has been quietly memorizing camera fingerprints, not just semantics? If so, the fix isn't just better augmentation — it's asking whether the shortcut survives when you strip EXIF and re-encode. I'd love to see how much of the effect persists through JPEG recompression and resizing, because that's what actually happens to images before they hit most training pipelines. If it survives that, this is a much bigger deal than a dataset hygiene footnote.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.05424 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.05424 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.05424 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.