Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale—and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame’s representation plus the encoded motion equals the next frame’s representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.</p>\n","updatedAt":"2026-06-16T16:19:26.537Z","author":{"_id":"63fbbff41b4b1bd4e7042fd1","avatarUrl":"/avatars/d186d0859046b35723fe1156d8f68917.svg","fullname":"Alexi Gladstone","name":"alexiglad","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":7,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9139363765716553},"editors":["alexiglad"],"editorAvatarUrls":["/avatars/d186d0859046b35723fe1156d8f68917.svg"],"reactions":[],"isReport":false}},{"id":"6a318102eced63a31e8acf54","author":{"_id":"6704728241373954c11bb49a","avatarUrl":"/avatars/be88206031eea78acdff9a0f2864bf0a.svg","fullname":"Ninad Daithankar","name":"ninaddaithankar","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-06-16T16:59:46.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"\n\n\n\n\n\n","html":"<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/6704728241373954c11bb49a/brn65-EnZXH2MQQJVf0-I.gif\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/6704728241373954c11bb49a/brn65-EnZXH2MQQJVf0-I.gif\" alt=\"tdv-intuition\"></a></p>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/6704728241373954c11bb49a/-vZNeF4TcaspUHBa9xUYg.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/6704728241373954c11bb49a/-vZNeF4TcaspUHBa9xUYg.png\" alt=\"Screenshot 2026-06-16 at 9.29.17 AM\"></a></p>\n","updatedAt":"2026-06-16T16:59:46.141Z","author":{"_id":"6704728241373954c11bb49a","avatarUrl":"/avatars/be88206031eea78acdff9a0f2864bf0a.svg","fullname":"Ninad Daithankar","name":"ninaddaithankar","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.3614974617958069},"editors":["ninaddaithankar"],"editorAvatarUrls":["/avatars/be88206031eea78acdff9a0f2864bf0a.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.15956","authors":[{"_id":"6a316be1bc818ff14e453b61","user":{"_id":"6704728241373954c11bb49a","avatarUrl":"/avatars/be88206031eea78acdff9a0f2864bf0a.svg","isPro":false,"fullname":"Ninad Daithankar","user":"ninaddaithankar","type":"user","name":"ninaddaithankar"},"name":"Ninad Daithankar","status":"claimed_verified","statusLastChangedAt":"2026-06-16T16:14:18.400Z","hidden":false},{"_id":"6a316be1bc818ff14e453b62","name":"Alexi Gladstone","hidden":false},{"_id":"6a316be1bc818ff14e453b63","name":"Yann LeCun","hidden":false},{"_id":"6a316be1bc818ff14e453b64","name":"Heng Ji","hidden":false}],"publishedAt":"2026-06-14T00:00:00.000Z","submittedOnDailyAt":"2026-06-16T00:00:00.000Z","title":"You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences","submittedOnDailyBy":{"_id":"63fbbff41b4b1bd4e7042fd1","avatarUrl":"/avatars/d186d0859046b35723fe1156d8f68917.svg","isPro":false,"fullname":"Alexi Gladstone","user":"alexiglad","type":"user","name":"alexiglad"},"summary":"Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale -- and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame's representation plus the encoded motion equals the next frame's representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.","upvotes":2,"discussionId":"6a316be1bc818ff14e453b65","projectPage":"https://temporal-difference-vision.github.io/","githubRepo":"https://github.com/ninaddaithankar/tdv","githubRepoAddedBy":"user","ai_summary":"Temporal Difference in Vision (TDV) presents a novel self-supervised learning approach for video data that eliminates traditional inductive biases by leveraging causal relationships between past and future frames.","ai_keywords":["self-supervised learning","visual representation learning","inductive biases","temporal difference","image encoder","motion encoder","causal assumption","dense spatial tasks"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":3,"organization":{"_id":"60212a089f64108326fac7c2","name":"illinois","fullname":"University of Illinois at Urbana-Champaign","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1612786274096-6021121cfb1b47827d667074.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63fbbff41b4b1bd4e7042fd1","avatarUrl":"/avatars/d186d0859046b35723fe1156d8f68917.svg","isPro":false,"fullname":"Alexi Gladstone","user":"alexiglad","type":"user"},{"_id":"6704728241373954c11bb49a","avatarUrl":"/avatars/be88206031eea78acdff9a0f2864bf0a.svg","isPro":false,"fullname":"Ninad Daithankar","user":"ninaddaithankar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"60212a089f64108326fac7c2","name":"illinois","fullname":"University of Illinois at Urbana-Champaign","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1612786274096-6021121cfb1b47827d667074.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.15956.md","query":{}}">
You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences
Abstract
Temporal Difference in Vision (TDV) presents a novel self-supervised learning approach for video data that eliminates traditional inductive biases by leveraging causal relationships between past and future frames.
Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale -- and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame's representation plus the encoded motion equals the next frame's representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.
Community
Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale—and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame’s representation plus the encoded motion equals the next frame’s representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.15956 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.15956 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.15956 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.