AI language models are typically trained on massive, completely shuffled snapshots of the internet. Because their parameters remain fixed after training, their knowledge becomes frozen in time, making them unreliable at recalling recent, evolving events. However, we observe that their factual knowledge often lags several years behind even the most recent data they were trained on. Our first and main contribution is the design of a benchmark of time-sensitive questions to thoroughly evaluate this gap. Then, to understand and resolve this temporal confusion, we compared standard shuffled training against feeding data to models in strict chronological order. Our results show that chronological training allows models to successfully grasp the most up-to-date facts. In contrast, shuffled models—like many current open-source releases—perform best on older knowledge. While this sequential method causes models to forget some older history in favor of recent information, it proves that the chronological order of training data is a critical key to building AI systems that stay current with our ever-changing world.</p>\n","updatedAt":"2026-05-27T06:51:59.160Z","author":{"_id":"66ea9f89a43597a36208be6c","avatarUrl":"/avatars/7024721892d8171923a8d4dced143d09.svg","fullname":"Hippolyte Pilchen","name":"HippolyteP","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9568037986755371},"editors":["HippolyteP"],"editorAvatarUrls":["/avatars/7024721892d8171923a8d4dced143d09.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.22769","authors":[{"_id":"6a1013dea53a61ce2e422ee6","user":{"_id":"66ea9f89a43597a36208be6c","avatarUrl":"/avatars/7024721892d8171923a8d4dced143d09.svg","isPro":false,"fullname":"Hippolyte Pilchen","user":"HippolyteP","type":"user","name":"HippolyteP"},"name":"Pilchen Hippolyte","status":"claimed_verified","statusLastChangedAt":"2026-05-22T15:59:57.111Z","hidden":false},{"_id":"6a1013dea53a61ce2e422ee7","user":{"_id":"5ec59018334ef26386e8e3f1","avatarUrl":"/avatars/ecf9e12995a46707fbd43d99e1b94611.svg","isPro":false,"fullname":"Romain Fabre","user":"rfbr","type":"user","name":"rfbr"},"name":"Fabre Romain","status":"claimed_verified","statusLastChangedAt":"2026-05-22T15:59:54.198Z","hidden":false},{"_id":"6a1013dea53a61ce2e422ee8","name":"Signe Talla Franck","hidden":false},{"_id":"6a1013dea53a61ce2e422ee9","name":"Perez Patrick","hidden":false},{"_id":"6a1013dea53a61ce2e422eea","name":"Grave Edouard","hidden":false}],"publishedAt":"2026-05-21T00:00:00.000Z","submittedOnDailyAt":"2026-05-27T00:00:00.000Z","title":"Understanding Data Temporality Impact on Large Language Models Pre-training","submittedOnDailyBy":{"_id":"66ea9f89a43597a36208be6c","avatarUrl":"/avatars/7024721892d8171923a8d4dced143d09.svg","isPro":false,"fullname":"Hippolyte Pilchen","user":"HippolyteP","type":"user","name":"HippolyteP"},"summary":"Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at train time and whose temporal grounding remains poorly understood. In this work, we study the impact of pre-training dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pre-training yields improved factual freshness, while shuffled pre-training peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our code at https://github.com/kyutai-labs/kairos , checkpoints, and datasets at https://huggingface.co/collections/kyutai/kairos provide a foundation for future research on continual learning for LLMs.","upvotes":2,"discussionId":"6a1013dfa53a61ce2e422eeb","projectPage":"https://kyutai.org/blog/2026-05-26-kairos","githubRepo":"https://github.com/kyutai-labs/kairos","githubRepoAddedBy":"user","ai_summary":"Pre-training large language models on temporally ordered data improves their factual freshness and temporal precision compared to standard shuffled pre-training while maintaining general language understanding capabilities.","ai_keywords":["large language models","pre-training dynamics","temporally grounded questions","temporal ordering","Common Crawl snapshots","factual freshness","temporal precision","continual learning"],"githubStars":1,"organization":{"_id":"6683d6350b54a28aff6645fe","name":"kyutai","fullname":"Kyutai","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6355a3c1805be5a8f30fea49/8xGdIOlfkopZfhbMitw_k.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66ea9f89a43597a36208be6c","avatarUrl":"/avatars/7024721892d8171923a8d4dced143d09.svg","isPro":false,"fullname":"Hippolyte Pilchen","user":"HippolyteP","type":"user"},{"_id":"5ec59018334ef26386e8e3f1","avatarUrl":"/avatars/ecf9e12995a46707fbd43d99e1b94611.svg","isPro":false,"fullname":"Romain Fabre","user":"rfbr","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6683d6350b54a28aff6645fe","name":"kyutai","fullname":"Kyutai","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6355a3c1805be5a8f30fea49/8xGdIOlfkopZfhbMitw_k.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.22769.md"}">
Understanding Data Temporality Impact on Large Language Models Pre-training
Abstract
Pre-training large language models on temporally ordered data improves their factual freshness and temporal precision compared to standard shuffled pre-training while maintaining general language understanding capabilities.
AI-generated summary
Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at train time and whose temporal grounding remains poorly understood. In this work, we study the impact of pre-training dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pre-training. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pre-training yields improved factual freshness, while shuffled pre-training peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our code at https://github.com/kyutai-labs/kairos , checkpoints, and datasets at https://huggingface.co/collections/kyutai/kairos provide a foundation for future research on continual learning for LLMs.
Community
AI language models are typically trained on massive, completely shuffled snapshots of the internet. Because their parameters remain fixed after training, their knowledge becomes frozen in time, making them unreliable at recalling recent, evolving events. However, we observe that their factual knowledge often lags several years behind even the most recent data they were trained on. Our first and main contribution is the design of a benchmark of time-sensitive questions to thoroughly evaluate this gap. Then, to understand and resolve this temporal confusion, we compared standard shuffled training against feeding data to models in strict chronological order. Our results show that chronological training allows models to successfully grasp the most up-to-date facts. In contrast, shuffled models—like many current open-source releases—perform best on older knowledge. While this sequential method causes models to forget some older history in favor of recent information, it proves that the chronological order of training data is a critical key to building AI systems that stay current with our ever-changing world.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.22769 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.