<strong>train dense retrievers using the LLM’s next-token prediction loss</strong> instead of relying only on contrastive labels. </p>\n<p><strong>DREAM</strong> treats relevance as whether a document helps a frozen LLM predict the target output, then backpropagates that signal to the retriever through attention.</p>\n","updatedAt":"2026-06-24T02:53:44.239Z","author":{"_id":"647d834618274bce03013cc2","avatarUrl":"/avatars/a95c7df96dc4fb6a96193f6dd5068227.svg","fullname":"yixuan","name":"yixuantt","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":12,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7388074398040771},"editors":["yixuantt"],"editorAvatarUrls":["/avatars/a95c7df96dc4fb6a96193f6dd5068227.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.24667","authors":[{"_id":"6a3b423b0a86ac3098d5d6ea","name":"Yixuan Tang","hidden":false},{"_id":"6a3b423b0a86ac3098d5d6eb","name":"Yi Yang","hidden":false}],"publishedAt":"2026-06-23T00:00:00.000Z","submittedOnDailyAt":"2026-06-24T00:00:00.000Z","title":"DREAM: Dense Retrieval Embeddings via Autoregressive Modeling","submittedOnDailyBy":{"_id":"647d834618274bce03013cc2","avatarUrl":"/avatars/a95c7df96dc4fb6a96193f6dd5068227.svg","isPro":true,"fullname":"yixuan","user":"yixuantt","type":"user","name":"yixuantt"},"summary":"Dense retrieval embedding models are a fundamental component of modern retrieval-based AI systems. Most dense retrievers are trained with contrastive objectives, which require labeled positive and negative document pairs that are often costly and difficult to obtain. In this work, we investigate whether the autoregressive next-token prediction objective of a large language model (LLM) can provide supervision for dense retrieval. The intuition is simple: if a document contains information relevant to a query, conditioning on that document should make the target output easier for the LLM to predict. A key challenge is that the next-token prediction loss is computed inside the LLM, while the retriever is a separate embedding model. To address this challenge, we propose DREAM (Dense Retrieval Embeddings via Autoregressive Modeling), which injects retriever-generated query-document similarity scores into selected attention heads of a frozen LLM. During training, these scores determine how much attention each candidate document receives while the LLM predicts the target output. The resulting prediction loss provides gradients for retriever training through the attention mechanism. We evaluate DREAM on retrieval benchmarks BEIR and RTEB using embedding backbones ranging from 0.5B to 3B parameters. DREAM consistently outperforms existing baselines across different model scales. These results demonstrate that DREAM provides a promising approach for training dense retrievers through autoregressive modeling.","upvotes":1,"discussionId":"6a3b423b0a86ac3098d5d6ec","githubRepo":"https://github.com/yixuantt/DREAM","githubRepoAddedBy":"user","ai_summary":"DREAM trains dense retrieval embeddings using autoregressive language model attention mechanisms to supervise document-query similarity without requiring labeled examples.","ai_keywords":["dense retrieval embedding models","contrastive objectives","large language models","next-token prediction","autoregressive modeling","attention heads","frozen LLM","retrieval benchmarks","BEIR","RTEB","embedding backbones","query-document similarity","attention mechanism"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"647d834618274bce03013cc2","avatarUrl":"/avatars/a95c7df96dc4fb6a96193f6dd5068227.svg","isPro":true,"fullname":"yixuan","user":"yixuantt","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.24667.md","query":{}}">
DREAM: Dense Retrieval Embeddings via Autoregressive Modeling
Published on Jun 23
· Submitted by yixuan on Jun 24 Abstract
DREAM trains dense retrieval embeddings using autoregressive language model attention mechanisms to supervise document-query similarity without requiring labeled examples.
Dense retrieval embedding models are a fundamental component of modern retrieval-based AI systems. Most dense retrievers are trained with contrastive objectives, which require labeled positive and negative document pairs that are often costly and difficult to obtain. In this work, we investigate whether the autoregressive next-token prediction objective of a large language model (LLM) can provide supervision for dense retrieval. The intuition is simple: if a document contains information relevant to a query, conditioning on that document should make the target output easier for the LLM to predict. A key challenge is that the next-token prediction loss is computed inside the LLM, while the retriever is a separate embedding model. To address this challenge, we propose DREAM (Dense Retrieval Embeddings via Autoregressive Modeling), which injects retriever-generated query-document similarity scores into selected attention heads of a frozen LLM. During training, these scores determine how much attention each candidate document receives while the LLM predicts the target output. The resulting prediction loss provides gradients for retriever training through the attention mechanism. We evaluate DREAM on retrieval benchmarks BEIR and RTEB using embedding backbones ranging from 0.5B to 3B parameters. DREAM consistently outperforms existing baselines across different model scales. These results demonstrate that DREAM provides a promising approach for training dense retrievers through autoregressive modeling.
Community
train dense retrievers using the LLM’s next-token prediction loss instead of relying only on contrastive labels.
DREAM treats relevance as whether a document helps a frozen LLM predict the target output, then backpropagates that signal to the retriever through attention.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.24667 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.24667 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.24667 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.