Introducing <strong>DME</strong>, a two-stage framework for industrial multimodal embeddings! 🚀</p>\n<p><strong>Stage 1</strong> scales heterogeneous contrastive pre-training across text, image, video, and visual documents.<br><strong>Stage 2</strong> goes beyond pairwise alignment with <strong>Evidence-Grounded Typed Latent Reasoning</strong> and <strong>Cross-Conditional Self-Decoding via NTP/MTP</strong>.</p>\n<p>DME achieves the <strong>current SOTA on MMEB-v2</strong>, and delivers <strong>+2.92% offline</strong> and <strong>+0.1% online LT</strong> gains in a real Douyin search scenario.</p>\n","updatedAt":"2026-08-10T02:56:07.444Z","author":{"_id":"66add675c7a575aa0e03d5f3","avatarUrl":"/avatars/b72b18130664c1de197c1f8df371aa70.svg","fullname":"Haonan Chen","name":"Haon-Chen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6528986096382141},"editors":["Haon-Chen"],"editorAvatarUrls":["/avatars/b72b18130664c1de197c1f8df371aa70.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.02148","authors":[{"_id":"6a7587ece1228e04b323834b","user":{"_id":"66add675c7a575aa0e03d5f3","avatarUrl":"/avatars/b72b18130664c1de197c1f8df371aa70.svg","isPro":false,"fullname":"Haonan Chen","user":"Haon-Chen","type":"user","name":"Haon-Chen"},"name":"Haonan Chen","status":"claimed_verified","statusLastChangedAt":"2026-08-07T08:45:04.575Z","hidden":false},{"_id":"6a7587ece1228e04b323834c","name":"Chu Li","hidden":false},{"_id":"6a7587ece1228e04b323834d","name":"Zhicheng Wang","hidden":false},{"_id":"6a7587ece1228e04b323834e","name":"Yuanwei Liu","hidden":false},{"_id":"6a7587ece1228e04b323834f","name":"Yuanjiang Wang","hidden":false},{"_id":"6a7587ece1228e04b3238350","name":"Shaohua Jiang","hidden":false},{"_id":"6a7587ece1228e04b3238351","name":"Zhicheng Dou","hidden":false}],"publishedAt":"2026-08-03T00:00:00.000Z","submittedOnDailyAt":"2026-08-10T00:00:00.000Z","title":"Douyin Multimodal Embedding Model Technical Report","submittedOnDailyBy":{"_id":"66add675c7a575aa0e03d5f3","avatarUrl":"/avatars/b72b18130664c1de197c1f8df371aa70.svg","isPro":false,"fullname":"Haonan Chen","user":"Haon-Chen","type":"user","name":"Haon-Chen"},"summary":"Multimodal representation learning is a cornerstone of modern AI. By encoding multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an embedding is grounded in retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent Reasoning organizes retrieval evidence through hidden-space latent reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.","upvotes":2,"discussionId":"6a7587ece1228e04b3238352","organization":{"_id":"653b817d32c97d0655575872","name":"ByteDance","fullname":"ByteDance","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/0clr54wj5Ly-RkYU9OXPp.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66add675c7a575aa0e03d5f3","avatarUrl":"/avatars/b72b18130664c1de197c1f8df371aa70.svg","isPro":false,"fullname":"Haonan Chen","user":"Haon-Chen","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"653b817d32c97d0655575872","name":"ByteDance","fullname":"ByteDance","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/0clr54wj5Ly-RkYU9OXPp.png"},"query":{}}">
Douyin Multimodal Embedding Model Technical Report
Abstract
Multimodal representation learning is a cornerstone of modern AI. By encoding multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an embedding is grounded in retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent Reasoning organizes retrieval evidence through hidden-space latent reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.
Community
Introducing DME, a two-stage framework for industrial multimodal embeddings! 🚀
Stage 1 scales heterogeneous contrastive pre-training across text, image, video, and visual documents.
Stage 2 goes beyond pairwise alignment with Evidence-Grounded Typed Latent Reasoning and Cross-Conditional Self-Decoding via NTP/MTP.
DME achieves the current SOTA on MMEB-v2, and delivers +2.92% offline and +0.1% online LT gains in a real Douyin search scenario.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.02148 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.02148 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.02148 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.