Bernini is a unified framework for video generation and editing that uses an MLLM-based planner to predict latent semantic representations for a DiT-based renderer.</p>\n","updatedAt":"2026-05-22T02:04:44.940Z","author":{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","fullname":"taesiri","name":"taesiri","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":303,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.85256427526474},"editors":["taesiri"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.22344","authors":[{"_id":"6a0fb9a6a53a61ce2e422c38","name":"Bernini Team","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c39","name":"Chenchen Liu","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c3a","name":"Junyi Chen","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c3b","name":"Lei Li","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c3c","name":"Lu Chi","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c3d","name":"Mingzhen Sun","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c3e","name":"Zhuoying Li","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c3f","name":"Yi Fu","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c40","name":"Ruoyu Guo","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c41","name":"Yiheng Wu","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c42","name":"Ge Bai","hidden":false},{"_id":"6a0fb9a6a53a61ce2e422c43","name":"Zehuan Yuan","hidden":false}],"publishedAt":"2026-05-21T00:00:00.000Z","submittedOnDailyAt":"2026-05-22T00:00:00.000Z","title":"Bernini: Latent Semantic Planning for Video Diffusion","submittedOnDailyBy":{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user","name":"taesiri"},"summary":"Multimodal large language models (MLLMs) and diffusion models have each reached remarkable maturity: MLLMs excel at reasoning over heterogeneous multimodal inputs with strong semantic grounding, while diffusion models synthesize images and videos with photorealistic fidelity. We argue that these two families can be unified through a simple division of labor: MLLMs perform semantic planning, while diffusion models render pixels from high-level semantic guidance and low-level visual features. Building on this idea, we propose Bernini, a unified framework for video generation and editing. An MLLM-based planner predicts the target semantic representation directly in the ViT embedding space, and a DiT-based renderer synthesizes pixels conditioned on this plan, augmented by text features and, for editing, source VAE features for detail preservation. Because semantics serve as the interface, the planner and renderer can be trained separately and only lightly co-trained, preserving the pretrained strengths of both components while keeping training efficient. To better handle multiple visual inputs, we introduce Segment-Aware 3D Rotary Positional Embedding (SA-3D RoPE), and further incorporate chain-of-thought reasoning in the planner to better transfer understanding into generation. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks, with the MLLM's pretrained understanding translating into strong generalization on challenging editing tasks.","upvotes":3,"discussionId":"6a0fb9a6a53a61ce2e422c44","projectPage":"https://bernini-ai.github.io/","ai_summary":"A unified video generation and editing framework combines multimodal large language models for semantic planning with diffusion models for pixel rendering, achieving state-of-the-art performance through semantic interface separation and enhanced positional embeddings.","ai_keywords":["multimodal large language models","diffusion models","semantic planning","pixel rendering","ViT embedding space","DiT-based renderer","text features","VAE features","Segment-Aware 3D Rotary Positional Embedding","chain-of-thought reasoning","video generation","video editing"],"organization":{"_id":"653b817d32c97d0655575872","name":"ByteDance","fullname":"ByteDance","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/0clr54wj5Ly-RkYU9OXPp.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6507fbecffc738079ca592bf","avatarUrl":"/avatars/1cb0f39ac6dc2dba2292846a8d7746da.svg","isPro":false,"fullname":"Ming Chen","user":"ChenMing-thu14","type":"user"},{"_id":"67e9fc3797cd6860c81d5838","avatarUrl":"/avatars/6c37731156bf52c123bd390823890d28.svg","isPro":false,"fullname":"Jangho Park","user":"jhpark96","type":"user"},{"_id":"699bd7d023b82d12c563b395","avatarUrl":"/avatars/e9c065d79fca4946a1e9e872cca09cb9.svg","isPro":false,"fullname":"Jaxon Scott","user":"jullopez","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"653b817d32c97d0655575872","name":"ByteDance","fullname":"ByteDance","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/0clr54wj5Ly-RkYU9OXPp.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.22344.md"}">
Bernini: Latent Semantic Planning for Video Diffusion
Authors: ,
,
,
,
,
,
,
,
,
,
,
Abstract
A unified video generation and editing framework combines multimodal large language models for semantic planning with diffusion models for pixel rendering, achieving state-of-the-art performance through semantic interface separation and enhanced positional embeddings.
AI-generated summary
Multimodal large language models (MLLMs) and diffusion models have each reached remarkable maturity: MLLMs excel at reasoning over heterogeneous multimodal inputs with strong semantic grounding, while diffusion models synthesize images and videos with photorealistic fidelity. We argue that these two families can be unified through a simple division of labor: MLLMs perform semantic planning, while diffusion models render pixels from high-level semantic guidance and low-level visual features. Building on this idea, we propose Bernini, a unified framework for video generation and editing. An MLLM-based planner predicts the target semantic representation directly in the ViT embedding space, and a DiT-based renderer synthesizes pixels conditioned on this plan, augmented by text features and, for editing, source VAE features for detail preservation. Because semantics serve as the interface, the planner and renderer can be trained separately and only lightly co-trained, preserving the pretrained strengths of both components while keeping training efficient. To better handle multiple visual inputs, we introduce Segment-Aware 3D Rotary Positional Embedding (SA-3D RoPE), and further incorporate chain-of-thought reasoning in the planner to better transfer understanding into generation. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks, with the MLLM's pretrained understanding translating into strong generalization on challenging editing tasks.
Community
Bernini is a unified framework for video generation and editing that uses an MLLM-based planner to predict latent semantic representations for a DiT-based renderer.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.22344 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.22344 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.22344 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.