This paper presents two new prompt-aware weighting mechanisms for multi-concept LoRA composition in text-to-image diffusion models.</p>\n","updatedAt":"2026-06-04T12:45:05.646Z","author":{"_id":"68e651cdecf63bf29d0b7e14","avatarUrl":"/avatars/d7c55251f2b4de5e50423092aed35619.svg","fullname":"Georgios Tsoumplekas","name":"gktsoump","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7738147377967834},"editors":["gktsoump"],"editorAvatarUrls":["/avatars/d7c55251f2b4de5e50423092aed35619.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.03792","authors":[{"_id":"6a20604215100c5272a84492","user":{"_id":"68e651cdecf63bf29d0b7e14","avatarUrl":"/avatars/d7c55251f2b4de5e50423092aed35619.svg","isPro":false,"fullname":"Georgios Tsoumplekas","user":"gktsoump","type":"user","name":"gktsoump"},"name":"Georgios Tsoumplekas","status":"claimed_verified","statusLastChangedAt":"2026-06-04T12:40:27.728Z","hidden":false},{"_id":"6a20604215100c5272a84493","name":"Stella Bounareli","hidden":false},{"_id":"6a20604215100c5272a84494","name":"Vasileios Argyriou","hidden":false}],"publishedAt":"2026-06-02T00:00:00.000Z","submittedOnDailyAt":"2026-06-04T00:00:00.000Z","title":"Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting","submittedOnDailyBy":{"_id":"68e651cdecf63bf29d0b7e14","avatarUrl":"/avatars/d7c55251f2b4de5e50423092aed35619.svg","isPro":false,"fullname":"Georgios Tsoumplekas","user":"gktsoump","type":"user","name":"gktsoump"},"summary":"Low-Rank Adaptation (LoRA) successfully enables personalization in text-to-image generation by adapting pre-trained diffusion models to specific visual concepts and styles. However, extending such models to multi-concept customization remains challenging. Naively combining multiple LoRA weights or their outputs often leads to interference among concepts, resulting in degraded visual quality and reduced fidelity to the reference images of individual concepts. This paper proposes a simple yet effective approach for multi-concept customization by optimally combining the outputs of multiple LoRA modules. We leverage the relative importance of each concept during generation, as inferred from its corresponding prompt tokens and introduce two methods, W-Switch and W-Composite, that employ a prompt-aware importance weighting strategy in which each LoRA is weighted according to the semantic influence of its trigger words in the target prompt. In addition, we extend existing quantitative evaluation metrics by proposing a new image-based similarity evaluation framework that assesses image fidelity and identity preservation through comparisons between real-world reference images and automatically segmented concept regions from generated images. We evaluate our approach on the ComposLoRA testbed and demonstrate consistent improvements over existing state-of-the-art methods in terms of visual quality, identity preservation and compositionality. Qualitative evaluations, including a Large Language Model (LLM) based assessment and a user study, further validate the effectiveness of the proposed methods and align with the newly introduced quantitative image-based metrics. Our code is available at https://github.com/GeorgeTsoumplekas/Prompt-Aware-Multi-LoRA-Composition.","upvotes":0,"discussionId":"6a20604315100c5272a84495","githubRepo":"https://github.com/GeorgeTsoumplekas/Prompt-Aware-Multi-LoRA-Composition","githubRepoAddedBy":"user","ai_summary":"Multi-concept customization in text-to-image generation is improved through prompt-aware weighting strategies that reduce interference between learned visual concepts.","ai_keywords":["Low-Rank Adaptation","diffusion models","multi-concept customization","LoRA modules","prompt-aware importance weighting","W-Switch","W-Composite","image-based similarity evaluation","ComposLoRA","visual quality","identity preservation","compositionality"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.03792.md"}">
Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting
Abstract
Multi-concept customization in text-to-image generation is improved through prompt-aware weighting strategies that reduce interference between learned visual concepts.
Low-Rank Adaptation (LoRA) successfully enables personalization in text-to-image generation by adapting pre-trained diffusion models to specific visual concepts and styles. However, extending such models to multi-concept customization remains challenging. Naively combining multiple LoRA weights or their outputs often leads to interference among concepts, resulting in degraded visual quality and reduced fidelity to the reference images of individual concepts. This paper proposes a simple yet effective approach for multi-concept customization by optimally combining the outputs of multiple LoRA modules. We leverage the relative importance of each concept during generation, as inferred from its corresponding prompt tokens and introduce two methods, W-Switch and W-Composite, that employ a prompt-aware importance weighting strategy in which each LoRA is weighted according to the semantic influence of its trigger words in the target prompt. In addition, we extend existing quantitative evaluation metrics by proposing a new image-based similarity evaluation framework that assesses image fidelity and identity preservation through comparisons between real-world reference images and automatically segmented concept regions from generated images. We evaluate our approach on the ComposLoRA testbed and demonstrate consistent improvements over existing state-of-the-art methods in terms of visual quality, identity preservation and compositionality. Qualitative evaluations, including a Large Language Model (LLM) based assessment and a user study, further validate the effectiveness of the proposed methods and align with the newly introduced quantitative image-based metrics. Our code is available at https://github.com/GeorgeTsoumplekas/Prompt-Aware-Multi-LoRA-Composition.
Community
This paper presents two new prompt-aware weighting mechanisms for multi-concept LoRA composition in text-to-image diffusion models.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.03792 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.03792 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.03792 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.