CLoRA: A Contrastive Approach to Compose Multiple LoRA Models

Thank you for the supportive feedback. We respond to your comments below in two separate answers due to space constraints. Please feel free to let us know if you have any further questions.

Efficiency and Scalability Analysis: Thanks for pointing this out. We run a comparison between competing methods and ours in terms of time consumption when generating a composition. Our test-time optimization takes 24 seconds in an Nvidia H100 GPU for image composition using 2 LoRA models, while other models that require finetuning or training consume an additional 5-15 minutes before they can generate the composition. Moreover, while methods such as LoRA Merge or Multi-LoRA seem more efficient in terms of runtime, our results (Figures 3–5, Table 1) demonstrate much better performance in both qualitative and quantitative metrics for handling LoRA compositions efficiently. We revised our manuscript with an in-depth discussion. More details can be found in Appendix A.1, and Table 2 in our new manuscript.

Technical Novelty Requires Justification: Thank you for your comment. As highlighted in our paper (Lines 097-099), prior research has identified attention overlap and attribute binding issues as challenges in image generation tasks. Our novel observation, presented for the first time in this paper, is that these issues also arise in LoRA composition tasks. Specifically, when multiple LoRA models are composed, their attentions can overlap, leading to failures in generating the intended compositions. While this observation is novel, the core technical innovation of our work lies in the proposed contrastive objective function. This objective effectively separates the attention of multiple LoRA models during test-time, without requiring any additional training, user-provided masking, or conditions. To the best of our knowledge, both the observation regarding attention overlap in LoRA composition and the design of our method are novel contributions. Moreover, both our visual and quantitative results (Figures 3–5, Table 1) demonstrate superior performance for handling LoRA compositions compared to competing methods, showing the efficiency of the proposed method.

Missing citation and comparison: Thank you for bringing these related works to our attention. We included a comparison in our revised manuscript (please see Section 2). Furthermore, we would like to clarify the key differences between our method and LoRA-Composer. While LoRA-Composer also operates at test time, it requires user-provided bounding boxes as additional conditions, which significantly limits its practicality and ease of use. Additionally, their method is restricted to specific LoRA models, such as ED-LoRA, and is not compatible with the wide range of community LoRAs available on platforms like Civit.ai. Moreover, their method has substantially higher memory requirements, needing 60GB of memory for generating a composition, as noted in their GitHub repository, while our method only requires 25GB for composing two LoRA models. To provide a comprehensive evaluation, we included a comparison with LoRA-Composer in Appendix E and Figure 12. As shown in the results, while LoRA-Composer achieves comparable performance when user-provided bounding boxes are available, it fails to generate coherent compositions without them. In contrast, our method does not require any additional conditions, such as bounding boxes, and works seamlessly with any standard LoRA models, including those available on community platforms like Civit.ai.

Please see the next answer for other comments.

Efficiency Analysis: Thanks for pointing this out. We run a comparison between competing methods and ours in terms of time consumption when generating a composition. Our test-time optimization takes 24 seconds for image composition using 2 LoRA models, while other models that require finetuning or training consume an additional 5-15 minutes before they can generate the composition. Moreover, while methods such as LoRA Merge or MultiLoRA seem more efficient in terms of runtime, our results (Figures 3–5, Table 1) demonstrate much better performance in both qualitative and quantitative metrics for handling LoRA compositions efficiently. More details can be found in Appendix A, Table 2, and Figure 7 in our revised manuscript.

Does the approach work in cases where two similar concept categories are generated (e.g., “catA” and “catB”)?: Thank you for your thoughtful question. While contrastive loss might seem contradictory for similar concepts such as two cat LoRAs, our method is designed to handle such cases by applying contrastive loss to “attention maps” corresponding to distinct tokens, even when those tokens represent similar concepts. We included examples of our model depicting two cat LoRAs in the same composition, please see our revised manuscript, Figure 10. For example, in the case of the prompt "A L1 cat and a L2 cat," where the first cat is depicted by L1 LoRA, and the second cat is depicted by L2 LoRA, the model generates two separate attention maps, one for the “cat” with L1 features and one for the “cat” with L2 features. The contrastive loss is then applied to ensure that these attention maps remain distinct and non-overlapping, effectively separating the two cats in the final image. Therefore, our objective function is robust enough to handle similar concept categories by focusing on the token-level distinction in the attention maps, ensuring that each concept is properly represented without interference, even in the case of similar categories like "catA" and "catB."

Conditions for Mix-of-show experiments: To ensure a fair comparison between methods, we did not use additional control inputs like pose or canny edges for both qualitative and quantitative comparisons. It's important to note that in quantitative comparisons, using such controls would be very prohibitive, as we would need to generate specific control conditions for each prompt and LoRA combination in our experiments.

Thank you for the supportive feedback. We respond to your comments below in two separate answers due to space limitations. Please feel free to let us know if you have any further questions.

Lack of Uniform Notation and Typos: Thanks for bringing these to our attention. We used the same color convention for contrastive token pairs to facilitate easier reading while explaining the intuition behind our method. We have revised our manuscript to clarify this notation in Lines 249-250 and have also corrected the mentioned typos. Please refer to the updated version we have uploaded, where the changes are highlighted in green.

Poor Visualization Quality/Animated Style in Fig 3 and Lack of Resemblance in Fig 4: Regarding the Animated Style in Fig 3: We apologize for the error in Figure 3, where the original LoRA model for the dog example was depicted incorrectly. The correct version of this LoRA model, which indeed has an animation-style appearance, is shown in our teaser image, Figure 1. Therefore, our depictions in our LoRA compositions are accurate. We updated Fig 3 to reflect the correct image in our updated manuscript. Moreover, as can be seen from our results in both Fig 3 and Fig 4, when LoRA models depict realistic images such as a realistic shoe, plant, or bicycle, our compositions reflect these objects realistically.

Regarding the example that depicts the cup and book objects together in Figure 4: we acknowledge that some essential characteristics of the book might not be closely depicted to the original LoRA model. This challenge is likely due to the quality of the original LoRA model itself. It's important to note that none of the five competing methods performed this composition correctly; they either generated completely different cup or book objects or blended the book cover with the cup object. Compared to the competitors, our method successfully generated both objects without blending and with a closer resemblance than the other methods. We added a clarification about this example to the revised manuscript, please see lines 429-431.

Additional Evaluation Metrics: Thank you for this suggestion. We computed CLIP-I and CLIP-T scores, as can be seen from the following table. Note that our method performed the highest scores for both CLIP-T and CLIP-I scores. These results also support the DINO metrics presented in the main paper. We revised our manuscript to include these metrics, along with a discussion. Please see our revised manuscript, Table 1.

Please see our next answer for other comments.

Dear Reviewer KmNX,

We would like to kindly remind you that tomorrow is the deadline for reviewers to submit their comments. Should you have any further questions or need additional clarification, please do not hesitate to reach out.

We addressed all your concerns (see our rebuttal response above) and revised our manuscript accordingly. If you find our responses and revised manuscript have adequately addressed your concerns, we would appreciate if you could consider revising your score. Your support is invaluable to us and greatly motivates our ongoing efforts in this field.

Thank you again for your time. Your constructive feedback has significantly enhanced the quality of our revised paper.

Best,

Paper 3307 Authors

Dear Reviewer KmNX,

We hope this message finds you well. As today is the last day for us to provide comments, we are reaching out with a final message to draw your attention to our rebuttal and revised manuscript. We have carefully addressed all the concerns initially raised in your reviews through additional experiments, detailed clarifications, and revisions to our manuscript. These updates directly respond to the key issues and suggestions highlighted in your initial assessments.

Given the extensive efforts we have invested in addressing these points, we kindly encourage you to review our rebuttal and revised manuscript. We believe that our responses and additional experiments presented in our rebuttal effectively address the concerns, and we hope this will encourage you to revisit the score you previously assigned to our paper. Your insights are incredibly important to us, and your reevaluation would be greatly appreciated.

Thank you for your attention and contribution to the review process.

Warm regards,

Authors of 3307

Thank you for the supportive feedback. We respond to your comments below. Please feel free to let us know if you have any further questions.

Prompts with complex scenes and interactions: Thanks for pointing this out. While we used simple prompts in line with prior research and readability, our method does not constrain the usage of complex scenes or interactions between objects. CLoRA focuses mainly on managing the attention maps of subject tokens, ensuring their separation and fidelity while allowing other tokens to coexist and interact freely. Importantly, we do not explicitly diminish the attention values of tokens unrelated to the subject tokens, enabling the model to naturally handle contextual and interaction-based details in the scene. To illustrate our method’s effectiveness, we generated additional results with the following text prompts: “An L1 cat and an L2 dog playing with a ball together, near the beach with a ship in the background” and “An L1 cat and an L2 dog eating from the same plate, in a playground”. Please see Fig. 11 in the revised manuscript. As can be seen from the figures, our method can handle the interaction between objects such as a cat and a dog playing with a ball or eating from a plate. Moreover, our submitted copy includes examples where two subjects interact; such as Fig. 5 (b) where a person is holding a bunny, and Fig 5 (c) where a person is holding a flower (both people, bunny, and flowers are depicted by different LoRA models). Therefore, our model does not have a limitation in terms of handling more complex scenes.

Why not directly generate masks based on attention rather than a contrastive approach: Thank you for your insightful question. While generating masks based solely on the attention maps of specific concepts might seem sufficient at first glance, our ablation study (Section 4.2 and Figure 6) demonstrates that masking alone cannot fully resolve the challenges of attribute binding and attention overlap. Without the latent updates introduced by our contrastive framework, the model often struggles to correctly distinguish between multiple concepts. This limitation can lead to issues such as duplicate objects, incorrect attribute binding (e.g., generating two dogs instead of a cat and a dog), or the omission of certain concepts. Moreover, in cases of catastrophic neglect, where the cross-attention mechanism fails to allocate attention adequately to some subjects, the masks derived from these attention maps are inherently incomplete or inaccurate, as discussed by earlier works such as Attend & Excite. This failure results in subjects being omitted or misrepresented in the final image. The latent updates provided by our contrastive framework mitigate this issue by ensuring that each subject receives distinct and sufficient attention, enabling the model to faithfully incorporate all specified concepts into the composition. In contrast, masking alone, without latent updates, lacks this consistency and fails to prevent attribute blending or omissions, as evidenced by our experiments.

CLoRA has a limitation in the number of LoRA objects it can support: Thanks for pointing this out. We would like to highlight that supporting compositions with a large number of LoRA models is a challenging task for all competitor methods as well. This is primarily due to the memory consumption of individual LoRA models and the difficulty in effectively separating multiple LoRA concepts without blending them. Upon further examination of competing methods, we observed that ZipLoRA showcased compositions up to 2 LoRA models, MultiLoRA up to 3 LoRAs, and Mix of Show up to 5 LoRA models. Also, as can be seen in Figure 5, while our method faithfully generates multiple concepts such as 3 successfully, other methods such as LoRA Merge and MultiLoRA fail to generate the correct concepts. It is important to note, however, that Mix of Show requires additional inputs from the user, such as human pose or canny edges, whereas our method operates without these conditions. Further experiments conducted on a larger GPU (80 GB) allowed our model to handle up to 8 LoRA compositions, with runtimes ranging from 24 seconds (for 2 LoRAs) to 95 seconds (for 8 LoRAs). We have included a discussion and analysis of these findings in Appendix A, Table 2, and Figure 7 of the revised manuscript.

Dear Reviewer Sdkc,

We hope this message finds you well. As today is the last day for us to provide comments, we are reaching out with a final message to draw your attention to our rebuttal and revised manuscript. We have carefully addressed all the concerns initially raised in your reviews through additional experiments, detailed clarifications, and revisions to our manuscript. These updates directly respond to the key issues and suggestions highlighted in your initial assessments.

Given the extensive efforts we have invested in addressing these points, we kindly encourage you to review our rebuttal and revised manuscript. We believe that our responses and additional experiments presented in our rebuttal effectively address the concerns, and we hope this will encourage you to revisit the score you previously assigned to our paper. Your insights are incredibly important to us, and your reevaluation would be greatly appreciated.

Thank you for your attention and contribution to the review process.

Warm regards,

Authors of 3307

Insufficient contribution with the proposed dataset: As detailed in our Appendix, our benchmark dataset builds upon CustomConcept101. However, we would like to note that our contribution extends beyond merely adding 20 new concepts to CustomConcept101. While CustomConcept101 features multiple images across 101 concepts, it only includes 15 LoRA models and provides 125 unique prompts for composition. In contrast, we have trained 131 LoRA models that encompass both CustomConcept101 and our additional concepts. Furthermore, we offer 200 prompts designed for multi-LoRA composition, ranging from simple to complex arrangements. The contribution of our benchmark dataset lies in these trained LoRA models and the expanded set of prompts, all available for public use. However, we revised our manuscript to clarify these contributions, and instead of referring to it as a ‘benchmark dataset’ we changed it to ‘benchmark LoRA collection’ throughout the paper’. Please refer to the revisions in our manuscript, specifically Lines 895-897, 912-915, and Appendix D.

Clip-T and Clip-I should also be compared: Thank you for this suggestion. Based on your request, we computed CLIP-I and CLIP-T scores, as can be seen from the following table. Note that our method performed the highest scores for both CLIP-T and CLIP-I scores. These results also support the DINO metrics presented in the main paper. We revised our manuscript to include these metrics, along with a discussion. Please see our revised manuscript, Table 1.

Thank you for the supportive feedback. We respond to your comments below in 2 seperate answers due to length limitations. Please feel free to let us know if you have any further questions.

Novelty of Observations Mentioned in the Paper: We appreciate the reviewer pointing out prior works on attribute binding and attention overlap, such as Attend-and-Excite [1] and DisenDiff [2]. As noted in our paper (Lines 097–099), Attend-and-Excite [1] as well as other related works have highlighted these challenges in image generation. Our novel observation, presented for the first time in this paper, is that these issues also arise in LoRA composition tasks. Specifically, when multiple LoRA models are composed, their attentions can overlap, leading to failures in generating the intended compositions. While this observation is novel in itself, the main contribution and the core technical innovation of our work lies in the proposed contrastive objective function. Unlike existing methods that focus on a single model or require specialized variants (e.g., EDLoRAs), or user-provided controls, our approach operates on community-standard LoRAs in a plug-and-play, training-free, and test-time manner. This enables the seamless combination of LoRAs without blending or losing fidelity, as shown in our results (Figures 3–5, Table 1). To the best of our knowledge, both the observation regarding attention overlap in LoRA composition and the design of our method are novel contributions.

Additional Experiments with DisenDiff[2]: DisenDiff focuses on disentangling multiple concepts within a given image using attention calibration techniques. In contrast, our method addresses a fundamentally different problem: the composition of multiple pre-trained LoRA models into a single image. Our work deals with managing multi-LoRA attention overlap and attribute binding across independently trained LoRA models, enabling seamless and faithful LoRA compositions. Note that DisenDiff is not capable of creating images using multiple LoRA models unless a composed image is given as an input in the first place. For instance, if one wants to create a composition using a cat LoRA and a dog LoRA, DisenDiff would require an image that already depicts both concepts, to begin with. Composing both concepts in a single image is exactly the problem we are interested in. Our method leverages community-developed LoRA models to compose entirely new multi-concept images. This difference highlights that DisenDiff is not a framework for concept merging but rather one for disentangling concepts already present in a given image, making direct comparisons between the two approaches not possible. However, we included this work in our Related Work section in the updated manuscript, please see Section 2.

Additional Time for Test-time Optimization: Our method only requires test-time optimization, resulting in no additional time consumption beyond this phase. Unlike many of the competitors, we do not require finetuning or training. This test-time optimization takes 24 seconds for image composition using 2 LoRA models in H100 GPUs. Additionally, we have included a detailed comparison of runtime between our method and competing approaches, as detailed below. Also, we revised our manuscript to include a detailed discussion, please refer to Appendix A.

Please see our next answer for other comments.

Dear Reviewer qREh,

We would like to kindly remind you that tomorrow is the deadline for reviewers to submit their comments. Should you have any further questions or need additional clarification, please do not hesitate to reach out.

We addressed all your concerns (see our rebuttal response above) and revised our manuscript accordingly. If you find our responses and revised manuscript have adequately addressed your concerns, we would appreciate if you could consider revising your score. Your support is invaluable to us and greatly motivates our ongoing efforts in this field.

Thank you again for your time. Your constructive feedback has significantly enhanced the quality of our revised paper.

Best,

Paper 3307 Authors

Thank you for the supportive feedback. We respond to your comments below. Please feel free to let us know if you have any further questions.

User study details: The user study was made on 50 participants, where each participant was shown 48 images. The order of images shown to the users was randomized. We revised our manuscript to add screenshots (Please refer to Figure 9) and more details about the user study (please refer to the revised manuscript, Appendix B).

Too brief limitation and conclusion sections/data for processing times and number of LoRA models: Thanks for pointing this out. Based on your suggestion, we added a figure showing the processing time vs. the number of LoRA models for up to 8 LoRA models (see Fig. 7 in the revised manuscript). We also extended the discussion in both the limitation and conclusion sections (please refer to Section 5 and Section 6 in the revised manuscript). Also, for a more in-depth exploration of the runtime and resource consumption of our method and competitors, please see Appendix A in our revised manuscript.

Core innovation lacks novelty: As highlighted in our paper (Lines 097-099), prior research has identified attention overlap and attribute binding issues as challenges in image generation tasks. Our novel observation, presented for the first time in this paper, is that these issues also arise in LoRA composition tasks. Specifically, when multiple LoRA models are composed, their attentions can overlap, leading to failures in generating the intended compositions. While this observation is novel, the core technical innovation of our work lies in the proposed contrastive objective function. This objective effectively separates the attention of multiple LoRA models during test-time, without requiring any additional training, user-provided masking, or conditions. To the best of our knowledge, both the observation regarding attention overlap in LoRA composition and the design of our method are novel contributions. Moreover, both our visual and quantitative results (Figures 3–5, Table 1) demonstrate superior performance for handling LoRA compositions compared to competing methods, showing the efficiency of the proposed method.

Attention maps restrict information exchange between concepts: We would like to clarify this misunderstanding. The contrastive loss we designed in this paper does not enforce physical separation or distance between subjects in the generated image. Instead, it encourages that the attention maps corresponding to different subject-specific tokens (such as cat or dog) are non-overlapping. We note that this does not restrict information exchange or interactions between subjects. This is because the attention to tokens that are responsible for contextual relationships, spatial interactions, or broader scene coherence remains unconstrained and, therefore tokens can coexist and interact with each other. Additionally, we do not restrict self-attention, which ensures that pixels can interact freely with one another, preserving the flexibility and coherence of the diffusion process. Given that our contrastive objective operates in an optimization-based manner, based on a given text prompt, the degree of separation between attention maps across subjects will be optimized in a way that minimizes attention overlap and attribute binding; while still allowing for the generation of the context described by the specified text prompt. Our results (e.g., Figures 3–5) demonstrate that this balance enables the accurate and natural composition of multiple LoRA concepts without sacrificing the interplay between elements in the image.

Ethical Concerns: Thank you for raising this point. We initially used celebrity images in our work since related papers such as OMG (ECCV’24) have extensively utilized celebrity images. Additionally, the use of well-known faces allows readers to easily assess whether facial preservation has been maintained effectively. However, upon further reflection and research, we acknowledge that using these images without the celebrities’ consent could be considered unethical. For the camera-ready version of our paper, we will replace the celebrity images with the authors' images. Unfortunately, due to anonymity constraints, we were unable to make this change in the current submission. Thank you again for bringing this to our attention, and we will remove celebrity images from our camera-ready version.

Dear Reviewer 4FCc,

We would like to kindly remind you that tomorrow is the deadline for reviewers to submit their comments. Should you have any further questions or need additional clarification, please do not hesitate to reach out.

We addressed all your concerns (see our rebuttal response above) and revised our manuscript accordingly. If you find our responses and revised manuscript have adequately addressed your concerns, we would appreciate if you could consider revising your score. Your support is invaluable to us and greatly motivates our ongoing efforts in this field.

Thank you again for your time. Your constructive feedback has significantly enhanced the quality of our revised paper.

Best,

Paper 3307 Authors

Dear Reviewer 4FCc,

We hope this message finds you well. As today is the last day for us to provide comments, we are reaching out with a final message to draw your attention to our rebuttal and revised manuscript. We have carefully addressed all the concerns initially raised in your reviews through additional experiments, detailed clarifications, and revisions to our manuscript. These updates directly respond to the key issues and suggestions highlighted in your initial assessments.

Given the extensive efforts we have invested in addressing these points, we kindly encourage you to review our rebuttal and revised manuscript. We believe that our responses and additional experiments presented in our rebuttal effectively address the concerns, and we hope this will encourage you to revisit the score you previously assigned to our paper. Your insights are incredibly important to us, and your reevaluation would be greatly appreciated.

Thank you for your attention and contribution to the review process.

Warm regards, Authors of 3307

Dear Reviewers,

Since the rebuttal deadline is ending soon, we wanted to send a kind reminder about our responses and revised manuscript. We addressed all your concerns point by point, and revised our manuscript to reflect your suggestions. Your feedback is very important to us, and we will do our best to answer any further questions or concerns as soon as possible.

Best,

Paper 3307 Authors