EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM

Dear Reviewer gzGG,

Thanks for your advice. We will address your concerns below.

Q1. Missing references and discussions.

Thank you for raising this point. We have discussed the differences between these MLLM-based frameworks and EasyRef in the Q1 of rebuttal for Reviewer nUGh. We will cite and discuss these image editing and generation methods with MLLMs in the Related Work Section of our revised manuscript.

Q2. The novelty of this paper.

Please refer to the Q1 of rebuttal for Reviewer nUGh.

Q3. Multi-reference consistent generation capability of other MLLM-based frameworks.

1. Multi-image consistent generation is not achievable simply by supporting multi-image input in a framework; it requires architecture optimization, novel training algorithms, and collaborative integration with data construction to achieve the desired results.
2. Kosmos-G is another MLLM-based framework that can accept multi-image input. However, it focuses more on the composition of distinct elements and lacks the capability to extract consistent elements and generate consistent images based on those elements. For example, in its official setting for benchmarking DreamBench, which contains multiple images per instance, Kosmos-G uses only a single reference image. In addition, as shown in our comparison with EasyRef on the multi-image consistent generation task (https://github.com/anonymous-projectuser/image), visualizations demonstrate that EasyRef produces higher-fidelity and more consistent results.

Dear Reviewer nUGh,

Thank you for appreciating our approach. We will address your concerns below.

Q1. The novelty of this paper.

While we acknowledge that our project does not introduce groundbreaking new architectures, we wish to emphasize that it offers several important conclusions and methods that contribute novel insights to the community that are untouched by prior works. These conclusions and methods include identifying key limitations, developing effective data construction strategies, optimizing architecture and training for efficiency, and creating a new benchmark. Together, they provide a practical blueprint for building and evaluating a simple yet effective pipeline for multi-image consistent generation. We want to highlight three parts of differences.

1. Although some methods also utilize MLLMs and diffusion models, as acknowledged by Reviewer sCdX and nUGh, our design objectives, architecture and training optimizations, and tasks differ fundamentally. Other approaches leveraging MLLMs primarily use them for feature extraction in explicitly defined tasks (e.g., single-subject customization with a single reference image). In contrast, we demonstrate that MLLMs possess strong task instruction and multi-image comprehension capabilities, enabling them to extract consistent information based on instructions and multi-image contexts. Our method emphasizes the flexible understanding and extraction of consistent elements across multiple references, guided by instruction-based control.
2. We demonstrate that EasyRef with a simple architecture, can serve as a general-purpose model for multi-image customization and effectively cover various types of reference customization, such as subject, style, character, and face, rather than being limited to single-subject preservation or single-image subject-driven tasks. Consequently, EasyRef avoids the need for elaborate image preprocessing, such as subject-specific masking, making the process simpler and applicable to a broader range of scenarios.
3. By leveraging the unique capabilities of MLLMs combined with data construction methods and efficient designs, we achieve strong performance on the general multi-image consistent generation task. Our work focuses on building a comprehensive pipeline, including data construction, optimization of architecture and training strategies, and benchmark creation, rather than proposing an entirely new architecture.

Together, we believe these underscore the impact and utility of our work, providing practical insights and methodological advancements that benefit the broader research community.

Q2. Additional computational complexity introduced by using an MLLM to process multiple reference images and text prompts.

We analyzed the model's performance and inference efficiency when varying the number of reference images, as shown in Figure 8. The table below presents the inference latency. The "baseline" refers to the standard SDXL text-to-image generation without image references. Our results show that incorporating an MLLM with multiple reference images only increases latency by 10%-20%.

Q3. Customized generation when the multiple reference images contain the same two or more distinct targets.

We provide visualizations of customized generation using two subjects as references at https://github.com/anonymous-projectuser/image. As shown in the examples, the generated cat and dog exhibit high fidelity to their respective reference subjects. EasyRef leverages the instruction-following and multi-image comprehension capabilities of MLLM to flexibly identify and preserve consistent elements (e.g., two subjects) without the need for complex image preprocessing.

Q4. The impact of different LLMs and different diffusion models.

We compare several EasyRef variants on MRBench, as shown in the table below. First, we observe that using a stronger base MLLM or diffusion model improves performance. Second, larger MLLMs significantly increase model complexity and latency. Therefore, we select Qwen2-VL-2B and SDXL to achieve the best trade-off between performance and efficiency.

Dear Reviewer 35bv,

Thanks for your comments. We will address your concerns below.

Q1. What are the "Inconsistent Result" shown in figure 2? There's no spatial information encoded in subject embeddings.

1. The essence of the "Inconsistent Result" lies in the insufficient understanding of multiple reference features. Traditional methods fail to distinguish whether these features correspond to multiple subjects or different representations of the same subject. As a result, features corresponding to different spatial positions of the same subject are uniformly decoded, leading to the repetition of multiple subjects in the output.
2. Some subject-driven methods utilize subject-specific segmentation masks to exclude background and spatial information from the reference images. However, using a mask limits the generalizability of the framework. For instance, it becomes unsuitable for style and position-aware reference. General-purpose methods, such as IP-Adapter, avoid relying on masks, which can lead to such inconsistencies. Similarly, EasyRef does not elaborately preprocess input images, as it prioritizes generality over being limited to subject-driven tasks.
3. Additionally, using masks increases the complexity of the pipeline by requiring an extra segmentation model. Furthermore, inaccurate masks can degrade performance. For users, manually annotating masks during inference adds to the overall cost and reduces usability.
4. In cases where multiple subjects occlude each other (e.g., as discussed in Q5 of the rebuttal for Reviewer sCdX), using masks or bounding boxes may result in inaccurate subject embeddings.

Q2. The novelty of this paper.

Please refer to the Q1 of rebuttal for Reviewer nUGh.

Q3. EasyRef uses significantly more training data compared to MOMA.

1. MOMA is specifically designed for subject-driven generation task using a single reference image, whereas EasyRef is a general framework for multi-image consistent generation. EasyRef is capable of preserving various consistent elements, including subjects (e.g., common animals or objects), characters, styles, and human faces. Given their distinct design objectives and pipelines, directly comparing the data scales of the two approaches is neither fair nor meaningful.
2. Furthermore, our multi-image fine-tuning leverages approximately 350K clean subject-driven target images, which is comparable in scale to the data used by MOMA.

Q4. EasyRef has almost identical performance as MOMA on DreamBench.

1. The DINO score is preferred by DreamBooth because its self-supervised training objective encourages the model to preserve fine-grained subject features. This makes it uniquely well-suited, compared to CLIP scores, for evaluating subject fidelity in generated images.
2. As shown in Table 3, EasyRef surpasses MOMA by 1.8% in DINO score. This performance gain is not trivial since MOMA improves the DINO score of IP-Adapter from 61.2% to 61.8% on the DreamBench.

Q5. It's not necessary to include baseline methods that are too weak and non-SOTA.

We will carefully exclude some weak baselines in the final version.

Q6. Comparing EasyRef with SOTA facial reference methods and face similarities are not evaluated.

We conducted qualitative comparisons (visualizations in https://github.com/anonymous-projectuser/image) showing EasyRef’s superior facial fidelity and quantitative evaluations using the face similarity metric adopted by PuLID on the MRBench:

EasyRef outperforms SOTA methods in identity preservation, and these comparisons will be emphasized in the revised manuscript.

Q7. This paper follows the terminology of MOMA to call the VLM as multimodal LLM (MLLM), it's more accurate to refer to it as VLM.

We use "MLLM" to align with its growing adoption in the community for architectures combining vision encoders, projectors, and LLMs (e.g., LLaVA). While terms like "LMM", "VLM" or "LVLM" are also commonly used, our choice reflects current conventions rather than solely following MOMA.

Q8. The LLM of the model is quite weak.

Please refer to the Q4 of rebuttal for Reviewer nUGh.

Q9. Both MOMA and EasyRef don't use complicated language capabilities.

Our synthetic captions (typically 1–3 sentences) are more detailed than conventional single-sentence prompts and require strong language comprehension capability to encode. Unlike MOMA, which only focuses on subject categories, EasyRef explicitly controls the extraction of consistent elements like face, style, and subject through instructions, better utilizing the model’s instruction-following capabilities.

Q10. The first 3 pages are written disproportionately.

We will streamline the opening sections and expand Sections 2.2 and 2.3 in the revised manuscript.

Dear Reviewer sCdX,

Thanks for appreciating our work and your advice. We will address your concerns below.

Q1. Can EasyRef capture intricate details?

We provide visualizations on the DreamBench benchmark (https://github.com/anonymous-projectuser/image), demonstrating our method's ability to preserve intricate texture details. We also show a failure case in the last row. EasyRef fails to accurately preserve the textual details, primarily due to the inherent limitations in text rendering capabilities of the SDXL base model.

Q2. The collection method does not guarantee that only the same instance will be grouped together.

1. We acknowledge that subjects belonging to the same category but not the same instance can be grouped together. However, this does not significantly impact subject-driven performance. Multi-image subject-driven generation requires that the main subjects across multiple reference images represent the same instance, and the generated outputs should contain the same instance. When multiple reference images depict subjects from the same category but not the same instance, using a target image with a subject from the same category is still a reasonable training approach.
2. To further enhance subject-driven performance, we have also constructed image groups using a stricter data filtering strategy (described in line 797 of our paper) by applying a higher DINO score threshold. Additionally, as discussed in line 250, we collected high-quality images and used them to train subject LoRA models. These LoRA models were then used for generating additional training data.
3. We randomly sampled 200 images from the training dataset and found that the proportion of such data is small (11 out of 200).

Q3. Ablating the design of the reference aggregation.

We have already conducted ablation studies on the aggregation designs, as shown in Table 5 and Table 6 of our paper. Additionally, we provide more ablations in the table below:

From these results, we observe (1) using bidirectional attention in the final layer enables better interaction among reference tokens, (2) inserting reference tokens before the first layer of the LLM increases training costs but does not yield significant performance gains, and (3) the adopted decoupled cross-attention mechanism is more effective and efficient compared to adding new cross-attention layers.

Q4. The improvement of EasyRef seems marginal on DreamBench.

Please refer to the Q4 of rebuttal for Reviewer 35bv.

Q5. It would be more helpful if other failure examples could be provided.

We have included two additional failure cases for analysis in https://github.com/anonymous-projectuser/image:

1. Attribute Confusion: When the regions of the dog and the chair in the reference images significantly overlap (e.g., the dog partially or fully covers the chair), simply averaging the features from these reference images can result in attribute confusion. For instance, the generated dog might inherit the chair's color, while the chair adopts the dog's color, leading to inconsistent generation results.
2. Subject Hallucination: When the dog appears in front of the chair in one reference image but is seated on the chair in another, the simple fusion method may be misled by the positional discrepancy. This can result in subject hallucination, where the diffusion model generates an additional dog-shaped object on the chair.

Q6. EasyRef should be compared with more strong baselines.

We will cite and discuss these state-of-the-art methods, such as CustomDiffusion [1], MS-Diffusion [2], and λ-ECLIPSE [3] in the final version. Their performance scores will be incorporated into Table 3.

Q7. In Fig. 1, the task is not very clearly showed.

We will adjust the teaser figure and modify its caption to make the task clear.

Q8. During the alignment pretraining, are the MLLM and Diffusion trained together or is the MLLM the only model pretrained?

We insert newly initialized reference tokens into the final layer of the LLM for aggregation. This makes the alignment pretraining process efficient, as we only train the final layer of the LLM, the reference tokens, the newly added cross-attention adapters in the U-Net, and the condition projector. The rest of the model remains frozen.

[1] Multi-Concept Customization of Text-to-Image Diffusion

[2] MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance

[3] λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space