EZIGen: Enhancing zero-shot subject-driven image generation with precise subject encoding and decoupled guidance

1. Novelty: The idea of encoding features from given images and injecting them into the Unet has become a very common method for current identity preservation methods. Besides, the idea of adding features from pre-trained UNET is somewhat like the famous ControlNet method. The difference is that ControlNet fine-tunes the controlnet work while this work fine-tunes the main unet.
2. The method is complex and requires three UNET in the generation process (ref. unet, main unet (fixed) and main unet (trainable)), which would limit the application scenarios with very large computational cost and resources.
3. Quantitative Performance: This paper provides several quantitative metrics to demonstrate its effectiveness. However, the proposed method fails to achieve SOTA results on many metrics and also fails to make significant improvements on other metrics. It makes me doubt about the contribution made by this work.
4. Qualitative Performance: This paper provides many visualized examples. However, I find most of them are very simple, the subjects are simple and the layout are basically in the center of images. For example, the clock in Fig. 6 1) is just the clock example shown in DreamBooth. So I wonder why prior works cannot deal with it correctly. Besides, there exist beard chaneg in the example of Fig. 6 2) as well.
5. In addition, for the example of subject interpolation of a dog astronaut, I found quality degradation of other parts and unnatural animal heads (with blurs and artifacts).

1. The method’s use of distinct prompts at different time steps, while beneficial, has been thoroughly explored in prior work, notably in methods such as “P+: Extended Textual Conditioning in Text-to-Image Generation.” This limits the novelty of the contribution in EZIGen, as previous research already demonstrates the benefits of staging prompts to improve detail and fidelity in generated images

2. The experiments rely heavily on quantitative benchmarks without incorporating human assessments, which are crucial for assessing subjective factors like image realism, diversity, and prompt alignment in a nuanced manner.

3. While the paper identifies a crucial problem—differences between the layout of the generated image and the reference subject image—it proposes maintaining layout similarity as a solution. This approach, however, may inadvertently limit text-prompt alignment, especially when prompts require transformations or settings different from the original subject image. Addressing this conflict with a more adaptive mechanism could improve alignment without constraining diversity.

1. The comparison with existing methods is somewhat lacking, particularly with methods such as [1, 2, 3] that focus on image prompt adaptation and subject-driven generation.
2. The experiments do not include a user study.
3. The strategy of "repeating until the similarity between the newly generated image and the previous image exceeds a predefined threshold" may lead to overconsumption. In my understanding, the iteration number possibly exceeds 10 if the threshold is wrongly set.
4. There is a lack of analysis on how performance changes with an increase in the number of iteration steps.

[1] IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models [2] SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation [3] MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance

There are several issues limiting the quality of the paper. First, extracting features from the reference image and injecting it to the generation process for identity preservations is a widely used stratagem. Second, the proposed method decouples the generation process into three parts, namely injection, layout generation, and appearance transfer. This may lead to extra complexity compared to other methods. Actually, it seems that the appearance transfer step contributes most to the visual identity preservation effect. Such a process can actually be added to any existing method to improve visual quality. Third, according to Tables 1,2,3, the performance improvements of the proposal is limited. Also, some of the visual results are not convincing enough. Also, it seems to me that the diversity of the object appearance is too limited probably due to the appearance transfer module. For example, in Figure 6 2), it seems that the face of the subject is simply pasted to different backgrounds.