DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation

Q1: Lack of State-of-the-Art Results Across All Metrics: DisEnvisioner does not outperform all baseline models in every metric. For instance, its performance in instruction response (C-I) and identity consistency (D-I) does not reach the top scores achieved by IP-Adapter and DreamBooth. This mixed performance limits DisEnvisioner’s claim to outright superiority across all customization aspects.

Thanks for your comment. C-I and D-I are both metrics used to assess identity consistency (please see lines 364-348). The key difference lies in their choice of image feature extractors: C-I uses the CLIP image encoder, while D-I employs the DINOv2 model. Together, these two metrics offer a more comprehensive evaluation of the performance in ID concsistency. The lower image-alignment scores of DisEnvisioner are due to its superior editability while avoiding merely copying the reference image. For more details, please refer to Common Concern-2.

Q2: Limited Test Dataset: The dataset used for evaluating DisEnvisioner is relatively constrained, potentially affecting the generalizability of the results. A more extensive and varied dataset would provide a clearer picture of the model's adaptability and robustness across diverse tasks and use cases.

Thanks for your valuable comment. To the best of our knowledge, the DreamBooth dataset we used is the only available testing dataset, and it is already in-the-wild, cover a diverse range of subjects, e.g., gods, cats, toys, shoes, packages, vases & teapots, robots, candles, etc. Moreover, since in the main paper, all our comparing methods have been evaluated exclusively on this dataset, we are also required to report metrics on it to ensure a fair comparison. Also for those transformer-based baselines (please see our response to Q2 of Reviewer cZ86), they are also primarily tested on DreamBooth dataset.

Q3: How does the feature enrichment step improve ID consistency, and could the authors provide more details on its implementation?

The feature enrichment in EnVisioner involves mapping the disentangled features from DisVisioner into a sequence of granular tokens using projectors (from one token to multiple tokens), as illustrated in Fig. 4(b) of our main paper. Specifically, separate projectors are used for the subject token and the irrelevant token, ensuring the disentanglement between subject-essential and irrelevant features. To inject the projected features into the SD model, we introduce two new cross-attention layers specifically for subject-essential tokens and irrelevant tokens within the pre-trained SD model. This separate injection strategy also ensures that subject features remain unaffected by irrelevant factors. During training, we optimize the projectors and new cross-attention layers while keeping other modules frozen, using the original diffusion loss. For inference, we set the weight of irrelevant features to 0 to filter out the irrelevant information.

As shown in Fig. 8 of our main paper and Tab. D and Fig. F of the updated supplementary material, EnVisioner produces high ID-consistency images, based on the the input image. Consequently, not only the evaluation results on image alignment (C-I, D-I) are improved significantly, the overall visual quality of the customized images are also clearly boosted.

Dear Reviewer 6rcz,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to end on November 26th, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 5736

Q1: Limited Theoretical Justification: The paper lacks sufficient theoretical grounding for the proposed disentangling mechanism. For example, the method for separating subject-essential and irrelevant features relies primarily on empirical observations without rigorous theoretical backing.

Thanks for your comment. In fact, the analysis of disentanglement mechanism can be found in previous image retrieval paper, i.e., Tokenizer[1]. We use orthogonal tokens to represent different semantic concepts of the image, where the image tokens are aggregated into different groups by the query $Q$ using spatial-wise attention. This aggregation of tokens is similar to clustering algorithm that clustering the features with same semantics into the same group. Therefore, in DisEnvisioner, two tokens are separated effectively, considering as subject-essential and subject-irrelevant (Lines 258-261 of the main paper). Pleaser refer to Common Concern-1 for further detailed explanation.

[1] Learning token-based representation for image retrieval

Q2: Comparison with Existing Methods: The paper does not provide extensive comparisons with some of the most recent advancements in text-to-image generation (e.g., Transformer-based approaches). Including a wider range of baselines would help position this work within the current state of the field.

Thanks for your advice.

We have conducted a detailed survey on the existing customized generation methods and found three latest Transformer-based methods[1][2][3]. We compare them in the table below and the results and corresponding discussions are also updated in the Sec. I of our supplementary material. In this table, we can show that DisEnvisioner still achieves SoTA performance when compared to Transformer-based methods, demonstrated by the highest overall ranking.

[1] Subject-driven Text-to-Image Generation via Apprenticeship Learning, NeurIPS2023

[2] KOSMOS-G: Generating Images in Context with Multimodal Large Language Models, ICLR 2024

[3] Customization Assistant for Text-to-image Generation, CVPR2024

Dear Reviewer cZ86,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to end on November 26th, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 5736

Q3: The EFFECT OF ENVISIONER has not been sufficiently substantiated; the paper's explanation regarding the EFFECT OF ENVISIONER only presents a few cases (as shown in Fig. 8), which lacks persuasiveness. You could provide a larger-scale comparison by human evaluation or image-alignment for ID-consistency (C-I, D-I) to measure the improvement in ID consistency or image quality.

Thanks for your valuable advice. We have conducted quantitative experiments for EnVisioner following the setup in Fig. 8 of the main paper. Please refer to the table in the last question. The results show a significant improvement in C-I and D-I compared to those of without EnVisioner, highlighting the effectiveness of EnVisioner. Additional visualizations are provided in Fig. F of the supplementary material. We also updated the quantitative results and analysis in Sec. F and Tab. D of our supplementary material.

Q4: Ablation on token’s numbers in DisVisioner lack of QUANTITATIVE result.You might measure performance across various metrics (like those in Table 1) for different token number configurations.

Thanks for your comment. We have quantitatively evaluated the effect of the number of tokens in DisVisioner, as shown in the table below (which also updated to the Tab. E of the supplementary material) .

1. When $n_s \neq n_i$, the image is primarily represented by the larger set without any disentanglement. Specifically, when $n_s > n_i$, the model tends to reconstruct the input image, resulting in lower C-T and higher I-V scores. C-I and D-I scores are also higher due to copying the reference image. When $n_s<n_i$, the subject is not properly represented in the $n_s$ tokens, and the model generates images solely based on the textual description. This leads to higher C-T and lower I-V scores but significantly lower C-I and D-I scores.
2. When $n_s = n_i>1$, the random initialized subject tokens have more “freedom”, capturing most image information and also leading to reconstruction. Under this configuration, the model exhibits lower C-T and higher I-V scores, with increased C-I and D-I scores due to copying the reference.
3. In contrast, when $n_s = n_i=1$, DisVisioner effectively disentangles the subject from irrelevant factors. This configuration achieves the best C-T, I-V , C-I and D-I scores, where C-I and D-I are not excessively high, as the model avoids simply reconstructing the reference image.

Q5: Could you please explain how subject features and irrelevant features are learned, and why they can be disentangled by transformer blocks without additional supervision?

Thanks for your comment. For a clearer explanation of disentanglement, please refer to Common Concern-1 and Sec. D, Fig. D of our supplementary material for more details. The ability to disentangle the image primarily relies on the Tokenizer technique [1], which is designed to learn semantically orthogonal features. These orthogonal features are disentangled through the SoftMax(·) function applied at spatial dimension within the spatial-wise attention. The role of Transformer blocks is to refine these disentangled features for enhanced representations as referred to [1], rather than to achieve disentanglement itself.

[1] Learning token-based representation for image retrieval

Q6: Supplement the comparison experiments for $n_s=1, n_i=0$ and $n_s=0, n_i=1$.

The experiments you suggested, $n_s=1, n_i=0$ and $n_s=0, n_i=1$, are the same configuration. $n_i=0$ or $n_s=0$ means there is no disentanglement effect under the training objective of reconstruction. Regarding disentanglement, we have already conducted experiments on comparing configurations with and without DisVisioner, as detailed in our response to Q2.

Q1: The advancement of the methodology has not been sufficiently demonstrated; some indicators (e.g. image-alignment for ID-consistency (C-I, D-I) as shown in Table 1) are inferior to other similar methods (e.g., IP-Adapter, BLIP-Diffusion, etc.).

Thanks for your comment, however, we respectfully disagree. The other three reviewers noted that our results are promising and impressive. We kindly recommend you referring to the Common Concern-2 for more details.

Furthermore, the evaluation results are considered as the strengths of DisEnvision, by Reviewer cZ86 ("DisEnvisioner demonstrates several strengths over competing methods in the areas of customization quality, editability, identity consistency, and efficiency. DisEnvisioner achieves the highest overall ranking among the methods tested, suggesting that it consistently performs well across different evaluation metrics.") and 6rcz ("The experiments are well-conducted, demonstrating the effectiveness of the proposed method in preserving ID consistency and reducing subject-irrelevant features.").

Q2: The EFFECT OF DISVISIONER has not been adequately explained; there is a lack of comparative experiments with and without DISVISIONER to validate its effectiveness. You cloud compare results with and without the DiVisioner component while keeping other parts of the system constant.

Thanks for your comment. In practice, it is infeasible to implement without DisVisioner while keeping other components unchanged, as EnVisioner relies strictly on the output of DisVisioner. In response, we implemented a replacement module, a simple MLP network with fully-connection layers and ReLUs, that eliminates the disentanglement capability of DisVisioner. Specifically, the CLIP global image feature, with the shape of $`(B, 1, 1024)`$, is fed into this MLP, and the output, which has the shape of $`(B, 1, 768)`$, is passed to EnVisioner. The batch-size $`B`$ is usually set to 1 during inference.

As shown in Sec. F and Fig. F of our supplementary material, “w/o DisVisioner”, whether combined with EnVisioner or not, struggles with editability. This is because irrelevant factors remain entangled with the subject. When DisVisioner is employed, it effectively disentangles the subject from other factors via feature aggregation in spatial-wise attention layer, resulting in improved editability. As the table below, we also computed metrics to quantitatively validate the effectiveness of DisVisioner. Poor editability is reflected in a low C-T score and high C-I, D-I, and I-V values.

We also updated the quantitative results and analysis in Sec. F and Tab. D of our supplementary material.

Dear Reviewer p8Sz,

Thank you for taking the time to review our paper and provide your valuable feedback. We have submitted detailed responses to your comments and suggestions, addressing the key points raised. As the discussion is scheduled to end on November 26th, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely important to us.

Thank you again for your time and effort.

Best,

Authors of Submission 5736

Dear Reviewer p8Sz,

I hope this message finds you well.

Thank you for your time and effort for reviewing our submission. As the discussion deadline (November 26th) approaches, we would greatly appreciate it if you could review our responses and provide any further clarifications at your earliest convenience. Your feedback is extremely valuable!

Thanks again for your time and effort!

Best,

Authors of Submission 5736

Dear Reviewer p8Sz,

Thanks so much for your prompt and kind response! We are thrilled to see that the you improved your rating!!!

Alao, we apologize for any inconvenience caused by our previous reminders. We were not aware about the extented discussion at that time.

However, we noticed that your current rating is "5: marginally below the acceptance threshold". We are not sure whether you have fully determined, if possible, we would greatly appreciate it if you could kindly consider sharing your further concerns that prevent you from rating this submission above the "acceptance threshold". With a few days remaining before the discussion closes, we highly value any opportunities to address any potential remaining issues. This will be extremely helpful to us.

We deeply appearciate your time and effort in reviewing our submission and writing responses.

Thanks,

Authors of submission 5736

Dear Reviewer p8Sz,

We sincerely apologize for any inconvenience our last comment may cause, we were not aware that your rating has been improved from 5 to 6.

Furthermore, please accept our sincerest gratitude for your positive feedback to our submission!!! We are so excited!!!

Thanks so much and wish you a great day!!!

Authors of submission 5736

Dear Reviewer p8Sz,

Thanks again for your kind response and improving the rating from 3 to 5!!!

However, as the discussion deadline approaches, we would greatly appreciate it if you could kindly consider sharing your concerns that prevent you from rating this submission above the "acceptance threshold". We highly value any opportunities to address any potential remaining issues. This will be extremely helpful to us!

Again, we deeply appearciate your time and effort in reviewing our submission and writing the responses.

Thanks,

Authors of submission 5736

Q2: Unclear Mechanism for Ensuring Disentanglement: While the concept of disentangling subject-specific and unrelated features is intriguing, the method section does not clearly explain how this disentanglement is achieved. It remains unclear what guarantees that only subject-related information is retained.

Thanks for your comment. The mechanism of disentanglement is explained in Common Concern-1. Please refer to it for further details.

Q3: Could the authors provide visualization examples or experiments of the learned disentangled tokens?

Thanks for your comment. We provide the visualization of the learned disentangled tokens in Fig. 7 of the main paper and more additional examples in Sec. D-F, Fig. D (Right), Fig. E and Fig. I of our supplementary material.

Q4: What types of augmentations are used during DisVisioner’s training?

The augmentations we used are mainly geometric transforms: HorizontalFlip, Rotate, Blur and ElasticTransform.

Q5: Can this method be extended to use multiple images as conditioning to enhance reference image details?

Yes, we can. As shown in Sec. H of our supplementary material, we extended DisEnvisioner to utilize multiple reference images. This extension is straightforward and does not require re-training the model. Specifically, for a set of images $X = \{\mathbf{x_1}, \mathbf{x_2}, \dots, \mathbf{x_n}\}$, we first extract their disentangled and enriched subject-essential features $f_i$ using DisVisioner and the projector $P^s$ in EnVisioner, i.e., $f_i = P^s(\text{DisVisioner}(\mathbf{x_i}))$. Next, we compute the average feature $f_\text{avg}$ across all subject features as $f_\text{avg} = \frac{1}{n} \sum_{i=1}^n f_i$. By injecting this averaged feature into the diffusion model, customized generation based on multiple reference images can be achieved.

However, in our paper, we primarily focus on single-reference image scenarios. This is because single-image scenarios are more practical, as multiple reference images are often unavailable in real-world applications.

Q6: Since DisVisioner’s training requires the class name from ImageNet for prior initialization, does this limit its generalization to classes outside ImageNet?

Thanks for your comment. In fact, we used the prompt templates from ImageNet and class names from OpenImage V6, which includes 600 classes. These classes cover most common categories and are sufficient for DisEnvisioner to generate images in diverse scenarios.

Thanks for your valuable advice.

Q1: Missing Ablations: The importance of CLIP prior initialization and augmentation in training DisVisioner has not been fully investigated.

1. Ablation on CLIP prior initialization (updated in Sec. D, Tab. B and Fig. D of the supplementary material):

   The “DisVisioner w/o CLIP prior” was implemented by randomly initializing both subject and irrelevant tokens. As shown in Fig. D, although rough disentanglement is achieved, it becomes difficult to clearly identify the subject token. The red borders highlight the uncertainty in the ordering of subject and irrelevant tokens. We further initialize the tokens in different orders: “CLIP prior (q0)“ represents the first token in the sequence is initialized with CLIP prior and “CLIP prior (q1)” denotes the second token is initialized with CLIP prior. As shown in Fig. 8, the CLIP prior effectively determines which token represents the subject-irrelevant information, and the effectiveness of disentanglement is further improved by CLIP prior.

   In terms of quantitative metrics, as shown in the table below, the performance of "CLIP prior (q0)" and "CLIP prior (q1) (Our DisVisioner)" are similar, as both effectively disentangle the subject and irrelevant tokens. When using "w/o CLIP prior," since the subject token cannot be clearly identified, we default to selecting the first token as the subject token. However, this results in degraded editablity and identity metrics due to the reduced quality of the subject token, which sometimes represents irrelevant components.

2. Ablation on augmentations in DisVisioner (updated in Sec. E, Tab. C and Fig. E of our supplementary material):

   We conduct an ablation study by removing augmentations during the training of DisVisioner. As shown in Fig. E of our supplementary material, although disentanglement is achieved through our tokenizer and CLIP prior, the generated images exhibit relative poor editability, as the model tends to directly copy irrelevant patterns and semantics from the input image, such as the poses of animals, the position of the subject in the reference image, and environmental lighting conditions. The red borders highlight these issues. When augmentations are applied during training, the model learns to extract essential subject features to reconstruct the original image under different transformations, rather than merely reconstructing exactly the same image.

   Due to poor editability, “DisVisioner $w/o$ augmentation” results in lower C-T score, as shown in the table below (also in Tab. C of our supplementary material). Additionally, since the model reproduces both subject-essential features and some irrelevant semantic and environmental features, it results in slightly higher interval variance (I-V) scores. This indicates that the generated images are influenced by irrelevant factors from the input images.

Dear Reviewer N4Js,

Thanks so much for your response! Your feedback is extremely important to us. Please do not hesitate to comment upon any further concerns.

Best,

Authors of Submission 5736