IFAdapter: Instance feature control for grounded Text-to-Image Generation

Please kindly provide visual comparisons between the ground truth (GT) and the generated images.

We thank the reviewer for their feedback. Since Fig. 1, Fig. 4, Fig. 9, and Fig. 10 all showcase zero-shot results, there are no corresponding ground truth (GT) images. However, we agree that it is important to provide comparisons that include GT results for reference. Therefore, we have added visual comparisons between the ground truth (GT) and the generated images, which can be accessed at this link or Fig. 11 in the revised manuscript.

References:

[1] Avrahami, Omri, et al. "Spatext: Spatio-textual representation for controllable image generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Jia, Chengyou, et al. "Ssmg: Spatial-semantic map guided diffusion model for free-form layout-to-image generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.

We thank the reviewer for their in-depth review and questions. We are pleased that they recognize the design philosophy behind our method and appreciate their acknowledgment of the valuable insights provided by the user study. Below are our point-by-point responses to the questions.

The quality of the proposed dataset has not been evaluated. It appears that all ground truths (GTs) are generated by existing Vision Language Models (VLMs). A human-level quality assessment would be beneficial for greater impact within the community.

We believe that the cost of manually labeling a large-scale dataset is substantial. In order to ensure the quality of the labeling, we have chosen the state-of-the-art (SOTA) VLM and optimized the prompt engineering. To further assess the extent to which hallucinations affect our dataset, we conducted a user study. We manually reviewed 500 randomly selected samples from the training dataset, of which about 3% were identified as having location errors and 6% as having labeling hallucination errors, suggesting that the overall negative impact on model training is not particularly significant.

The assertion that the proposed component can “seamlessly empower various community models with layout control capabilities without retraining” (l.113) may be misleading. The IFAdapter is fundamentally a training-based method, and the phrase “without retraining” only holds true when applied to spaces closely aligned with COCO IFG, as the IFAdapter does not demonstrate zero-shot capabilities in this paper.

We sincerely apologize for the misunderstanding, and we have revised our wording to eliminate the ambiguity. Our method is not limited to performing well only within the COCO IFG space, as all the cases shown in Fig. 1, Fig. 4, Fig. 9, and Fig. 10 are zero-shot results.

The semantic-instance map does not appear to be novel. Please refer to "BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion" (ICCV 2023) and other zero-shot L2I methods for comparison.

We sincerely thank the reviewer for the valuable feedback. While the semantic-instance map approach is not proposed for the first time by us (as demonstrated in works like Spatext [1] and SSMG [2]), those methods fill the semantics of all instances into a single semantic map to guide generation. This can lead to semantic conflicts in regions where multiple instances overlap. In contrast, our method generates feature maps for each instance separately and fuses them into the final feature map using a gated semantic fusion mechanism, effectively resolving the semantic conflict issues might present in previous methods. We have analyzed this gated semantic fusion mechanism in detail in Appendix E.

Regarding zero-shot methods, we compared our approach with DenseDiffusion and Multi-Diffusion. We also considered including BoxDiff as a baseline but ultimately decided against it for the following reasons:

• BoxDiff requires concatenating all instance descriptions into the caption, and the total length of these descriptions cannot exceed the maximum input length for CLIP. However, in our instance-level dense captioning setup, there are cases where the concatenated descriptions exceed this limit.
• In BoxDiff, each bounding box (bbox) typically corresponds to a single noun. In contrast, our benchmark associates an entire description with each instance, requiring a bbox to correspond to multiple words. Our experiments show that this mismatch causes BoxDiff to generate corrupted results. We provide some example cases at this link to illustrate the issues.

The appearance tokens show only minor improvements in Table 3. Additional explanations regarding this observation would be appreciated.

Thank you very much for your thoughtful suggestion. The appearance tokens improved the IFS rate by 10 points, which we believe is not insignificant.

The relatively smaller improvement compared to the boost brought by EoT tokens can be attributed to the specific role of appearance tokens in assisting the generation of high-frequency details. Since the EoT token already encapsulates a certain degree of coarse appearance semantics, it is sufficient to effectively generate basic instance appearances in many cases. Appearance tokens, on the other hand, primarily aid in handling more challenging cases, which constitute a smaller proportion of all cases. Consequently, their overall contribution appears relatively modest.

Dear Reviewer,

As the rebuttal is coming to a close, we would like to kindly provide a gentle reminder that we have posted a response to your comments. If you don't mind, may we please check if our responses have addressed your concerns and improved your evaluation of our paper? We are happy to provide further clarifications to address any other concerns that you may still have before the end of the rebuttal.

Sincerely,

Authors

Authors introduce a new benchmark and evaluation metric for this task. Why can't they use the evaluation setup and metrics as InstanceDiffusion? If authors find flaws in InstanceDiffusion's setup, I recommend authors point it out and discuss the advantages of the IFG Benchmark (setup) and IFS Rate (metric). There is no point in creating multiple new benchmarks when existing ones are already rigorous. For IFS Rate authors use Grounding DINO whereas InstanceDiffusion uses YOLO. Please compare with InstanceDiffusion using their exact setup (COCO and LVIS val set and their metrics) to support your claims.

We agree that the experimental setup of InstanceDiffusion is comprehensive and rigorous. However, the main experiments of InstanceDiffusion (Table 1) primarily focus on evaluation metrics for object positions and do not include metrics for assessing instance features, which is the core focus of our setup. Therefore, our IFG Benchmark is designed not only to evaluate positional accuracy (AP) but also to complementarily assess the quality of instance feature generation using the IFS Rate.

We chose to use Grounding DINO because it has been pre-trained on large-scale datasets, offering higher generalization capabilities compared to YOLO. Given that our setup targets the generation of instances with complex features, we believe Grounding DINO is a more suitable choice.

Based on the reviewer's suggestion, we have reported the performance of our method on the COCO validation set as a reference. We did not include results on the LVIS dataset because InstanceDiffusion does not provide validation scripts for the LVIS dataset.

The experimental results suggest that our method performs worse than InstanceDiffusion under its experimental setup. As shown in the table, our method achieves lower $AP^{box}_{s}$ for small objects. After visualizing the samples, we believe this is primarily due to the presence of many small instances in the COCO validation set. Our approach leverages an Instance Semantic Map with a 16×16 resolution as guidance. Consequently, for bounding boxes occupying small areas, positional deviations are relatively large, leading to inaccuracies in the position generation of small instances. In contrast, methods like InstanceDiffusion, which represent objects using tokens, are relatively better for small-object generation.We appreciate the reviewer for pointing out this limitation of our method, and we will include this shortcoming in the limitations section of our paper.

But we would like to highlight that this shortcoming does not impact IFA's performance under our setup. This is because only objects with larger areas typically exhibit noticeable appearance features, and generating small objects is not the primary target of the IFG task.

Authors claim that the a lightweight network $f$ provides an "importance" score for location (x,y) in the ISM construction and use it to compute D(x, y). Please show qualitative or quantitative evidence that the network f infact does what is claimed in the paper. While the idea sounds reasonable, I suspect how f learns to predict the right "importance" scores without supervision.

We thank the reviewer for their suggestion. We have added Appendix E to discuss the relevant content, and the importance score visualization can be found there. Here, we provide a brief summary of the discussion.

Despite the lack of explicit supervision, 𝑓 implicitly learns from the dataset through the denoising loss to identify salient features in the Instance Semantic Map at various positions and assign greater weights to the corresponding instances. We hypothesize that this may be due to diffusion model features inherently containing a certain degree of depth priors, which 𝑓 learns to map to weights during the training process.

Please remove point 3 in contributions. Comprehensive experiments are not a contribution but are rather required to support the claims made in the paper.

We thank the reviewer for the feedback. In response to the identified weaknesses, we have added the corresponding experiments to strengthen our claims. We hope this addresses your concerns.

We thank the reviewer for their careful review and toughtful questions. We are honored that they found our model design to be both simple and effective. Below are our point-by-point responses to the questions.

Authors claim IFG as their first contribution. But this task has already been introduced in InstanceDiffusion. InstanceDiffusion uses detailed instance captions in addition to the global caption. What is the difference between their setup and IFG?

Thank you very much for your feedback. InstanceDiffusion is an impressive and pioneering work. Based on our understanding, the primary setup of InstanceDiffusion consists of three key aspects:

• Allowing flexible conditioning (points, scribble, etc).
• Enabling basic attribute (texture, color) binding for generated instances.
• Supporting dense multi-instance generation.

For attribute binding, the experiments in InstanceDiffusion primarily demonstrate its capability to bind single colors or textures. In the limitations section of InstanceDiffusion, the authors state: "We also find that texture binding for instances poses a challenge across all methods tested, including InstanceDiffusion." Additionally, through experiments (Fig. 1), we observed that more complex attribute binding problems remain unresolved. Therefore, we consider the InstanceDiffusion setup to be well-suited for generating accurate instance positions with multi-type conditioning and simple attribute binding.

In contrast, our setup specifically targets the generation of complex instance-level attributes, such as textures, mixed colors, patterns, and more—problems that InstanceDiffusion has yet to solve. The appearance tokens and instance semantic maps proposed in our IFA are designed specifically to address these challenges in complex attribute binding. Furthermore, to evaluate the accuracy of complex instance-level attribute generation, we are the first to propose a verification pipeline based on Vision-Language Models (VLMs).

The related works section has not been used correctly in this work according to my opinion. Authors just cite all relevant work but fail to differentiate their work from the literature. Please discuss how IFG is different from prior work and how their method is different from previously proposed methods in the related works section.

We thank the reviewer for their suggestion. We have revised the Related Work section accordingly in the newly uploaded manuscript.

If authors believe local CLIP score is suboptimal. I would recommend authors show (quantiatively) why VLMs are better than CLIP for this task. Please refrain from introducing a new metric and benchmark unless absolutely necessary.

Thanks very much for the reviewer’s suggestion. There are several reasons why we do not use CLIP score as an evaluation metric:

• CLIP score measures the alignment between the global feature vectors of an image and a text description. However, previous works [2, 3] have pointed out that CLIP tends to treat sentence features as a "bag of words," making it difficult to distinguish fine-grained details. For example, CLIP struggles to differentiate between "a person wearing a white shirt and black pants" and "a person wearing black pants and a white shirt." This limitation means that CLIP score cannot effectively evaluate the correctness of feature binding. In contrast, Vision-Language Models (VLMs) encode images at the patch level, offering finer-grained perception. Additionally, our prompt engineering guides VLMs to focus more on detailed features, making them more effective for detecting instance-level attributes.
• When calculating evaluation metrics, it is generally desirable for the metric to have a clear upper bound, typically represented by the performance on ground truth (real images). However, as observed in Table 2 of MIGC [4] paper, CLIP scores for some methods exceed those of real images. This makes it difficult to interpret the meaning of the CLIP score values.
• In contrast, our IFS Rate can be interpreted as the generation success rate, offering a more intuitive understanding compared to CLIP score.

As a reference, we also report experimental results using local CLIP score in the table below. The results show that our method achieves competitive performance on this metric, closely approaching the scores obtained on the dataset.

L77-78 is a repetition.

We thank the reviewer for the careful review. The issue has been addressed.

References:

[1] Cheng, Jiaxin, et al. "Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation." The Thirty-eighth Annual Conference on Neural Information Processing Systems.

[2] Yuksekgonul, Mert, et al. "When and why vision-language models behave like bags-of-words, and what to do about it?." arXiv preprint arXiv:2210.01936 (2022).

[3] Wu, Yinwei, Xingyi Yang, and Xinchao Wang. "Relation Rectification in Diffusion Model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Zhou, Dewei, et al. "Migc: Multi-instance generation controller for text-to-image synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

We thank the reviewer for the extensive review and insightful questions. We are glad they found our paper clear and appreciated the effectiveness of our approaches and also value the recognition of our method's compatibility with community models. Below are our point-by-point responses to the questions.

The design module is supposed to interface with the text encoders, and both Appearance Tokens and Instance Semantic Map introduce the attention mechanism. Will it be computational costly during the inference process. There should be a detailed discussion.

We thank you for the suggestion. We have added a dedicated section (Appendix. F) to conduct experiments and discuss the computational cost of our method. The main results are summarized below. The findings indicate that, compared to the base model, IFA does not introduce significant computational overhead. We attribute this to the fact that IFAdapter applies control to only a small subset of cross-attention layers, making it relatively lightweight.

The size and shape of different objects seem unstable when applying IFA to different community models (Fig. 4), is it caused by the re-weight strategy from Instance Semantic Maps?

We thank the reviewer for the feedback. We believe this issue is not caused by the re-weighting strategy, as it does not influence the shape of objects. Instead, we attribute it to two factors:

• The BBox serves as a relatively flexible condition, primarily controlling object placement, which allows for the generation of objects with varying sizes and shapes. If specific object shapes are desired, they can be explicitly constrained in the instance description.
• Different community models are fine-tuned on various datasets, which may result in differing preferences for object shapes. A clear example is that LEGO-style community models tend to generate more geometrically regular instances, whereas clay-style models are inclined to produce thinner objects.

The L2I problem is not a novel task, and the main novelty mainly lies in the implementation detail of the layout incorporation strategy, which may not bring significantly inspirations to the community.

We fully agree with the reviewer's perspective. Layouts to images (L2I) generation based solely on category labels (e.g., a "dog") is not a new problem. Therefore, our IFG task represents a rethinking and enhancement of the L2I task. In our task, the focus is not only on ensuring the accurate generation of object categories and positions but also on guaranteeing the fidelity of each object's appearance (e.g., a "dog wearing a swimsuit"). This makes the task more challenging and practical. For instance, in generating indoor layout images, we often want to specify not just the position of furniture but also its shape, color, material, and more. Such scenarios fall outside the scope of typical L2I methods but are effectively addressed by our proposed IFA.

Additionally, beyond introducing a layout incorporation strategy, our method represents the initial attempt to adapt L2I techniques to SDXL. This provides valuable insights for the community on how to upgrade existing L2I models to SDXL. Moreover, our method's ability to empower the design of various community models offers a novel perspective that can serve as a reference for future developments.

Is the proposed model able to generate images with some intricate prompts. For example, a blue dog with red legs running on a colorful river, (with dog, legs, river assigned to different boxes). I want to see some limitations of the proposed method, or in other words, I want to know how IFA copes with the semantic issues which may inherit from the base model given out-of-domain prompts and instructions.

The primary role of IFA is to exert control over the generation of instance appearance and position; it is not designed to address out-of-domain semantic issues. As such, IFA does not enhance the model's ability to generate content for prompts that are outside the semantic understanding capabilities of the base model. In other words, the semantic comprehension of prompts remains limited to that of the base model, and any semantic issues inherent to the base model will not be resolved by introducing IFA.

For example, in the case of "a blue dog with red legs running on a colorful river," IFA can generate the image relatively accurately because the prompt does not completely deviate from the base model's generation space. However, for another typical out-of-distribution (OOD) case, such as "a horse rides the astronaut," where the horse and astronaut are assigned to different bounding boxes, IFA fails to produce a satisfactory image. If you are interested, we have showcased the image generation results for these two examples at OOD cases.

Thanks the authors for their detailed responses. Most of my concerns are addressed, and I decide to keep my original score.

We thank the reviewer for the extensive review and insightful questions. We are pleased to note the importance of the Instance Feature Generation task and appreciate the recognition of our benchmark, verification pipeline, and the effectiveness of our method in both qualitative and quantitative experiments. Below are our point-by-point responses to the questions.

Compared with instance diffusion and dense diffusion, this paper shows a small number of objects and does not show the situation where the object is denser.

We believe that the vast majority of scenes contain less than 10 instances, but our method is also able to generate images with more dense instances while maintaining consistency of detail. As shown in this link or Fig. 12 in the revised manuscript, the flowers strictly follows the color and position of the instruction, while other details such as a black hat, a goldfish in a pond, and a cat's sunglasses are generated accurately, although we do not assign additional bounding boxes to them. In another indoor case, we also used a larger number of instances, and again, the fine-grained details of the instances were well generated (e.g., cutted avocados and oranges, smoke from candles), and we were pleasantly surprised to find that our method was also well generated on some counterintuitive objects (e.g., plastic-like blue pears, green fluffy peaches, black apple, etc.)

The details of the method are not explained clearly. The details of the dataset construction and the baseline setting are not presented clearly. Ablation experiments lack a basic baseline presentation.

We have corrected several errors in the paper, including those in Equation 5 and its corresponding description. But as Reviewer V57v mentioned that our method is easy to follow, we are unsure which parts might require clarification. If you could point out specific sections where our explanation is unclear, we would be very grateful. Regarding the dataset construction process, we have further refined the description and updated it in Appendix A. For the baseline settings, we have included detailed explanations in Appendix B. Additionally, we have conducted ablation experiments on the baselines and provided the corresponding analysis, you can see Table 3 in the revised manuscript.

Authors should carefully check for errors in the text. For example, a sentence appears twice in succession in Introduction.

We thank the reviewer for their careful review. The issue has been addressed.

Why IFAdapter enables it to be seamlessly applied to various community models. For example, appearance queries are trained , how do the authors ensure that the feature has sufficient generalization capabilities.

The IFAdapter is a lightweight plugin designed to control the generation process of the base model. For instance, appearance queries are trained to extract semantic information that benefits high-frequency details in instance generation, which may exist in a specific subspace of the text embedding. Community models, on the other hand, are stylized versions of the base model obtained by fine-tuning a small number of parameters on different datasets. These models typically share the similar feature space as the base model. Consequently, the control learned by the IFAdapter on the base model can be effectively transferred to community models.

Why can appearance-related features be extracted from the bounding box through Fourier and MLP, especially when the model inference cannot obtain image input?

Appearance-related features are not extracted from the image; instead, they are derived from text embeddings through a resampler. These features capture semantics beneficial for generating high-frequency details, complementing the coarse instance semantics encoded in the EoT token. Since these features are used for appearance generation, we refer to them as appearance tokens. In Fig. 2, the green box labeled "BBox" represents a vector containing only [x1,y1,x2,y2]. The Resampler, on the other hand, serves as the primary structure for extracting appearance semantics from sentence features.

Dear Reviewer,

As the rebuttal is coming to a close, we would like to kindly provide a gentle reminder that we have posted a response to your comments. If you don't mind, may we please check if our responses have addressed your concerns and improved your evaluation of our paper? We are happy to provide further clarifications to address any other concerns that you may still have before the end of the rebuttal.

Sincerely,

Authors