Make It Count: Text-to-Image Generation with an Accurate Number of Objects

1.The comparison of baselines is unfair.

In Table 1, the authors compare CountGen with other baselines, but CountGen is based on the SDXL model, while some of these baselines are not. For example, the base model for Reason Out Your Layout [1] is SD1.4. And Counting Guidance [2], as a further improvement of Universal Guidance, uses SD1.5 or SD2. If you haven’t modified its source code (as mentioned in Line 797), it means the base models are different, making this comparison unfair. Therefore, migrating CountGen to other SD models for a fair comparison should be part of your work. Otherwise, this kind of unfair comparison is detrimental to the progress of the entire field. I think you should conduct experiments on other Diffusion Base Models (e.g, sd1.4, sd1.5, sd2) to achieve fair comparisons. It's worth noting that this is not an unnecessary increase in experimental workload.

2.No quantitative experiments in T2IBenchmark.

The authors use both human metrics and YOLO metrics. They apply these metrics in the CoCoCount benchmark, but what confuses me is why they do not use objective quantitative indicators (YOLO) in the T2ICompBench, and only report human evaluation scores in the paper. Given that CoCoCount is one of the contributions of this paper and has not yet been publicly released, the quantitative evaluation only on the self-built dataset is not convincing. I hope the authors can first explain why they only report human evaluation results in T2IBench? What is the reason for this? If possible, further provide the evaluation results of T2IBench.

3.The importance of the "Corrected Layout" is not sufficiently emphasized.

The core of CountGen lies in first providing the original layout and then correcting it. What is the advantage of CountGen over models like RPG[3]? CountGen first detects the incorrect layout and then corrects it, finally guiding the generation. Why not directly plan the layout from the beginning and then directly guide the generation using this layout? I guess the author's idea is similar to the work in OMG [4], where they first use the original capabilities of Diffusion to provide a layout and then modify it on this basis, possibly improving the quality. However, this paper does not provide sufficient quantitative analysis to compare the advantages and disadvantages of the "GPT-Based-Layout" and "Corrected Layout" methods, but only conducts ablation experiments on "Random Layout". I think this paper should further explain the necessity of using the "Corrected Layout" and provide qualitative or quantitative ablation experiments for comparison.

Additionally, the "ReLayout" module can only handle the case where the number of extra objects is "1" (this is due to the setup of the training dataset, see line 266). However, in Diffusion generation, there are often more than one extra objects, for example, generating 2 cats but the actual generated image has 4 cats. In such cases, ReLayout cannot be applied, so the universality of this method is not strong. The authors might try to construct more diverse datasets and add conditional quantity signals in ReLayout to output the corrected Layout.

4.The selection of UNet features lacks generality.

During the first stage of object segmentation, the authors use a specific layer in UNet and empirically believe that this layer works the best. Similarly, like weak 1, is this layer the best for sd, sd2, and sdxl? If not, then for each different Diffusion Model, the optimal layer would have to be manually and empirically selected again, increasing the difficulty of deploying this method. I think the authors should further research this part to enhance the generality of feature selection.

[1] Chen, Xiaohui, et al. "Reason out your layout: Evoking the layout master from large language models for text-to-image synthesis." arXiv preprint arXiv:2311.17126 (2023).

[2] Kang, Wonjun, Kevin Galim, and Hyung Il Koo. "Counting guidance for high fidelity text-to-image synthesis." arXiv preprint arXiv:2306.17567 (2023).

[3] Yang, Ling, et al. "Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms." Forty-first International Conference on Machine Learning. 2024.

[4] Kong, Zhe, et al. "OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models." arXiv e-prints (2024): arXiv-2403.

I think the paper introduces many exciting components that address the counting problem. However, I believe the paper suffer from a serious shortcoming which should be discussed/addressed in the next version of the paper:

Overfitting to one specific prompt format: In all visualizations and explanations of the paper, I notice that it consistently uses a prompt template structured as "A photo of [number of objects] [object name]." For example, "A photo of five apples" or "A photo of five backpacks." Why only use this prompt template in all ablations and qualitative visualization? What about exploring additional prompt templates, such as those provided in the T2I-CompBench dataset? I would suggest authors to discuss about following points during rebuttal or future submissions:

1. Explicitly state the limitations of their current approach in handling more complex prompts.
2. Provide results on a wider range of prompt templates from T2I-CompBench, even if performance is lower.
3. Discuss potential ways their method could be extended to handle multiple object types and attributes in future work.

More specifically, given following prompts:

Prompt 1: one apple and two oranges
Prompt 2: four balloons, two rabbits, three cars, four butterflies, and a dog created a festive atmosphere
Prompt 3: three printers, four people, a bicycle, three microwaves, and two refrigerators cluttered the office

Can authors:

1. show results of their instance localization, ReLayout, and layout-guided generation steps on 2-3 example prompts with multiple object types
2. discuss which components of their pipeline would need to be modified to handle these more complex prompts
3. quantify performance on a subset of T2I-CompBench prompts with multiple object types, if possible

Note: The above issue is concerning because I believe this solution cannot scale to other prompt templates, which require more object types, each associated with an attribute, such as "A photo of 6 green apples." If the authors believe their proposed method can effectively generate different objects from various categories within a scene, I would ask them to provide this information during the rebuttal. Otherwise, I believe this paper has limited contributions/applications.

1. CountGen relies on the specific architecture of the SDXL model, particularly the self-attention features extracted from layer $L_{52}^{up}$. This dependence raises concerns about the method's transferability to other diffusion models. Further investigation is needed to assess whether the concept of objectness, as captured by CountGen, can be generalized to different architectures or whether model-specific adaptations would be required. Since the PCA visualization in Fig.3 should be easily reproduced using other models, it could be verified in time.

2. The ReLayout module is trained on a dataset of layout pairs generated by SDXL using the same random seed but slightly different object counts. This approach, while clever in its utilization of consistent layout generation by SDXL, could potentially lead to overfitting. ReLayout might learn to exploit specific artifacts or biases in SDXL's output, limiting its effectiveness when applied to layouts generated by other models or even by SDXL with different seeds. For example, how to make sure that SDXL is able to always generate correctly with "two apples" and "three apples" with the similar layout. Over here, I'm not sure if any VLM model (like visual question answering) could help.

3. While the paper briefly mentions the time taken by CountGen to generate an image (~36 seconds), it lacks a detailed discussion of the computational cost compared to other methods, especially in relation to the achieved accuracy gains.

• The analysis in the introduction is unclear. Speficically, the work identifies the cause of inability to generate a specified number of objects to be the lack of instance-level features (L61-65). However, the work later discovers the presence of such features (L199-210), which contradicts the claim. The fact that previous works have not found such features is not an explanation on why current diffusion models do not perform well in counting tasks.
• Lack of comparisons with existing baselines. The ReLayout proposed in this work is very similar to the closed-loop generation proposed in Self-correcting Diffusion Models [1], which aims at solving problems in a larger scope that includes count-specific generation. The authors are recommended to discuss the relevance and compare with [1] in terms of performance.
• The proposed method requires finding features that are related to instanceness for each model. Furthermore, the authors did not provide explanations on the process of finding such features, which are the core of the method.
• The authors only experimented with SDXL, which does not show generalization capabilities of the proposed method.
• The proposed method introduces an additional module to generate the layouts. This might increase the run-time, impacting downstream use cases. The authors are recommended to show the wall-clock runtime and compare with previous baselines.

[1] Self-correcting LLM-controlled Diffusion Models. T. Wu, et al. CVPR 2024. https://arxiv.org/abs/2311.16090.