Kitten: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities

W1. Q2. Empirical evidence supporting the selection of domains and entities is representative of real-world scenarios.

Our domain and entity selection is comprehensive and diverse. We note that datasets such as OntoNotes and WordNet do not provide images of entities, making them unsuitable for image generation evaluation. We sample them from OVEN [1], the most widely covering open-domain image recognition dataset (which is compiled from 14 existing image classification datasets). This results in the largest number of visual entities among all existing datasets. By sampling from OVEN [1], our dataset includes 322 entities from 8 domains and 6,440 prompts, offering broader coverage than alternative evaluation benchmarks [2-7] to reduce data bias and improve the generalizability of the results. Existing benchmarks on image customization [2-7] are not focused on generating knowledge entities and often test on a more limited number of subjects, with only a few hundred evaluation prompts. For example, DreamBench [2], DreamBench-v2 [3], CustomConcept101 [4], DreamBench++ [5], MM-Diff [6], and StoryMaker [7] evaluate 30, 30, 101, 150, 25, and 40 subjects, and 750, 220, 2020, 1350, 500 and 800 prompts, respectively. To the best of our knowledge, no current evaluation benchmarks for visual entity generation cover more entities and prompts than ours. We emphasize this work is an initial step in constructing a large dataset to evaluate model performance in the novel problem of visual entity generation, and we aim to expand its scope in future work.

Additionally, existing vision-and-language benchmarks [8-14] on different problems often include far fewer evaluation prompts. For example, Winoground [8], WHOOPS! [9], LLaVA-Bench [10], Visit-Bench [11], ConTextual [12], VibeEval [13], and Visual Riddles [14] include 400, 90, 500, 576, 500, 269, and 400 prompts, respectively. Our benchmark has provided 6,440 prompts—more than ten times the number found in these benchmarks—ensuring comprehensive coverage of real-world scenarios to accurately reflect the models' understanding of knowledge entities.

---

W2. Justification for the cost of human evaluation.

We emphasize that KITTEN introduces a novel evaluation problem focused on generating visual entities, measuring both fidelity to entities and instruction-following capability. Since there are no established metrics for this task, human evaluation is a critical first step toward gaining a reliable understanding of different model performances. In addition, we explore two widely-used automatic metrics from prior work, and analyze their alignment with human evaluation. This comparison provides valuable insights into the strengths and limitations of automated evaluations.

Our evaluation framework and dataset represent an important contribution to the field, encouraging future research on developing better automatic evaluation methods and improving image-generation models. To support this, we provide detailed descriptions of our human annotation interface and a comprehensive analysis of a variety of models to assist future research efforts. Lastly, we note that our 6k annotations are not exceptionally large compared to other datasets with extensive human annotations. For example, Sherlock [15] and GenAI-Bench [16] include 363K and 40K human annotations, respectively.

---

W3. Q1. Theoretical basis and rationale for choosing the four types of prompts.

While our work focuses on a novel problem, it is not an isolated effort. Instead, it builds upon prior research in customized image generation [2-7], which emphasizes generating specific entities based on a few reference images. Accordingly, the design of our four types of prompts aligns with common practices from prior works in customized image generation [2-7] and is tailored to address the unique challenges of our problem.

For instance, DreamBench [2] employs prompts that (1) generate a subject in various contexts, (2) create artistic renditions of the subject, (3) modify the subject’s properties (e.g., color and material), and (4) combine the subject with accessories. Similarly, CustomConcept101 [4] evaluates prompts that (1) alter background scenes, (2) insert additional objects into a scene, and (3) generate stylistic variations of the main subject. Our prompt design comprehensively incorporates these key scenarios, ensuring that we evaluate aspects of image generation that are both significant to researchers and representative of real-world applications.

We generate prompts by instructing ChatGPT to propose four types of prompts for each domain. These prompts are then manually refined and adjusted by the authors to better align with the characteristics of each domain. To mitigate potential biases, we further extend our prompt sets to include those used in prior benchmarks [2,4].

Q3. Trade-offs between manual and automated evaluation methods.

While manual evaluation ensures high accuracy, automated methods provide a cost-effective alternative, albeit with slightly lower alignment to human perception. In Section 5.2 of the paper, we present a detailed analysis of two popular automatic metrics. Appendix A3 includes the detailed scores. Our analysis in Section 5.2 shows that automatic metrics generally align with human evaluations, though notable discrepancies exist. These insights represent a valuable contribution to the field, fostering future research into refining automatic evaluation metrics.

---

Q4. Strategies for expanding the evaluation dataset and incorporating additional types of prompts.

Since our prompt generation pipeline is semi-automatic and leverages large language models, future benchmarks can easily extend to include other types of prompts with minimal cost while maintaining quality. For instance, the dataset can be expanded by instructing ChatGPT to propose prompts such as (1) generating entities with specific poses, viewpoints, and expressions and (2) generating multiple specific entities. On the other hand, the domains and entities can be expanded by sampling additional classes from the OVEN [1] dataset, as well as from Wikimedia data.

---

Q4. The prompt designs are aligned with the objectives of our study and cover key scenarios involving knowledge entities.

To align the prompts with our objective of evaluating knowledge-entity generation, we modified the curated prompts by incorporating the entity names directly into the text. This ensures that all prompts explicitly involve the targeted knowledge entities. For example, the template prompt "A peacock in front of the [landmark]" is transformed into specific examples like "A peacock in front of the [Bandinelli Palace landmark]." This process resulted in a dataset comprising 322 entities and 6,440 prompts, significantly expanding the dataset's coverage compared to existing benchmarks. For reference, DreamBench [2], DreamBench-v2 [3], CustomConcept101 [4], DreamBench++ [5], MM-Diff [6], and StoryMaker [7] evaluate 30, 30, 101, 150, 25, and 40 subjects, and 750, 220, 2020, 1350, 500 and 800 prompts, respectively.

---

W4. Q5. Comparison with existing benchmarks in terms of design goals and our distinct advantages.

We outline the comparison of KITTEN to existing benchmarks in Section 2 of the paper. Compared to HELM, GenAI-Bench, and ConceptMix, our benchmark evaluates a distinct task. HELM focuses on evaluating language models, while KITTEN is designed for evaluating image generation models. GenAI-Bench and ConceptMix primarily evaluate image generation models on compositional text prompts (e.g., objects with specific colors, shapes, or spatial relationships), such as "A red novel is placed next to a spoon." In contrast, KITTEN focuses on assessing image generation models' ability to generate fine-grained details of specific visual entities by incorporating entity names into the prompts, such as "A peacock in front of Bandinelli Palace." Our benchmark evaluates a novel problem of generating fine-grained visual entities, which is orthogonal to prior benchmarks on image generation. We will include the discussion in the paper to clarify these distinctions.

References

[1] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of Wikipedia entities. In ICCV, 2023.

[2] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.

[3] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, William W. Cohen. Subject-driven Text-to-Image Generation via Apprenticeship Learning. In NeurIPS, 2023.

[4] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In CVPR, 2023.

[5] Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, Shu-Tao Xia. DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation. arXiv preprint arXiv:2406.16855, 2024.

[6] Zhichao Wei, Qingkun Su, Long Qin, Weizhi Wang. MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration. arXiv preprint arXiv:2403.15059, 2024.

[7] Zhengguang Zhou, Jing Li, Huaxia Li ,Nemo Chen, Xu Tang. StoryMaker: Towards consistent characters in text-to-image generation. arXiv preprint arXiv:2409.12576, 2024.

[8] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, Candace Ross. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. In CVPR, 2022.

[9] Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, Roy Schwartz. Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images. In ICCV, 2023.

[10] Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee. Visual Instruction Tuning. In NeurIPS, 2023

[11] Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, Ludwig Schmidt. VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use. In NeurIPS, 2023.

[12] Rohan Wadhawan, Hritik Bansal, Kai-Wei Chang, Nanyun Peng. ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models. In ICML, 2024.

[13] Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, Ethan Yeo, Eugenie Lamprecht, Qi Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng, Cyprien de Masson d'Autume, Dani Yogatama, Mikel Artetxe, Yi Tay. Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models. arXiv preprint arXiv:2405.02287, 2024.

[14] Nitzan Bitton-Guetta, Aviv Slobodkin, Aviya Maimon, Eliya Habba, Royi Rassin, Yonatan Bitton, Idan Szpektor, Amir Globerson, Yuval Elovici. Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models. arXiv preprint arXiv:2407.19474, 2024.

[15] Jack Hessel, Jena D. Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, Yejin Choi. The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning. In ECCV, 2022.

[16] Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, Deva Ramanan. GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation. In CVPR Workshop, 2024.

We further provide the detailed human evaluation scores across prompts along with the analysis below.

Instruction-following score.

The Instruction-Following scores across prompts align with the average performance, with Imagen-3 and Instruct-Imagen-3 emerging as the top performers overall. In particular, Instruct-Imagen-3, Flux, and Imagen-3 excel in "Entity in Context," achieving scores of 90, 90, and 87.8, respectively, while Imagen and other models fall below 80. Similarly, Instruct-Imagen-3 and Imagen-3 demonstrate strong performance in "Composition of Entity," with scores of 92.9 and 94.6. These results highlight the robust instruction-following capabilities of Imagen-3 and its retrieval-augmented variant, Instruct-Imagen-3, in these aspects. In contrast, DreamBooth leads in "Styles of Entity" with the highest score of 90.2, followed by Custom-Diffusion, Instruct-Imagen-3, and Stable-Diffusion, which score 84.3, 84.3, and 80.4, respectively. This demonstrates the strong ability of Stable Diffusion and its retrieval-augmented variants to generate accurate styles. For the "Materials of Entity" category, all models struggle to follow instructions effectively, though Imagen-3 achieves a relatively higher score. These observations align with the Image-Text Alignment metric, underscoring the consistency of the models across evaluation criteria.

---

Faithfulness to Entity score.

The ranking across prompts is consistent with the overall ranking. Instruct-Imagen achieves the highest score, followed by Instruct-Imagen-3 and Imagen-3. The retrieval-based methods, DreamBooth and Custom-Diffusion, follow next, while the other base models—Imagen, Flux, Stable-Diffusion—score comparatively lower.

W1. Novelty and contribution.

Our paper is the first systematic study of a comprehensive set of image-generation models on both entity fidelity and instruction-following. To the best of our knowledge, this evaluation framework is a unique contribution to the field, offering insights that will promote advancements in models that balance knowledge fidelity with instruction-following. Additionally, we provide detailed designs for our human evaluation process to support future research, along with a thorough analysis of current automated metrics to guide the development of more accurate, human-aligned evaluation methods in the future.

Our novel evaluation framework provides insights on model improvement: in Section 5.1 of the paper, these findings facilitate future modeling work to balance instruction-following and entity fidelity: 1) we find that strong backbone models (Imagen-3) can have higher faithfulness than weaker backbone models with retrieval (DreamBooth). 2) DreamBooth is able to improve entity fidelity compared to its baseline model, SD (2.51→3.08), without compromising SD’s instruction-following score (72.2→73.8). However, this is not the case for CustomDiffusion. Our evaluation shows that DreamBooth is a superior retrieval-augmented method compared to CustomDiffusion in terms of balancing fidelity and instruction-following.

---

W1. Exploring the achievability of coexistence and future research directions.

We explore the combination of a more powerful backbone model, Imagen-3, with the advanced retrieval-augmented method Instruct-Imagen. Preliminary results demonstrate that balancing entity fidelity and creative flexibility is achievable. The new model achieves an instruction-following score of 83.21 and an entity fidelity score of 3.48, surpassing Imagen-3's score of 3.17. Additional examples are included in the supplementary materials.

These findings also highlight future research directions, showing that enhancing the backbone model can improve both instruction-following capability and entity fidelity. Furthermore, combining a strong backbone with a retrieval-augmented method can achieve a coexistence of these two aspects.

---

W2. Performance variations across entity domains.

We show that the performance of each method is domain-dependent in Figure 5 of the paper. We also provide detailed statistics in Appendix A2. In Section 5.2, we provide an analysis of the image-entity alignment scores across domains. For example, retrieval-augmented models generally achieve higher image-entity alignment scores than backbone models in the insect, landmark, and plant domains. This is likely because these domains contain less frequent terms in common image datasets and are therefore underrepresented in the backbone model’s parameters, while retrieval-augmented models improve performance by incorporating reference images during inference.

Additionally, Table 3 shows that the insect and landmark domains have lower average faithfulness to entity scores of 2.12 and 2.61, likely due to the inherent challenges of generating fine-grained details of insects and the many specifications and features of specific landmarks. In contrast, the car category achieves the highest average score of 3.94, possibly due to its higher representation in existing image datasets.

W1. Concrete examples of the balance between entity fidelity and instruction-following.

We have provided examples illustrating a balance between entity fidelity and creative flexibility in the generated images. Specifically, Instruct-Imagen's results for prompts such as "A Wakame dish with a cherry flower on top of it" in Figure 9, "Alstroemeria on top of a mountain with sunrise in the background" in Figure 10, and "Satyrium liparops sitting at the beach with a view of the sea" in Figure 11 demonstrate enhanced entity fidelity while maintaining creative flexibility. These examples are supported by the highest alignment scores across both metrics when compared to other models.

In Section 5.1 and Figure 4, we show that the retrieval model DreamBooth improves entity faithfulness compared to its baseline model, SD (2.51→3.08), without compromising SD’s instruction-following score (72.2→73.8). This is another concrete example that indicates it is possible to achieve good entity fidelity without sacrificing creativity.

---

W2. Exploring the achievability of coexistence and future research directions.

We explore the combination of a more powerful backbone model, Imagen-3, with the advanced retrieval-augmented method Instruct-Imagen. Preliminary results demonstrate that balancing entity fidelity and creative flexibility is achievable. The new model achieves an instruction-following score of 83.21 and an entity fidelity score of 3.48, surpassing Imagen-3's score of 3.17. Additional examples are included in the supplementary materials.

These findings also highlight future research directions, showing that enhancing the backbone model can improve both instruction-following capability and entity fidelity. Furthermore, combining a strong backbone with a retrieval-augmented method can achieve a coexistence of these two aspects.

Dear Reviewer

Thank you for your comments on our paper. We have provided a response and a supplementary PDF on OpenReview. Please let us know if you have additional concerns or comments, many thanks!

Best, Authors

W1. Details on the creation of the benchmark.

Our domain and entity selection is comprehensive. We sample them from OVEN [1], the most widely covering open-domain image recognition dataset (which is compiled from 14 existing image classification datasets). This results in the largest number of visual entities among all existing datasets. By sampling from OVEN [1], our dataset includes 322 entities from 8 domains and 6,440 prompts, offering broader coverage than alternative evaluation benchmarks [2-7] to reduce data bias and improve the generalizability of the results. Existing benchmarks on image customization [2-7] are not focused on generating knowledge entities and often test on a more limited number of subjects, with only a few hundred evaluation prompts. For example, DreamBench [2], DreamBench-v2 [3], CustomConcept101 [4], DreamBench++ [5], MM-Diff [6], and StoryMaker [7] evaluate 30, 30, 101, 150, 25, and 40 subjects, and 750, 220, 2020, 1350, 500 and 800 prompts, respectively. To the best of our knowledge, no current evaluation benchmarks for visual entity generation cover more entities and prompts than ours. This is just an initial step in constructing a large dataset to evaluate model performance in the novel problem of visual entity generation, and we aim to expand its scope in future work.

We generate prompts by instructing ChatGPT to propose four types of prompts for each domain, categorized as follows: (1) basic descriptions, (2) entities in specified locations, (3) compositions of entities and other objects, and (4) entities in specific styles or materials. The prompts are then manually refined and adjusted by the authors to better align with the characteristics of each domain. To align the prompts with our objective of evaluating knowledge-entity generation, we modified the curated prompts by incorporating the entity names sampled from each domain directly into the text. This ensures that all prompts explicitly involve the targeted knowledge entities. For example, the template prompt "A peacock in front of the [landmark]" is transformed into specific examples like "A peacock in front of the [Bandinelli Palace landmark]." This process resulted in a dataset comprising 322 entities and 6,440 prompts, significantly expanding the dataset's coverage compared to existing benchmarks.

The prompts range from 4 to 24 words, with an average length of 9.91 words and a standard deviation of 2.86. We will include this detailed information in the final version of the paper.

---

W2. Details on human evaluation.

We thank the reviewer for their valuable feedback regarding our human evaluation. As detailed in Section 3.3 and Figure 3, with full instructions in Appendix A6, we employ five annotators per image to ensure robust assessments. The raters are hired through Prolific.com, a third-party rating service. For the binary task ("Adherence to Prompt Beyond References"), we observe agreement among at least 4 out of 5 annotators in 75% of cases and perfect agreement (5 out of 5) in 44% of cases. For the Likert scale task ("Faithfulness to Reference Entity"), we calculate Krippendorff’s Alpha at 0.60, indicating a good agreement for a subjective task of this complexity. Additionally, we achieve an IoU-like score of 0.53, which penalizes outliers and demonstrates moderate consensus, and an average pairwise Cohen’s Kappa of 0.25, reflecting fair pairwise agreement. The average standard deviation of ratings is 0.74, reflecting moderate variability in annotator judgments. Please let us know if further metrics or details are needed. These metrics collectively demonstrate the reliability of our human evaluation, and we will include these details, along with additional examples and explanations, in the camera-ready version.

---

Q1. Version of Flux.

We use Flux.1-dev and will clarify this in the final version of the paper.

References

[1] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of Wikipedia entities. In ICCV, 2023.

[2] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.

[3] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, William W. Cohen. Subject-driven Text-to-Image Generation via Apprenticeship Learning. In NeurIPS, 2023.

[4] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In CVPR, 2023.

[5] Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, Shu-Tao Xia. DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation. arXiv preprint arXiv:2406.16855, 2024.

[6] Zhichao Wei, Qingkun Su, Long Qin, Weizhi Wang. MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration. arXiv preprint arXiv:2403.15059, 2024.

[7] Zhengguang Zhou, Jing Li, Huaxia Li ,Nemo Chen, Xu Tang. StoryMaker: Towards consistent characters in text-to-image generation. arXiv preprint arXiv:2409.12576, 2024.