Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation

W5. The relationship between the claimed desired question properties and the final VQA-Human correlation is not well demonstrated. In other words, the relationship between Precision/Recall/Atomicity/Uniqueness in Table 2 and Spearman/Kendall correlation in Table 3 is unclear.

For context, we would like to clarify that the Precision/Recall/Atomicity/Uniqueness validation (Table 2) is intended to check the quality of the generated questions on the extent to which they match human judgment, i.e., “are these the right validation questions to ask to check T2I alignment?”, and importantly, achieving strong human correlation in the answers while maintaining the high quality in question generation.

A natural doubt that we expect may arise is: without having the atomicity, semantic coverage, etc. quality we use to evaluate generated questions, TIFA and VQ^2 can still achieve decent human correlation, then what’s the point in having these quality metrics?

We believe the answer lies with interpretability and reliable diagnostics, especially on the instance/question-level — for any individual question, we strive to achieve an unambiguous understanding of human judgments. For example, a response to a non-atomic question “Is there a blue bike parked next to a door?” would be hard to interpret — if the answer to this question is ‘no’, then is it that the ‘bike’ is not generated properly? or is it that the ‘location of the bike’ is wrong?

Therefore, to summarize:

• With the Precision/Recall/Atomicity/Uniqueness validation, we show that DSG produces high-quality questions per human intuition. This ensures interpretability and reliable diagnostics;
• With the human correlation experiments, we show that DSG results at the same time correlate well (and better than previous work) with human judgments.

W6. There is no comparison with TIFA and VQ^2A in terms of precision and recall in Table 2.

We agree it is possible to evaluate TIFA and VQ^2 by eliciting human judgments on precision and recall, however in practice, there exists a failure mode that renders the evaluation not implementable (indicated by their low atomicity in Table 2) — the issue is rooted in how TIFA/VQ^2 and DSG are designed – we clarify as follows:

Note that, due to the absence of the concept of atomicity, we frequently observe TIFA and VQ^2 questions that cover two or more semantic details (e.g. “is there a blue bike next to the door?” covers the entity “bike”, the color of it, and its relative position to another entity “door”). As we argue in the Sec. 1 introduction (the A-B-C-D bullet points on reliability), “loaded questions” like these do not lend themselves well to acquiring unambiguous answers (either by a VQA model or a human rater).

If we disregard the reliability concern to look at precision/recall alone, loaded questions can score high precision/recall, by simply adding a question mark to the original prompt to form a question (e.g., “A blue motorcycle parked by paint chipped doors.” $\rightarrow$ “A blue motorcycle parked by paint chipped doors?”). This would render the precision/recall validation uninformative.

Our DSG, on the other hand, uses semantic tuples as a structured intermediary semantic representation to handle atomicity – this wards it off the “loaded question” failure mode mentioned above in the precision/recall validation (e.g., if a “loaded question” that maps to more than 1 semantic tuple is produced, it is considered a wrong question).

Q1. It is stated on page 4 that DSG only involves binary yes/no questions. Why are there multi-choice questions in the human annotation UI in Figure 10?

It is true that DSG only involves binary yes/no questions. The questions inside Figure 10 are generated by TIFA, where multiple-choice questions are permitted. We use the same human annotation UI for questions generated by DSG and VQ2A, where we provide yes/no answer choices.

We thank reviewer p5Xv for the valuable comments on the design and evaluation of DSG / DSG-1k. We address/clarify all the points and report the revision we applied to our submission.

W1. It is not well-explained which part of DSG is designed to address the Atomic Question and Full Semantic Coverage properties. Why the QG methods of TIFA and VQ^A cannot achieve these two properties?

Thanks for the great suggestion — we made a revision to explain the point more clearly (Sec. 3, the paragraph above the subsection on DSG-1k dataset).

In gist, to the question “why the QG methods of TIFA and VQ^2 cannot achieve atomicity and semantic coverage?”, the short answer is: because they do not have a structured intermediary representation in the question generation process (like DSG’s semantic tuples) to a) define the semantic coverage for a prompt, and b) thereby allowing for the verification of atomicity.

To expand the answer in detail, DSG uses semantic tuples (e.g., attribute(bike, blue), entity(bike), relation(bike, wall, next to)) as an intermediary representation in question generation: prompt -> semantic tuples -> questions. As such, leveraging the structuredness of the semantic tuple intermediary representation, we can then clearly define "atomicity" and "semantic coverage" and thereby modulate DSG's behavior to generate questions accordingly:

• Atomicity: an atomic question corresponds to only a single semantic tuple. For example, "is there a bike?" maps to one semantic tuple "entity(bike)", and is atomic; "is there a blue bike?" maps to two semantic tuples "entity(bike)" and "attribute(bike, blue)", and is thus NOT atomic.
• Semantic coverage: the full semantics of a prompt is represented by a set of unique semantic tuples (no duplicates), such that no entities, attributes, relations, or global descriptions can be added.

TIFA and VQ^2, on the other hand, generate questions directly, thus there is no means available to encourage atomicity and semantic coverage (i.e., with a direct prompt-to-questions configuration w/o semantic tuples intermediary). In our human evaluation of TIFA and VQ^2 questions (Table 2), we indeed find large numbers of non-atomic questions and violations of semantic coverage (with duplication and missing entities/attributes/relations).

W2. Some categories are under-represented in the DSG-1k dataset. According to Table 9, six categories have less than 30 examples and VQA-Human correlation is not computed for these categories.

We apologize for the confusion introduced here. Please allow us to clarify (also added clarification in the caption the table referred to – Table 13 (Table 9 in the original PDF)):

Specifically, Table 13 of the revised PDF (Table 9 in the original PDF) examines questions generated from TIFA160 prompts (160 in total), where we used DSG-1k’s ‘question types’ to present specific scores, rather than from the whole DSG-1k prompts. TIFA160 is a relatively small sample, so they do not reflect some types of questions such as text rendering. In Table 14 of the revised PDF (Table 10 in the original PDF), we illustrated three T2I models’ VQA accuracy in the full range of DSG-1k’s semantic categories.

W3. There is no ablation study showing which design choice in DSG contributes to each of the four properties.

As a similar point has also been raised by others, to avoid duplication, we reply in the common response. Please refer to the common response. Thanks.

W4. It is unclear whether DSG correlates better with humans, compared with TIFA and VQ^A, in each fine-grained category.

We agree that a by-category human correlation study is a valuable addition. Following the suggestion, we conducted an additional experiment using TIFA160 prompts, by calculating correlation Likert 1-5 scores with each of their 4 data splits:

• COCO: general purpose;
• PaintSkills: focusing on spatial relations;
• PartiPrompts: focusing on attributional failure cases (e.g., color, counting, etc.);
• DrawBench: focusing on simple compositionality.

As shown in the following table, DSG produces higher human correlation scores on 3 out of 4 splits, as well as the best overall performance. Even in the only data split it does not come on top (i.e., DrawBench), the DSG’s correlation is very close to the top-performing TIFA’s. We incorporated the new experiment result into Appendix D.2 in our revised PDF, thanks for the suggestion!

(continued below)

Thanks for your additional comments!

I'm interested in understanding whether the better performance of DSG over TIFA stems from (1) the utilization of semantic tuples as a more effective representation than individual elements, (2) the incorporation of better in-context examples, or any other factors?

We would like to point out that the most major difference between DSG and TIFA is that DSG has a dependency structure across each entity/attribute/relation, while the extracted elements in TIFA does not.

Consider the caption "A blue motorcycle". There are two questions, "is there a motorcycle?" and "is the motorcycle blue?". If there is no motorcycle in the image, then it is just unreasonable to ask "is the motorcycle blue?". The design of DSG makes sure that, if the entity "motorcycle" is not there, then the attribute "blue" is considered missing. In TIFA, there is no such guarantee, and we observe that the VQA models sometimes just hallucinate answers even if the question is not reasonable, which leads to bias in evaluation.

As such, the dependency structure of DSG is critical, and it is where DSG is making a difference compared with TIFA and VQ2. Besides, the improvement on precision, recall, atomicity, uniqueness are side benefits we observed, and the points you listed are possible explanations for these side benefits.

We hope our response has addressed your concerns. As it is the end of the discussion period, we do not have time to write more. It would be great if you increase the rating. Thanks!

We thank reviewer A5bo for the feedback and comments.

W1. The VQA model is not good enough and hinders the final alignment to humans in T2I evaluation.

As a similar point has also been raised by others, to avoid duplication, we reply in the common response. Please refer to the common response. Thanks.

Q1. Can a better VQA model lead to better alignment in Tables 3, 4, and 6?

Yes. This can be found in the results we already report in the paper. Please allow us to clarify:

Empirically, based on the result in Table 5 (per-question VQA-Human matching accuracy), we see a common trend in Tables 3, 4, and 6 — if a VQA model’s result correlates better with human Likert scores, the model also tends to have a higher matching accuracy on the per-question level.

Q1. In 4.1, to validate the question dependencies, authors evaluate manually on 30 samples and automatically on the full TIFA160. However, the consistency of manual and automatic evaluation is not presented.

Following the suggestion, we additionally report the the consistency of 30 samples of manual v.s automatic evaluation — dependency valid ratio of DSG is 100% in manual evaluation (30 samples), and 99% in automatic evaluation (full TIFA160). In the revised PDF, we added a sentence to present the result more clearly.

Q2. The comparison of runtime among different methods should be added.

We compare the runtime of TIFA and our DSG. For apple-to-apple comparison, we use the same LLM (gpt-3.5-turbo-16k) and VQA model (mPLUG-large). On a single A6000 GPU, we measure the time for obtaining the final T2I alignment score for 20 image-text pairs. We use 20 prompts sampled from TIFA-160 prompts and images generated by Stable Diffusion v2.1.

As in the following result, we find DSG takes a shorter time than TIFA, primarily because TIFA tends to generate more questions than DSG, increasing the inference time of VQA models.

• TIFA: 481s for 20 image-text pairs (24s per image-text pair)
• DSG: 332s for 20 image-text pairs (16s per image-text pair)

Please note that the cost for many image-text pairs could be saved in many ways: e.g., batch processing (separate QG and QA stages, processing multiple images), caching (re-using image features when a VQA model answers to questions paired with the same image), using smaller models (through quantization, distillation, compression), etc.

Q3. In Table 3, for Instruct-BLIP, the Spearman’s ρ of DSG is lower than that of TIFA, authors should explain this phenomenon briefly.

While Spearman's rho shows "Instruct-BLIP & TIFA" > "Instruct-BLIP & DSG", Kendall's Tau shows "Instruct-BLIP & TIFA" < "Instruct-BLIP & DSG". There are some cases with tied ranks, where Kendall’s Tau correlation is more reliable than Spearman’s rho correlation when handling data with tied ranks [1,2].

[1] Taylor, Wilson L. “Correcting the Average Rank Correlation Coefficient for Ties in Rankings.” Journal of the American Statistical Association, vol. 59, no. 307, 1964, pp. 872–76. JSTOR, https://doi.org/10.2307/2283105. Accessed 19 Nov. 2023.

[2] Marie-Therese Puth, Markus Neuhäuser, Graeme D. Ruxton, “Effective use of Spearman's and Kendall's correlation coefficients for association between two measured traits”, Animal Behaviour, Volume 102, 2015, Pages 77-84, ISSN 0003-3472, https://doi.org/10.1016/j.anbehav.2015.01.010.

We thank reviewer tDEJ for the insightful and detailed comments. We address/clarify all the weaknesses they bring to our attention and report the revision we applied to our submission.

W1. This paper improves the existing QG/A methods with Davidsonian Scene graph, which is generated based on LLMs. The approach of the work could be enriched with more details and techniques.

As a similar point has also been raised by others, to avoid duplication, we reply in the common response. Please refer to the common response. Thanks.

W2. Ablation on separate steps of DSG should be presented. For example, without establishing dependencies, measure the changes of consistency between VQA score and the human 1-5 Likert Scores.

As a similar point has also been raised by others, to avoid duplication, we reply in the common response. Please refer to the common response. Thanks.

W3. Apart from TIFA and VQ2A, more methods could be compared, including CLIPScore and caption based approaches.

Please note that we mentioned “For human correlation, Hu et al. (2023) for example achieves 47.2 Kendall’s τ in a sizable correlation analysis with human 1-5 Likert judgments, with CLIPScore τ = 23.1.’’, For completeness, we also calculated and included correlation results on the 5 single-summary metrics considered in Hu et al. (2023), i.e., BLEU-4, ROUGE-L, METEOR, SPICE, (with BLIP-2 Captioning) and CLIPScore (with CLIP ViT-B/32) in Appendix E in the revised PDF.

(NOTE: To ensure consistency, we did not directly report the result from Hu et al. (2023) Table 2. Our numbers are calculated with the same human 1-5 Likerts we collected – the same Likerts are used in calculating the correlation results in our Table 3. Further, also note that Hu et al. (2023) calculated human Likerts based (by rules, see their Appendix C.1) on raters’ answers to the GPT-3 generated questions, whereas we collected them independently with a single 1-5 consistency question, please see our Figure 12.)

Further, we set focus on fine-grained QG/A as an overall more informative / diagnostic framework than single-summary metrics (represented by CLIPScore), and intend to show that, not only does QG/A work well on the coarse-grained model-level in evaluating performance (as TIFA/VQ^2 show in their comparison to a range of single-summary methods), we can improve it further (with DSG) to provide reliable fine-grained diagnostics.

(continued below)

tDEJ W2 & p5Xv W3 - DSG ablation

Both reviewers suggested ablation on our DSG method from different angles. We highly agree with the value in the additional experiments in this regard.

Reviewer tDEJ suggested that we look into the influence of DSG dependency annotations on the performance of VQA-human consistency. Per the suggestion, we conducted additional experiments and reported our findings in the revision (Appendix D.1 of the revised PDF).

We ablate the use of DSG dependencies in three conditions:

a) DSG: Treating child questions that depend on parent questions with NO answers as NO. E.g. the child “is the bike blue?” depends on the parent “is there a bike?”, if the answer to the parent question is NO, then we assign NO as the answer to the child question. b) DSG (drop): Skipping the child questions that depend on parent questions with NO answers. E.g. With the running example above, if the answer to the parent question is NO, we do not factor the child question into the subsequent metric accuracy calculation. c) DSG (w/o dependencies): Ignoring all dependencies and treating all the questions equally.

We then examined the effect in:

• VQA-Human correlation. The Spearman & Kendall between the per-item VQA accuracy and human 1-5 Likert judgment.
• VQA-Human matching accuracy. Per-question matching accuracy between VQA and human judgments. This is calculated as “number of questions where VQA answer and human answer agrees / number of total questions”

For VQA-Human 1-5 Likert score correlation, we observe that the dependencies have little impact:

However, on VQA-Human matching accuracy, using dependencies leads to a substantial improvement.

Our findings can be summarized as:

• While scores correlation to human 1-5 Likert has been the de-facto standard in (meta-)evaluation of QG/QA evaluation methods, and it is indicative of the rough tendency of the quality of the scores, we argue that (i) it is not fine-grained enough to tell apart the subtle differences between techniques, and (ii) it assumes that the human 1-5 Likert scores are the absolute source of truth, but it is likely that giving a holistic 1-5 classification of the alignment between a (potentially complex) prompt and an image is not a very well calibrated task.
• We believe the per-question VQA-Human matching accuracy is much more effective on the fine-grained level for understanding exactly where VQA-Human match/mismatches reside, and is thereby more informative for model development as a diagnostic tool. DSG dependencies are empirically very valuable in improving VQA-Human matching accuracy.

Beyond quantitative analysis, we firmly believe that taking into account the VQA model answer to “Is the car blue?” when there is no car in the image cannot bring any benefit to the evaluation results and thus we argue that those questions should not be asked.

Reviewer p5Xv suggested an ablation on DSG’s performance on the four properties we defined as the reliability criteria for a QG/A method — atomicity, uniqueness, dependency among questions, and full semantic coverage. We venture to interpret the question as “why aren’t there a comparison study on different configuration combinations of DSG such as (+atomicity), (+atomicity, +uniqueness), (+atomicity, +uniqueness, +dependency) and so on, to ablatively probe each property.

We agree that this would seem to serve as an informative dissection on the value of DSG in detail, however, a practical difficulty arises as a result. For example, say we have a condition where only atomicity is applied at the exclusion of all the other properties, we would need to, e.g., uniqueness: intentionally creating duplicates of questions; semantic coverage: intentionally avoiding covering the semantic details of a prompt in full. Each of the tasks falls on a continuous spectrum – how many duplications do we need to have to say the design is free of uniqueness? How many facts do we need to miss (percentage)?

As such, we opted the following strategy instead:

1. Implementing DSG in a way that the generated questions should fulfill all the four desired properties as much as possible (method description in Sec. 3);
2. Validating the extent to which each property is fulfilled with both auto and human evaluation (discussion in Sec. 4.1, result summary in Table 2).

tDEJ W1 & A5bo W1- enriched techniques & improvements for DSG

Both reviewers asked if the VQA module can be improved or enriched for stronger performance. Inspired by VPEval [1] which adopts different evaluation models for different evaluation queries, for the challenging text rendering and counting skills, we experiment with using an expert module for specific skill (e.g., OD for counting and OCR for text rendering), and compare the results in Metric vs. Human answer match accuracy.

For text rendering, we applied EasyOCR [2] for the questions measuring text rendering capability (e.g., “Do the letters say ‘Contour’?”) instead of the direct VQA method, on the DrawText split in DSG-1k. Specifically, we run OCR on images and check whether the target text is captured. As shown in the following table, we find the integration of OCR improved the Metric vs. Human answer match accuracy on the DrawText split with a strong margin.

For counting, we applied OWLv2 [3], a recent state-of-the-art open-vocab object detection model, for the questions measuring counting capability (e.g., "Are there five Halloween banners?”), instead of the direct VQA method, on the CountBench split in DSG-1k. Specifically, we detect bounding boxes with the target object name mentioned in the question and check whether the number of detected bounding boxes is equal to the number mentioned in the question. As shown in the following table, unlike OCR model in text rendering, the OWLv2 OD model didn’t improve the Metric vs. Human answer match accuracy on the counting skill.

In summary, we thank the reviewer again for the suggestion, and believe the inclusion of the result brings additional value in the understanding of the technique choice in handling the QA validation of the QG/A approach. We included the additional results in the Appendix H. Please also note that our DSG questions are labeled with the type of challenge that they probe (e.g., counting, text rendering), which makes them especially suited to be automatically routed to the area-specific techniques if they’re shown to perform better than the generalistic direct VQA approach.

In addition, reviewer tDEJ asked for further details on our DSG technique. In regard to this, we included the pseudo-code for our full pipeline (question generation → question answering → calculating final score) in Appendix E in our original PDF, which exposes all the details of our algorithm. Furthermore, we attach the DSG code in the supplementary file.

Reviewer A5bo is curious whether a stronger VQA model would further increase the VQA-human alignment with the direct VQA query method. Indeed, better VQA performance would help improve the quality of QG/A T2I evaluation in general. This is exactly the motivation for our extensive human evaluation in this work — identifying the semantic categories that can be automated with VQA models, and ones that remain in need of human judgments. We would like to emphasize that the question generation method (i.e. in the form of DSG) is what we consider the key innovation — on the back of the evidence illustrating the strong faithfulness of DSG-generated questions to human judgment (Table 2), we believe DSG is a QG approach that brings crucial and substantial improvement over previous work on the reliability of QG/A framework.

---

[1] Cho et al., Visual Programming for Text-to-Image Generation and Evaluation, NeurIPS 2023.

[2] Jaided AI,. Ready-to-use OCR with 80+ supported languages and all popular writing scripts including Latin, Chinese, Arabic, Devanagari, Cyrillic and etc., 2023.

[3] Minderer et al., Scaling Open-Vocabulary Object Detection, 2023

(continued below)