The Generative AI Paradox: “What It Can Create, It May Not Understand”

We thank reviewer W7YK for conducting a careful and comprehensive review, and for finding that “The motivation for the paper is superb”, and that the authors “do an excellent job introducing this paradox”. We note that key changes to the paper in the updated draft are colored red in the updated draft, and clarify points below with respect to the reviewer’s main concerns.

“I think it is not experimentally sound to have different vision models for understanding vs. generation”

We agree that a single vision model that both generates and understands would be an ideal system to test here. However, we make some clarifications to explain our decision:

1. Hypothesis 1 is referring to generative models as a technology rather than a single model in particular. That is, surprisingly good generation is achievable with generative models, compared to sometimes weak understanding that such models have achieved. We have worked to make this clearer in the paper text. (Section 2.1)

2. Vision models are capable of both image generation and understanding. Image generators can be used for understanding [1] but with significantly weaker performance than image understanding specialist models (which we use). Thus, our current results likely underestimate the extent of the generation-understanding discrepancy when considering a single model, as the generation performance will remain the same but understanding performance will be lower than what we have reported.

Thank you for bringing up this ambiguity. We have now clarified this in the paper text.

“What data was used for human generations in the language domain? This was not clearly spelled out from my reading?”

We use the human-written reference responses from the existing datasets that we test on. As these tend to be extensively validated, the only risk here is of overestimating human performance which would underestimate the prevalence of the paradox. This has been clarified in the paper text (Section 3 and Appendix D).

“Can you please provide further details on how participant agreement was calculated?”

There is a natural label bias in the annotations where many of the responses fell into a single label. In this case, standard inter-annotator agreement statistics are not reliable (the well-known paradoxes of kappa [2]). We thus reported the pairwise agreement, that is, percentage of the time two or more annotators agreed on a label. .This has been clarified in the appendix footnote 8.

“Can you provide more details on the skew noted?”

This is explained by the response above, with respect to label imbalance.

“the authors are perhaps a bit too flippant about the use of the word ‘understanding’ and ‘intelligence’ from a human cognition perspective.”

We use these terms following the conventions in the field, specifically considering commonly used terms like “Artificial Intelligence” and “Natural Language Understanding”. We have further clarified that these words are intended in a directed way, as when used in the given terms. (Section 1)

References

[1] Li, A. C., Prabhudesai, M., Duggal, S., Brown, E., & Pathak, D. (2023). Your diffusion model is secretly a zero-shot classifier. arXiv preprint arXiv:2303.16203.

[2] Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543-549. doi:10.1016/0895-4356(90)90158-l

Thank you for the rebuttal. The new figures and examples aid with clarity and understanding. Authors have addressed my concerns and answered my questions.

We thank reviewer n1Ya for a careful review, and are happy that they find that “novel and concrete findings are established” with “strong accompanying discussion”, and that our “work is well presented”. We note that key changes to the paper in the updated draft are colored red, and we outline changes and clarifications below:

“Additional diagrams showing the steps”

We have added a diagram (Figure 8) to provide a more detailed explanation of the full process, thank you for bringing up this potential point of confusion! We are happy to elaborate on this.

“Adjusting the plot to use alpha values”

We have adjusted the alpha values in the plot in Figure 2, and have also slightly reduced marker size so that the full set of data points is more visible. We are happy to make further changes as the reviewer requests.

“A handful of concrete outputs would provide further contextualisation for the kind of errors the models are making”

We have added a set of qualitative examples to the appendix (see Appendix C.3) demonstrating the inability of language models to understand their own generations (interrogative), and general errors of vision models (both interrogative and selective understanding).

“it does not mention ways it could be mitigated.”

We have added ideas for possible mitigations to our discussion section. In short, we suggest alternative optimization objectives, reduction of model memorization, and broad incentivization of understanding capabilities as 3 paths forward, inspired by the causes that we propose. Understanding the cause of this phenomenon will be necessary to find a mitigation strategy.

We thank reviewer icxt for carefully considering our work, finding it “important to the community” and our experiments “comprehensive”. We note that key changes to the paper in the updated draft are colored red, and respond to the reviewer’s specific concerns below:

“employs two metrics, selective and interrogative … doesn't delve into which metric more accurately reflects the models' comprehension”

We agree that a single measure of understanding would simplify analysis. However, understanding is complex and has multiple distinct pieces, as demonstrated by the diverse ways that humans are tested for understanding (e.g. multiple choice questioning, long-answer, oral examination). Selective and interrogative evaluation in our experiments attempt to capture two core aspects of understanding but future work will need to explore more aspects, rather than collapsing onto one. We agree that a better description of what each evaluation is testing will be useful, and we have elaborated on this in the paper text (Section 2.1, definitions). In short, selective understanding concerns the ability to recognize correct answers, while interrogative concerns the ability to explain correct answers.

“... comparative analysis of the generative models' selective and interrogative capabilities, and provide explanations for the observed results?”

We study a different set of tasks for the two forms of understanding, especially in the language domain. We largely evaluate selective understanding on short-answer questions, where it is feasible to judge the correctness of an entire response (for human annotators), while we judge interrogative understanding on longer texts, in which it is more meaningful to understand the possibly complex information contained in the generation of the model. However, there are 3 tasks for which these evaluations overlap (XSUM, Mutual+, and HellaSwag) and so we have added a comparison of these tasks to the Appendix (Figure 14). The two forms of understanding seem to correlate, likely related to the relative difficulty of understanding the underlying task and stylistic aspects of the data. Understanding this relation warrants further study.

“insights presented in this paper may not be particularly surprising”

While we agree that our work matches the intuition of many researchers, much work recently relies on the understanding ability of generative models (e.g. model-based NLG evaluation [1]), or even makes this idea explicit. For instance, recent highly-cited work on GPT-4 states that “one of the key aspects of GPT-4’s intelligence is its generality, the ability to seemingly understand and connect any topic” [2]. We believe that work like ours is needed to provide more concrete, experimental proof of the ways in which models can lack understanding, and break from human intuitions.

“The authors should consider enhancing the clarity and organization of the figures”

We thank the reviewer for pointing out points of possible confusions, and have made some improvements to our presentation, including the addition of an experimental setup diagram (Figure 9), and improving the visibility of all data points in Figure 2 (bottom right). We are happy to incorporate any specific suggestions the reviewer has.

References

[1] Liu, Yang et al. “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.” ArXiv abs/2303.16634 (2023): n. pag. [2] Bubeck, Sébastien et al. “Sparks of Artificial General Intelligence: Early experiments with GPT-4.” ArXiv abs/2303.12712 (2023): n. pag.

We thank reviewer izog for their feedback. We are excited that reviewer izog stated our paper is “timely”, “well written”, “principled” and “significant” for the research community and found our experiments, spanning a wide spectrum of tasks and two modalities, to provide a "complete" analysis. We note that key changes to the paper in the updated draft are colored red, and make the following clarifications:

“Did you collect any data which would reveal as to whether the participants can themselves figure if the models don't have understanding?”

While we don’t collect a large dataset of such examples, we have added our own interactions with GPT-4 to the appendix (C.3). Particularly, we include examples of simple questions which ask the model about its own generated stories, and the (often incorrect) answers that it outputs, to provide explicit examples of what the authors judge to be qualitative examples of misunderstanding.