Potemkin Understanding in Large Language Models

Thank you for your insightful review. We respond to your comments and describe new results; to summarize the main changes, we've added:

• An expansion of the coherence analysis to include 9 models
• A visualization of our mathematical framework
• A rewritten and simplified framework
• Analysis of question complexity

Your comments were very constructive and have improved the paper. We hope our comments have addressed your concerns. If not, please let us know if you have any more questions we can address in the follow-up.

While the experimental results support the claim that language models exhibit a discrepancy... the scope of the experiments is quite limited.

We like the way you've summarized our experimental results: "they support the claim that language models exhibit a discrepancy between the ability to explain a concept and the ability to use it." We'll update the language in our paper to reflect the way you wrote it.

We also agree that considering more models will make our results more robust. We've expanded our coherence evaluation to include 9 models -- see linked table. We'll use the time between the rebuttal and camera-ready deadline to add more models.

Our choice of domains spanned 32 subdomains that were intended to span three distinct forms of understanding: linguistic, formal, and behavioral. Potemkins were ubiquitous across all of these varied contexts, suggesting their presence in other areas as well.

The claim that "humans who can clearly define a concept necessarily possess a deep understanding and can effectively apply it" does not logically follow.

We fully agree that our original claim is too broad. In fact, this claim is not central to our work. What is central is that we test models on “keystones” - the set of questions that indicate understanding when people do well on them. We'll modify the language in our revision.

For the applications we choose (unlike math, as you point out) definitional accuracy is used to indicate understanding (e.g. it's what is used in exams to test understanding). For example, clearly defining sunk cost fallacy encompasses most of the understanding of that concept, as use cases are simple applications of it.

The presence of Potemkin understanding does not inherently invalidate the effectiveness of benchmarks.

We agree that the presence of potemkins doesn't invalidate benchmarks broadly. Benchmarks remain valid for assessing performance within the distribution they directly measure on held-out examples. Our point was subtler: Potemkins undermine the inference that high benchmark scores must reflect generalizable conceptual understanding. We'll clarify this point in the revision.

Even if models pass tests designed to assess Potemkin comprehension, one could still validly argue that they function as stochastic parrots, merely mimicking human reasoning rather than genuinely understanding or reasoning about concepts.

We agree that our experiments don't answer whether mimicking human reasoning would reflect genuine understanding, nor do they intend to. Rather, our goal is pragmatic: since developing models that mimic human reasoning is one of the goals of LLM research, it's important to measure this ability. We'll adjust our language to make this clear.

Section 2 contains lots of unnecessary math.

We've completely rewritten Section 2, simplifying and clarifying the mathematical presentation. See our response to reviewer voEv for more details. We've also included a new framework visualization here.

It's kind of already a consensus that generation is harder than understanding.

Our tasks testing "use" of concepts aren't exclusively about generation (e.g., classification tasks rely on recognition rather than generation). In fact, we show that models perform similarly poorly on classification and constrained generation tasks.

Prompted by your question, we've also gone back and re-examined our findings. While task complexity contributes in part to the Potemkin gap, we find that it can't fully explain Potemkin failures. For example, we find models correctly apply challenging definitions but fail simpler applications (e.g., correctly defining complex concepts like "Pareto Optimality" yet incorrectly classifying simpler instances of concepts like rhymes). This non-monotonic pattern of errors indicates conceptual gaps rather than just difficulty-driven mistakes.

How is this work contributing to future model development?

Good question. Once potemkins are discovered using the methods in our paper, we can build methods to train against them. Moreover the automatic evaluation tasks can be directly optimized during model fine-tuning. One strategy is to explicitly use the automated coherence experiment to penalize the inconsistencies that lead to potemkins. We feel this is the most important contribution of our benchmark and have added a discussion to the paper.

Thank you for your positive review of our paper. We're glad that you appreciated our paper and findings. We respond to your comments and describe new results; to summarize the main changes, we've added:

• An expansion of the coherence analysis to include 9 models
• Unaggregated results for table 2
• A rewritten and simplified framework

We hope our comments have addressed your concerns. If not, please let us know if you have any more questions we can address in the follow-up.

The definitions of \mathcal C_h, C_h, \mathcal C_l, C_l, are confusing to me. Are they sets of all possible strings to demonstrate people's understanding of the concept ?

Thanks for this feedback. It motivated us to significantly rewrite and simplify our framework, ensuring clarity while maintaining the core insights. First, we've included a new figure to provide a visual clarification of our framework (see attached).

Due to length constraints (and because OpenReview currently restricts submission updates), we can't include the full revised section here. However, here is an outline of the new structure:

• C*: set of correct interpretations of a concept. (Equivalently, a binary function over strings indicating correct or incorrect concept usage.)
• C_h: set of possible human interpretations (with C^* \in C_h)
• C_l: set of all possible model interpretations
• To test if a human's interpretation of a concept H is correct, it's intractable to compare all elements of H to C*.
• Because humans have structured ways to interpret concepts, though, we show there exist keystone questions - sets of questions that serve as proofs of understanding if answered correctly. By definition, these cannot be aligned with any human way to misinterpret a concept.
• But unless the set of LLM misunderstandings match human misunderstandings, there's no guarantee keystones work for LLMs: LLMs might interpret concepts in arbitrary ways.
• A potemkin is an instance where an LLM answers a keystone question correctly but doesn't understand the concept.

We hope this makes our theoretical framing clearer.

The authors provide verbal descriptions, but it would be helpful to provide some visualization (maybe a Venn graph?) and use the example of Haiku.

We appreciate your suggestion. We've added a new visualization of these concepts using a Venn diagram at the attached link.

In Line 153, why "keystone elements are valid tests for LLMs iff C_h=C_l"? Given the definition authors give about and C_k ("collection of sets, where each set represents a distinct and coherent category of how a human might understand a concept"), this seems like a very strong requirement.

We recognize our original language was unclear and as noted above have clarified the framework. We've also revised the manuscript to clarify the phrase “valid test”.

• By "valid," we specifically mean that answering keystone questions correctly implies true conceptual understanding in the LLM.
• We agree that it is a strong requirement for models to follow, which is part of the point of the paper. We will add a discussion of why if models misunderstand in ways that are distinct from how humans misunderstand, keystones that work for people will not work for LLMs.

In Table 2, the authors provided aggregated results across domains and models. In Table 1, we can see that the performance of different model-domain combinations can be significantly different (e.g., LLama-Game theory 0.10 vs Claude-3.5-Game theory 0.50), and thus aggregation can lead to misrepresentation/misinterpretation of results; standard error will not make sense here either. I suggest the authors provide un-aggregated results in Table 2.

Thanks for this suggestion. We've provided a new table with unaggregated results here. This new table is also expanded to include 9 models. We agree that the detailed breakdown makes it easier to explore specific model-domain combinations.

Thank you for your positive review. We’re glad you found our work novel, rigorous, and empirically interesting. We respond to your comments and describe new results; to summarize the main changes, we've added:

• An expansion of the coherence analysis to include 9 models
• Analysis of difficulty
• Proposed solutions for mitigating potemkins

We hope our comments have addressed your concerns. If not, please let us know if you have any more questions we can address in the follow-up.

Suggestions: (1) Incorporate evaluations of open-source models (e.g., LLaMA, Mistral) on this benchmark

This is a great suggestion. We've expanded our evaluation to include the following nine models on the coherence task, many of them open-source: LLaMA-3.3, GPT-4o, Claude-3.5, GPT-4.5, o1-mini, o3-mini, Mistral-7B, DeepSeek-v3, and DeepSeek-r1.

See this table for the full results. We'll use the time between the rebuttal and camera-ready deadline to expand our analysis in Table 1 to the same set of models. We also note that our initial analysis already included LLaMA-3.3, with results available here.

Suggestions: (2) Attempt to propose solutions or suggestions to mitigate this issue, reduce Potemkin failures, to assist subsequent researchers follow this work better.

As you suggest, once potemkins are discovered (e.g. using the methods in our paper) we can build methods to train against them. The automatic evaluation tasks in our benchmark -- e.g. concept use classification accuracy and self-coherence scores -- can be directly optimized during model fine-tuning. For example, a potential mitigation strategy is to explicitly use the automated coherence experiment to penalize the inconsistencies that can lead to potemkins. We feel this is the most important contribution of our benchmark and have added a section discussing it in our paper.

The paper primarily reveals the existence of the Potemkin phenomenon through experiments but does not delve deeply into its root causes, such as the model's training data, architectural characteristics, or optimization objectives.

We agree that our primary focus was documenting and quantifying potemkin understanding. As you point out, understanding the root causes would be very valuable. We speculate that this phenomenon arises due to a few reasons:

• Reinforcement Learning with Human Feedback (RLHF) prioritizes fluent and plausible-sounding explanations over accurate and coherent ones.
• No coherence training objectives: Models aren't explicitly trained to provide coherent responses, just accurate next-token predictions. Our automatic method for evaluating coherence provides a scalable and new way to train models and reduce the scale of potemkins.
• Limitations in Model Architecture: The architectural specifications vary by each model. However, models like transformers may inherently favor shallow associations due to limited inductive biases for structured reasoning or causality. We've expanded our discussion of these points in the manuscript. Future work should explore training interventions to directly address these causes.

The domains covered are incomplete; similar issues may arise in areas like mathematical reasoning and code generation, which could be included to enrich the dataset.

Our choice of domains—literary techniques, game theory, and psychological biases—was intended to span three distinct forms of understanding: linguistic, formal, and behavioral. Though incomplete, this set was intended to cover a wide range of types of understanding. Moreover, each domain includes multiple subdomains (32 in total). We found that potemkins were ubiquitous across all of these varied contexts, strongly suggesting their presence in other areas as well. Your suggestion to include mathematical reasoning and code generation is excellent because, like game theory, many of these problems can be evaluated automatically. We'll explicitly note this in our revision.

Although the gap between definition and application tasks is evident, the authors do not thoroughly explore the inherent differences in difficulty and complexity between these two types of tasks.

This is an important point. We've gone back and re-examined our findings. While task complexity contributes in part to the Potemkin gap, we find that it cannot fully explain Potemkin failures. For example, our benchmark contains cases where models correctly apply challenging definitions but fail simpler application tasks (e.g., correctly defining complex concepts like "Pareto Optimality" yet incorrectly classifying simpler instances of more intuitive concepts like rhymes). This non-monotonic pattern of errors—where easier tasks are sometimes failed, despite succeeding at harder conceptual definitions— indicates conceptual gaps rather than purely difficulty-driven mistakes. We'll clarify this point explicitly in our revised text.

Thank you for your careful and insightful review of our paper. We respond to your comments and describe new results below; to summarize the main changes, we've added:

• A rewritten and simplified framework
• A visualization of keystone questions and potemkins
• Analysis of the role of question complexity
• An expansion of the coherence analysis to include 9 models

The theoretical framework, while attempting to formalize the concept of potemkin understanding, adds complexity rather than clarifying the formulation.

Thanks for this feedback. It motivated us to significantly rewrite and simplify our framework.

First, we've included a new figure to provide a visual clarification of our framework.

While we don't have space to include the full revision here, an outline is below:

• C*: set of correct interpretations of a concept. (Equivalently, a binary function over strings indicating correct or incorrect concept usage.)
• C_h: set of possible human interpretations (with C^* \in C_h)
• C_l: set of all possible model interpretations
• To test if a human's interpretation of a concept H is correct, it's intractable to compare all elements of H to C*.
• Because humans have structured ways to interpret concepts, though, we show there exist keystone questions - sets of questions that serve as proofs of understanding if answered correctly.
• But unless the set of LLM misunderstandings match human misunderstandings, there's no guarantee keystones work for LLMs.
• A potemkin is an instance where an LLM answers a keystone question correctly but doesn't understand the concept.

The study doesn't sufficiently control for question complexity in the "use the concept" evaluations.

This is an important point, and the current draft does not address it. Prompted by your question, we've gone back and re-examined our findings. While task complexity contributes in part to the Potemkin gap, we find that it cannot fully explain potemkin failures. For example, our benchmark contains cases where models correctly apply challenging definitions but fail simpler application tasks (e.g., correctly defining complex concepts like "Pareto Optimality" yet incorrectly classifying simpler instances of more intuitive concepts like rhymes). This non-monotonic pattern of errors indicates conceptual gaps rather than just difficulty-driven mistakes.

However, I find that some cases for concept usage are overly complex for this setting. In Figure 2, while the first two problems seem reasonable, the third one is tricky...

This is an important observation. A reason why potemkins are common is that conceptual understanding of one concept often depends on other concepts. For example, even if a model successfully lists prime numbers, it must also accurately recognize digits (such as '1') to demonstrate full understanding. However, we agree with your point that a more direct example in the figure might better demonstrate Potemkin failures, and have updated the figure with a new example.

The evaluation methodology relies heavily on human expert annotation, which limits the scalability and practical applicability of the benchmarks.

We would like to highlight that while our methodology indeed uses human expert annotations in parts, much of it is automated, including:

1. Coherence evaluations (fully automated)
2. The classification task, which introduces 320 new questions as part of our benchmark. We'll highlight these points in the revision.

Could you elaborate on your claim "Keystone elements are valid tests for LLMs if and only if C_l = C_h" (Line 150)? Does this imply that your proposed benchmarks might not accurately reflect human understanding when tested on humans?

Thank you for pointing this out. We agree that this was unclear in the original text. What we meant (and hope the above clarifies) is that (i) keystone questions are constructed to precisely reflect human conceptual understanding and (ii) if LLMs do not misunderstand in the same way, models can do well on keystones while failing to understand a concept. As such, the benchmarks we use in the paper are explicitly designed so that human success on the benchmark aligns with genuine understanding.

Have you explored automating the evaluation process, perhaps using LLM-as-a-judge approaches?

As noted above, parts of our evaluation are already automated, making them scalable. But we appreciate your suggestion: LLM-as-a-judge methods are an interesting direction, and they're straightforward enough that we will implement them between the rebuttal period and camera-ready deadline. Because we already have expensive human labels, that will also help us evaluate how good LLM-as-a-judge methods are.

Typo suggestions

Thank you for finding typos. We've corrected these in the revision, and our simplified framework also reduces complexity.