We thank the reviewer for going through our work. We appreciate comments on novelty of the evaluation, on value of extensive experiments, on simple reproducibility and on clarity of the writing.

We further appreciate various points made by the reviewer and would like to address those in following:

Even though authors prove that SOTA LLMs fail on AIW task, I don't think we can claim that they are not capable of robust reasoning. On the contrary, paper shows that LLMs are capable of some types reasoning (like arithmetic, or basic family relations), but fail on the others (logical reasoning).

Based on evidence from our work, we do not claim that LLMs are entirely incapable of reasoning (see also Sec.5, p.10). As shown by AIW Light control experiments (Fig. 3, 4, 5), where problem structure is simpler than AIW original, tested models are able to infer problem structure and successfully select and execute operations across AIW Light variations - handling basic family relations, elementary arithmetic and set operations, binding female attributes to entities (Alice, sisters) - operations also required to solve AIW original. We do claim that all SOTA LLMs fail to generalize robustly and to perform robust reasoning. This is evident from strong performance fluctuations exhibited by all SOTA LLMs (Fig. 1, 2, 6) on AIW original variations that merely change numbers in otherwise the same fixed AIW problem template (Fig. B https://hackmd.io/_uploads/rJPFH38Gyl.png). Although problem structure and difficulty stays the same across AIW variations 1-4, and same operations are required for AIW solution as for control AIW Light problems, correct response rates vary strongly across variations. Even for most advanced models like GPT-4, rates can jump wildly from 1 to 0 despite entirely unchanged problem structure and difficulty. This cannot be called robust reasoning, especially not given the simplicity of AIW problem that can be arguably easily handled by elementary school children. It also though cannot be called entire absence of reasoning, as correct responses are still present across variations, albeit with strongly different frequency. We argue thus that what we observe is generic deficit in generalization consistently present in all SOTA LLMs, leading to lack of model robustness and fluctuations that cannot be explained alone by failures of specific low-level operations (required eg for arithmetical reasoning). Such failures would manifest equally across AIW variations - merely changing numbers N, M does not affect the way of operation execution, and fluctuations would not happen. Models claiming robust generalization and reasoning should be expected to solve simple problems like AIW across all variations with nearly 100% success rate without strong fluctuations, which is not what we observe. Open question for future work is what breaks problem structure inference and consequently proper selection and composition of low-level operations (like elementary arithmetics or set operations) to be executed already in such simple scenarios.

Why do you think Llama-3 performs so much worse then Llama-2 model? I wonder, if there might be any issues with prompts, as Llama-3 models require different special symbols in chat template than Llama-2.

Llama-3 does not perform much worse than Llama-2 - they rather perform equally bad. Misleading here is to look only at average correct response rates (Fig. 1), where the average is over AIW variations 1-4 and all RESTRICTED, STANDARD and THINKING prompt types. There, it might seem Llama 2 70B (p=0.3) outperforms Llama 3 70B (p=0.05) (see also Suppl. Tab. 7). However, a glance at the full distribution of correct response rates reveals both models are equally bad in handling the problem (Fig. D https://hackmd.io/_uploads/S1QjnkuMyx.png; paper Fig. 2). Eg Llama 2 70B shows extreme lack of robustness - significant performance is shown only on single AIW variation 3, all others being 0 or close to 0 (important to emphasize again that variations do not change problem structure and difficulty). Due to that single outlier, average correct rate is significant, but robust problem handling is obviously broken. Thus, Llama 2 has the same severe deficits as Llama 3. Llama 3 behavior cannot be explained by requiring special instruction templates, as also without templates it handles very well all the AIW Light control problems (Fig. 3,4,5, also fluctuations vanish) and shows competitive performance to Qwen 2 72B on female boost version (Fig. 6). Prompts are available at https://anonymous.4open.science/r/AITW_anonymous-69A6/prompts/prompts.json, collected data https://anonymous.4open.science/r/AITW_anonymous-69A6/collected_responses/raw_data_inspection/AIW_AIW_plus.json

formatting of the citations is wrong, punctuation issues

Thanks for helping to catch those. With regard to citations, using \cite instead of \citep caused havoc. We will correct all of that in the updated manuscript version.

We thank the reviewer for the time dealing with our work. We appreciate reviewer’s remarks on the paper being well written, and also pointing to the clarity of the ideas presented and the reproducibility of the findings by open-sourcing data and code.

We also appreciate various points made by the reviewer and would like to address those in following:

The problem setting of AIW has certain interfering factors … I believe that testing similar problems in mathematical reasoning tasks (like GSM8K or MATH) would be more convincing.

We think that problems formulated in GSM8K and MATH have on the contrary lengthier, overloaded formulations and thus a lot more interfering factors than the simple AIW problem (Fig. A, https://hackmd.io/_uploads/HJULH2IG1l.png ; Suppl. Fig. 16, 17 for bench failures). Main goal of our study was to find a “minimal” problem to convincingly falsify the hypothesis that current SOTA LLMs possess strong generalization and robust reasoning in zero-shot regime. Our intention was thus to reduce problem complexity as much as possible compared to standardized benchmarks and come up with a simple, short problem, unambiguously expressible in natural language, which despite its simplicity still could be used to stress test the core model capability for zero-shot generalization and reasoning. Contrary to problems used in GSM8K and MATH, AIW problem is a much simpler, short, common sense math problem that can be arguably easily solved by elementary school children. As we obtain strong evidence that all tested SOTA LLMs, including most advanced ones, cannot handle variations of that simple problem robustly and exhibit strong performance fluctuations, with correct response rate going from 1 to 0, despite variations being just different number instantiations (Fig. B, https://hackmd.io/_uploads/rJPFH38Gyl.png; paper Fig 1, 2), we can convincingly falsify the hypothesis without relying on more complex problems from GSM8K or MATH. Simple AIW structure also allows us to conduct control experiments (Fig. 3, 4, 5), ruling out failure in elementary operations as cause of the breakdown, which is unclear how to perform with GSM8K or MATH problems.

The issue here is that the LLM overlooks counting Alice herself, rather than lacking reasoning ability

While we observe various occasional failing operations that lead to wrong AIW solutions (and we agree that overlooking counting Alice herself is one of those), we think that we can convincingly rule out low-level operation failure like those as the main cause of the observed breakdown. We obtain clear evidence for strong performance fluctuations on problem irrelevant AIW variations 1-4 (instantiating different brothers and sisters numbers N,M) that all tested SOTA LLMs consistently exhibit (paper Fig 2, 6, Suppl. Fig 10; Fig. B https://hackmd.io/_uploads/rJPFH38Gyl.png). These fluctuations are impossible to explain when assuming the same low-level operations behind the performance - eg, failure to count Alice should then affect all variations the same way, as merely changing numbers N, M does not change the way of counting. Another evidence is obtained by our AIW Light control experiments (Fig. 3, 4, 5), where problems are constructed to contain operations that are also required to solve AIW original, though asking different questions to make the problem further simpler. We see fluctuations largely disappear, and models collapsing on AIW original show high correct response rates across all variations. Thus, we can prove low-level operations - handling family relations, binding female attributes, elementary arithmetics like counting - are actually intact. Remaining hypothesis is then generic generalization and basic reasoning deficits that result in failure to robustly infer problem structure. This explains the strong performance difference across different AIW variations despite the problem structure being the same, assuming different operations are inferred and executed.

AIW and GSM-Symbolic comparison

GSM-Symbolic (GSM-S) work follows our method to use problem templates and to introduce variations, measuring then how sensitive models are wrt. to variations with the same intention - drawing conclusions from observed model robustness or lack thereof about generalization and reasoning capabilities. We think GSM-S went the right way, however the evidence obtained there is rather weak to claim severe deficits like done in AIW (Fig. C, https://hackmd.io/_uploads/Sk7aBh8Gyx.png). There are two clear differences: 1) in our work, we observe strong fluctuations even for most advanced models like GPT-4, with scores going up to 1 and down to 0 across variations. GSM-S sees only weak fluctuation, eg. for GPT-4o being around 0.07. 2) we see many models collapsing to very low rates, while in GSM-S rates are still high. Eg Llama-3-8B scores $< 0.15$ on AIW, while $>0.69$ on GSM-S. Thus, while AIW provides clear breakdown evidence, evidence on GSM-S is inconclusive.

We thank the reviewer for the comprehensive feedback raising number of points, which would like to address those in following:

1. The whole paper is about one type of questions … hard to judge model’s capabilities based on one type of question alone.

Main goal of our paper is to provide convincing evidence for the falsification of the hypothesis stating current SOTA LLMs possess robust generalization and reasoning which is backed up by current standardized benchmarks. To do this, we follow the scientific method and search for a “minimal” experiment/problem - as simple as possible to already provide sufficient evidence - to convincingly falsify the strong function hypothesis. AIW problem satisfies this requirement - it is so simple that it can be arguably solved by elementary school children, having short, concise, unambiguous formulation in natural language (Fig. A https://hackmd.io/_uploads/HJULH2IG1l.png, Suppl. Tab. 2). Any system that claims robust zero-shot generalization and basic reasoning should be able to solve it across variations in numbers in the problem template with correct response rate close to 100%. As advanced SOTA LLMs like GPT-4/4o or Claude Opus/Sonnet claim generalization and reasoning on high school or even PhD level problems, observed lack of robustness and strong performance fluctuation across variations of such a simple problem (Fig. B https://hackmd.io/_uploads/rJPFH38Gyl.png; paper Fig 1,2,6) clearly disproves the claim, also revealing the deficits in standardized benchmarks to detect such clear generalization failures (Fig 7, Suppl. Fig. 16,17). Goal of hypothesis falsification is thus fulfilled by using one problem type that delivers sufficient evidence.

2. GPT-4o superior performance, model size or training?

We see clear effect of pre-training scale on the observed AIW performance. The only stronger performers that show correct response rates > 0.3 across variations, are large scale pretraining models like GPT-4 and Claude Opus (Fig. 1; Suppl. Tab. 7). All those stronger performers suffer though from strong fluctuations across AIW variations (Fig 2, 6). Models at smaller scales, eg Llama 3 8B, are staying well below 0.2, most of them residing close to 0 or collapsing entirely to 0 across all variations (Suppl. Fig. 8).

3. I don’t consider “female boost” as totally redundant information. … “She” as a sole indicator of Alice being a female is more a syntactic problem, which shouldn’t part of model’s burden …

We agree that to solve the AIW problem, SOTA LLMs have to infer the problem relevant information from natural language description and this requires also handling properly handling language syntax. Either using “Alice is female” or using “she” conveys same information - Alice being female - if language handling is successful, therefore we state that adding “Alice is female” does NOT add new information to natural language problem description. To rule out that such low level language parsing/handling issues are the problem behind observed failures, we conducted control experiments using AIW Light Arithmetic Total Girls problem version (Sec. 2.1, 3.1.1; Fig. E https://hackmd.io/_uploads/ByCpjM9M1x.png), which tests whether binding of female attribute via “she” is handled properly. We see most models that suffer clear breakdown on AIW solve AIW Light successfully, with high correct response rates and - importantly - vanishing fluctuations (Fig. 5). Thus, “female boost” (Fig. 6) is evidence for performance change due to entirely redundant information, again pointing to lack of model robustness and generalization deficit.

4. … paper adds nothing significantly to the existing discussion on whether LLM can reason or generalize. … a pure test on reasoning should not rely much on extra knowledge. The AIW test ... needs model’s understanding of “she” as Alice (syntactic) and basic family structure (external knowledge). The actual reasoning … is in my opinion … not the main bottleneck ... also supported in the paper where “female boost” can improve performance.

AIW problem is constructed exactly in a way to minimize the amount of knowledge necessary to handle it - in contrast to problems overloaded with various contexts used in standardized benchmarks (Fig. A). Using AIW Light control experiments (Sec. 2.1, 3.1.1), we prove that handling basic family structure, binding female attributes via “she”, elementary arithmetics like counting - are all intact (Fig. E https://hackmd.io/_uploads/ByCpjM9M1x.png; paper Fig. 3,4,5), ruling out that this additional knowledge poses an issue. Given the models show the ability to cope successfully with all required knowledge and operations to solve AIW, remaining hypothesis assumes generic generalization deficits responsible for observed strong fluctuations on AIW. “Female boost” is in line with that - behavior changes despite just adding redundant information, with strong fluctuations across variations still persisting (Fig. 6).

Thanks for the swift response. We think there is still a general misunderstanding how our work proceeds to demonstrate severe generalization deficit, as evident from the arguments in the response:

If I test human a bunch of tricky questions, and they get one question wrong, I wouldn't argue that human are fundamentally flawed at reasoning.

This is an entirely wrong analogy wrt. to our work, and we agree that works using such an approach would not be able to draw proper conclusions. We do not have an arbitrary bunch of tricky questions. We have an opposite situation. We have one simple question which can be arguably handled easily by elementary school children. We then generate variations using the question template, such that variations are natural for the posed problem and none of them changes the problem structure and difficulty. Thus, the bunch consists of problem instances generated from same simple problem where merely numbers are varied in the corresponding placeholder variables. We estimate correct response rate from many trials (at least 30) for each of problem instances in the bunch, such that we obtain correct response rate for each of AIW variations 1-4 (thus here, "get one question wrong" notion is actually not appropriate - we can only speak about lower or higher correct response rates over many trials). Puzzling is here exactly why there should be ANY difference in performance across such variations at all, as rate of getting "one question wrong", or right, across many executed trials should be roughly equal for all variations as they pose the very same problem - IF generalization were intact.

If we imagine how human would perform on such a task, we definitely DO NOT expect that their performance to solve such a bunch would wildly vary between very high and very low across variations (Fig. J https://hackmd.io/_uploads/B1MRvATzye.png). Imagine a test with human probands where a person is confronted with these variations of the AIW in many repetitive trials

Variation 1: Alice has 3 brothers and she also has 6 sisters. [Correct answer: 7 ]
Variation 2: Alice has 2 sisters and she also has 4 brothers. [Correct answer: 3 ]
Variation 3: Alice has 4 sisters and she also has 1 brother. [Correct answer: 5 ]
Variation 4: Alice has 4 brothers and she also has 1 sister. [Correct answer: 2 ]

How many sisters does Alice’s brother have?

Arguably, it would be shocking to observe persons having consistently over many trials close to 100% correct responses on Variation 4, while entirely breaking down on Variation 3 close to 0 (Fig. J; paper Fig. 2), as variations actually pose the very same simple problem without altering problem structure, only difference being in instantiated numbers. That is because human would be able to generalize, extracting the actual problem structure behind the bunch, and thus solve the problem independent of instantiated numbers equally well. This is how we also tested generalization of the SOTA LLMs, and see them consistently failing (Fig. B https://hackmd.io/_uploads/rJPFH38Gyl.png; paper Fig. 1,2,6), which we think given all the persisting claims about those models successfully handling math problems at olympiad level equally shocking and points to clear flaws not only in models, but also in benchmarks.

To evaluate reasoning, one has to look at the performance on average, otherwise the results hold no statistical value.

In our work, we test generic kind of reasoning - ability to infer underlying problem structure, which enables generalization and thus handling problems despite variations in their formulation. It matters which average to take to test this. Looking at performance averaged over all variations would be misleading, as breakdown of generalization is manifested in models' sensitivity to variations that is NOT visible if averaging over all variations. Eg, as evident in Fig. 1, 2, average correct response rates of better performers like GPT-4/4o or Claude 3 Opus, can be substantial. Only when looking at correct response rates of each variation separately, it becomes apparent that such average results from high performance on some, and low or very low close to 0 performance on other variations - despite variations being instances of the same problem (Fig. J https://hackmd.io/_uploads/B1MRvATzye.png). This breakdown in generalization would remain undetected if averaging over variations, and this is what also happens with standardized benchmarks that do not look into measuring performance on variations of problem instances (Fig. 7, Suppl. Fig. 16,17,18,19), stating high performance across static questions and creating misleading illusion of strong function where there are clear deficits. We thus average over right/wrong responses for each variation separately (>30 trials), obtaining statistics that allows conclusion about model sensitivity to variations, pointing to robustness or lack thereof (see again Fig. J for instructive overview).

We would like to emphasize again that our work introduces a measurement procedure for model generalization that makes use of the same simple problem template to generate problem structure and difficulty preserving variations (AIW variations 1-4; see Fig. K for overview https://hackmd.io/_uploads/rkPA9H-myg.png). Thus, we confront the models with instances of the same simple problem, which allows us to see how sensitive model is to problem perturbations that should NOT affect ability to cope with the problem IF generalization is intact. It is in contrast to testing models on a set of various problems that do not have much in common and where source of variations is not controlled, which is the usual case for standardized benchmarks (eg GSM8K, Fig. A https://hackmd.io/_uploads/HJULH2IG1l.png). To gather robust statistics, we execute many trials (> 30) for each such variation, estimating correct response rates for each variation from those multiple trials (Fig. J, https://hackmd.io/_uploads/B1MRvATzye.png). Amount of existing fluctuations in correct response rates across the variations reveals then model sensitivity/robustness and enables conclusions about generalization or its breakdown.

To further strengthen evidence that technique of introducing problem structure & difficulty preserving variations into same simple problem template can be used to measure generalization breakdown, we executed per reviewers request a number of additional experiments using various problem examples that we think provide conclusive evidence that measurement procedure works independent of a chosen problem type.

To confirm that the same observations hold for other simple problems of related kind, we conducted additional experiments with AIW versions with modified problem templates that differ from AIW Original. For instance, we either introduce Alice and Bob as two entities in the problem structure to deal with, or we replace brothers and sisters entities with male/female friends, abandoning family specific frame. Using same experimental procedure to create variations of these problem versions, we observe the same pattern as for the AIW original, especially the strong fluctuations across variations, confirming the existence of the same generalization deficits for further problem examples (Fig. I https://hackmd.io/_uploads/BJ1nqj3MJx.png)

We hope this amounts to convincing evidence that experimental setup we describe in our work is useful for community as systematically reproducible measurement method for falsification of the strong function hypothesis and for debunking overblown claims that rely on standardized benchmarks and overlook such clear deficits.

We also provide another illustrative example of such a debunking procedure on an example of recent case of NuminaMath-7B that was ranked 1st at the recent AIMO competition, solving 29/50 private set problems of olympiad math level (https://huggingface.co/AI-MO/NuminaMath-7B-TIR). Based on that evals, the claim was widely put forward that the model is capable of solving high school olympiad math problems (https://www.aimodels.fyi/models/huggingFace/numinamath-7b-tir-ai-mo). AIW problem has average elementary school level and does not require any advanced math knowledge. We tested NuminaMath-7B on AIW and observed a strong collapse of this model on AIW problem, with correct response rates close to 0 across AIW variations 1-4. Using AIW Light control problems, we can also see that NuminaMath-7B can handle all the low level operations and knowledge required to deal with family structure, ruling out that those are the issues. Using the AIW problem setting, we thus can contradict the strong claim of being capable to robustly deal with math problems (Fig F, https://hackmd.io/_uploads/SybG2hqz1x.png). Especially, breakdown in such a simple setting rules out that model will be able to deal robustly with olympiad level math tasks, debunking the claim and raising further questions about AIMO benchmark procedure used to measure model performance.

(For the collected data for this debunking experiment, see anonymous repo https://anonymous.4open.science/r/AITW_anonymous-69A6/collected_responses/raw_data_inspection/NuminaMath-7B_AIW_versions_problem_set_checked.json)

We thank the reviewer for dealing with our work and taking the time to try out the experiments with the AIW problem and test the findings. We appreciate positive feedback emphasizing the surprising nature of the discovery being valuable, the robustness of the findings across various models and well-written text easy to follow. We also appreciate further points raised, which we address in following:

The authors did not try to add illustrations (e.g., k-shot) to mitigate the issue. I tried to add an illustration, but some models still failed. That analysis would add value to the work. While more expensive, fine-tuning a small model would also add value to the consistency of the case study they present.

In our work, we focus on zero-shot generalization as an important mode of foundation/instruction fine-tuned model operation as it reveals a lot about its core capabilities to handle novel scenarios. Therefore, we leave the very interesting questions of either few-shot in-context learning or few-shot fine-tuning for future work. We were though too curious and ran preliminary experiments with few shot in-context learning. There, we were presenting few shots of solved AIW problem variations, posing then an unseen AIW variation as test problem. We observe that models often discover and stick with the “shortcut” solution C = M + 1 (M being number of Alice’s sisters) without learning to extract the true underlying problem structure. This becomes visible when switching in the test problem the question such that correct answer is not anymore obtainable by adding 1. Models often still respond to the altered test question by just adding 1 to the queried variable (Fig. G https://hackmd.io/_uploads/SyvWuj3fkx.png). This preliminary evidence hints that in-context few shot learning may have the same issues with obtaining strong robust generalization as observed for zero-shot case. Furthermore, we think that given the widespread claims of advanced models like GPT-4/4o, Claude 3 Opus to possess strong generalization and reasoning, such an embarrassingly simple problem as AIW should be handled with ease in zero-shot mode, were the claim to be defended.

… biggest concern is that the authors tried and reported only one example of failure across multiple models. To make an analogy with the adversarial robustness literature, this is equivalent to finding a single adversarial example in computer vision that makes most models misclassify an input (an example of a ‘universal trigger’).

We think that the analogy with the adversarial examples is correct only in one sense - adversarial examples also reveal lack of model robustness to variations that should not affect model function/performance, pointing to generalization deficits. However, our work is not of an adversarial kind. We do not start from a simple, solvable problem and probe various tweaks that do not correspond to “natural” problem variations to search for a problem input alteration that breaks the model. On the contrary, variations introduced into AIW problem template are simple, “natural” variations entirely in the sense of the problem structure, corresponding to instantiation of various numbers which leave problem structure and difficulty unchanged (Fig. A https://hackmd.io/_uploads/HJULH2IG1l.png). Those naturally generated problem instances do not contain any backdoors or tweaks aiming to trigger model glitches. We additionally make use of AIW Light control experiments to make sure that observed breakdowns and strong fluctuations do not origin in problem formulation specific issues, eg failing to parse language/numbers or to execute required low-level operations or access specific knowledge necessary for problem solution (Fig. E https://hackmd.io/_uploads/ByCpjM9M1x.png; Fig 3,4,5). Thus, our approach is almost the opposite of adversarial, as we attempt to make sure that problem formulations and variations are actually easy for the models to handle.

Further, to confirm that same observations hold for other simple problems of related kind, we conducted experiments with various AIW versions. For instance, we introduce Alice and Bob as two entities in the problem structure to deal with, or we replace brothers and sisters entities with male/female friends, abandoning family specific frame. Using same experimental procedure to create variations of these problem versions, we observe same pattern as for AIW original, especially the strong fluctuations across variations, confirming existence of same generalization deficits for further problem examples (Fig. I https://hackmd.io/_uploads/BJ1nqj3MJx.png)

The paper lacks a consistent analysis of other examples, and, in this form, it reduces the contribution to an exciting yet anecdotal showcase of failure.

We respectfully disagree with notion that failures that we study are anecdotal. Contrary to various indeed anecdotal examples, we conduct systematic evaluation of correct response rates of various SOTA models across variations while controlling variation source (Fig 1,2,6). We also execute control experiments to rule out various low level issues as source of observed failures and fluctuations (Fig. E https://hackmd.io/_uploads/ByCpjM9M1x.png; Fig 3,4,5). The failures and exact full statistics of model behavior can be reproduced following our approach of using fixed problem template while introducing problem compatible variations, which lead to strong fluctuations. As our main goal was to falsify the still widespread hypothesis stating current SOTA LLMs possess robust generalization and reasoning, which is usually backed up by current standardized benchmarks (for bench failures, see Fig. 7, Suppl. Fig. 16,17,18,19) , we searched for a minimal sufficient problem setting that will allow such falsification. AIW problem that is made so simple that arguably elementary school children can easily handle it (Fig. A https://hackmd.io/_uploads/HJULH2IG1l.png, Suppl. Tab. 2), allowed us to obtain such evidence falsifying the strong function claim, as it would be expected from models with even basic generalization and reasoning capabilities to solve AIW without strong fluctuations with correct response rates close to 100% across all variations. Given this clear evidence, no further other examples were necessary.

Plenty of articles show failure cases of LLMs on many examples and variations; one very popular is [1].

In line with preceding response, to our knowledge, ours is the first study that systematically quantifies severe lack of model robustness pointing to generalization deficit in a simple, well reproducible scenario, gathering statistics executing experiments across many repeated trials (at least 30 per each variation and prompt type combination, Suppl. Fig. 20), controlling for conditions and source of variations, obtaining evidence consistently across many SOTA LLMs including the most advanced large scale ones, and using control experiments to rule out various low level causes like natural language/numbers parsing, tokenization, problem specific ambiguities, failures to access specific knowledge (Fig. E https://hackmd.io/_uploads/ByCpjM9M1x.png; Fig 3,4,5).

We would also like to emphasize that variations we use are not rooted in prompt engineering or other arbitrary problem modifications that put model into vastly different operation modes - we make use of variations in a simple problem template that are natural part of problem instantiation (Fig. A https://hackmd.io/_uploads/HJULH2IG1l.png, Suppl. Tab. 2) and do not change problem structure, its difficulty or aim to heavily alter processing mode in general (like done in prompt engineering). Our approach makes it thus possible to draw from collected evidence about model sensitivity conclusions about generalization deficits, contrary to various previous anecdotal examples where such systematic measurements were not performed, one or only few models were used, control experiments were not executed, problem setting was ambiguous or prone to low level issues like tokenization, findings were hard to reproduce and thus no clear conclusions about conditions, extent and nature of model failures were possible.

We think the work done in [1] (https://arxiv.org/abs/2302.08399) is exactly of this "anecdotal" kind - it takes single model (GPT-3.5), goes through selected cases of failure without executing many trials under well controlled conditions to gather statistics on average failure rates related to various conditions, so that it is indeed impossible to conclude whether failures were due to specific problem type, input presentation, prompt type etc, or indeed due to deficits in core model function, such that those examples remain in anecdotal realm. One of motivations behind our work was to depart from anecdotal failure reports and do systematic evaluation study showing how such failures manifest in reproducible manner (all data available in the repo https://anonymous.4open.science/r/AITW_anonymous-69A6/README.md) and whether they indeed relate to core generalization deficits in al SOTA LLMs, which we hope to have succeeded in.

We thank the reviewer for taking the time and involving intensively into discussion. We appreciate raising the score following our initial rebuttal. In following, we would like to iterate on points raised:

I still hold the opinion, shared by most reviewers in this batch, that one example is not sufficient to say much about a model's capabilities … my biggest concerns still hold (one example is not sufficient to show any lack of reasoning …)

We think that there is still ongoing misunderstanding among reviewers when pointing out we deal with a single example only of what the study’s main goal is. Main goal of the study is to provide convincing falsification of the hypothesis that posits strong zero-shot generalization and robust basic reasoning exhibited by current SOTA LLMs (let’s call it “strong function hypothesis”). To falsify the hypothesis, following scientific method we construct a minimal experimental setup - as simple as possible to perform this duty - that gives us sufficient evidence to reject the strong function hypothesis.

Thus, we do not aim to test all the various abilities of the models. We aim to test whether the claim of strong function holds, to falsify which it is sufficient to present one clear contradiction backed up by proper systematic measurements. AIW problem and measurement procedure using its variations (see Fig. K for overview https://hackmd.io/_uploads/rkPA9H-myg.png) have two crucial properties for providing this contradiction and thus falsification:

1. It is a very simple, unambiguous, common sense math problem, arguably simple enough to be solved by elementary school children. Any system that claims to have robust problem solving capabilities - and current advanced SOTA LLMs (e.g GPT-4, Claude 3 Opus) maintain quite a strong claim to be able to deal with math problems of graduate student level ! - should be able to solve such a simple problem with correct response rates close to 100% if measured across many trials.

2. It uses natural variations around fixed simple problem template to systematically probe models’ sensitivity to problem perturbations that keep problem structure and its difficulty unchanged. Controlling for source of variations in this way, it is possible to make conclusions about models’ generalization from the observed robustness to those variations in a simple scenario. Again, any system capable of robust generalization should be able to handle the problem independent of such “natural” variations, showing strong performance on each variation as the problem structure is unchanged and so problem solving should be not affected. We would like to emphasize again that variations belong naturally to the problem, being mere number instantiations in the problem template (Fig. K https://hackmd.io/_uploads/rkPA9H-myg.png). Variations are not adversarial variations tweaked specifically to break the models, or prompt engineering like variations to tune the model performance. On the contrary, variations should keep the problem and model operation mode unchanged.

We obtained following evidence:

1. Overall low average correct response rates, averaged across many trials (> 30 trials for each combination of AIW variations 1-4 and 3 various prompt types, STANDARD, THINKING, RESTRICTED) (Fig. K; paper Fig. 1)

2. Strong fluctuations across variations (Fig. K; paper Fig. 2,6) - despite each variation being instance of the very same problem, with the only difference between variations being the instantiated numbers, performance can vary wildly from high to low, also for most advanced tested models like GPT-4 and Claude 3 Opus.

We thus think this clear evidence from experiments using single problem type is indeed sufficient to debunk strong function claim and to point to generic lack of reasoning - reasoning about problem structure that would allow robust problem structure inference and would enable robust generalization despite problem irrelevant variations. As pointed out in previous discussions, it is otherwise impossible to explain strong performance fluctuations observed across variations - assuming any operation or set of operations necessary to solve AIW being broken, performance should have been affected equally across variations, as those pose the very same problem. Via control experiments, we also provide additional evidence to rule out that the observed breakdown is due to failures in executing low level operations specific to AIW (Fig. E https://hackmd.io/_uploads/ByCpjM9M1x.png)

We thank the reviewer for taking the time to deal with our work. We appreciate feedback on its positive aspects emphasizing interesting finding of model breakdown, extensive experiments and analysis involving a large set of tested models. We also appreciate further points raised, which we address in following:

The study while interesting and definitely highlights a problem with modern LLMs is quite limited by the actual test set (basically being based on a single question and variations thereof).

Main goal of our paper is to convincingly falsify the hypothesis stating current SOTA LLMs possess robust generalization and reasoning, which is backed up by current standardized benchmarks. Following the scientific method to obtain such falsification, we search for a “minimal” experiment/problem, as simple as possible to provide sufficient evidence for hypothesis falsification. AIW problem, despite being a specific problem, together with the developed technique to measure models’ sensitivity to variations introduced into the problem template without changing problem structure or difficulty, satisfies this requirement (Fig. A https://hackmd.io/_uploads/HJULH2IG1l.png, Suppl. Tab. 2). For us, it is surprising to observe i) low correct response rates (Fig. 1, Suppl. Fig. 8) and even more importantly ii) strong performance fluctuations (Fig 2, 6) across AIW variations that do nothing but only change numbers N, M in such a simple problem setting. Models claiming robust zero-shot generalization and basic reasoning should have been able to solve AIW variations without fluctuations with correct response rate close to 100%. Together with control experiments that rule out failures in low-level issues like natural language/numbers parsing or handling specific knowledge about family structure, the evidence we obtain by using AIW problem setting is thus sufficient to disprove strong function hypothesis and also reveal generic generalization deficits beyond specific AIW problem scenario that are not detected by standardized benchmarks (Fig 7, Suppl. Fig. 16,17,18,19).

Finding specific phrasings that break a model is very common to most problems and to most people that have done prompt engineering. … single examples that work poorly and then others that work well … very common for most tasks

In our work, we do not deal with finding specific phrasings or formulations that break models. On the contrary, we intentionally design AIW problem formulation to be simple, short and unambiguous to focus on the effect of variations built into the template, which are as well constructed to be simple, natural variations (Fig. A, Suppl. Tab. 2). We think it is the most intriguing part of our work that strong fluctuations observed in all tested SOTA LLM are not caused by prompt engineering (which is indeed well known), but appear across problem variations that change neither problem structure nor its difficulty, observed behavior being consistent across various prompt types (Fig. B https://hackmd.io/_uploads/rJPFH38Gyl.png; paper Fig 2,6). The examples that work poorly and examples that work well differ in nothing else than instantiated numbers in the template, which allows conclusion about severe lack of robustness and generic generalization deficits. To our knowledge, ours is the first study that systematically quantifies such lack of robustness in a simple scenario, seeing all SOTA LLMs exhibiting this phenomena and using control experiments to rule out other causes. This approach makes it possible to draw conclusions about generalization deficits, contrary to various previous anecdotal examples where such systematic measurements were not performed, findings were hardly reproducible and no clear conclusions about extent and nature of model failures were possible.

… analysis should be much more detailed in terms of how and why models perform poorly or well on these tasks … "Physics of Language Models" [line of work] provides excellent analysis of how model perform and why they generalise poorly.

While we agree that looking into various types of model breakdowns and their causes is very fruitful in general, we would like to note that science is an iterative process. We hope to have made with our work one important iteration, discovering a minimalistic AIW problem and a technique to measure performance fluctuations across variations in the problem template that do not change problem structure or difficulty that allowed us to obtain clear evidence for severe generalization deficits in all SOTA LLMs (https://hackmd.io/_uploads/rJPFH38Gyl.png; paper Fig 2,6), contrary to previous claims. Follow up work can be to take this newly obtained evidence and conduct systematic studies elucidating origins of the fluctuations (where it helps that our control experiments could rule out low level issues; Fig. E https://hackmd.io/_uploads/ByCpjM9M1x.png; Fig. 3,4,5), which in turn may give hints what is required to get robust generalization.

Thank you for providing a detailed answer to our review.

(We apologise for answering late - as the current workload is quite high).

We raise the score to 5, however, we still have concerns and believe a revision is needed:

---

1. First study:

To our knowledge, ours is the first study that systematically quantifies such lack of robustness in a simple scenario, seeing all SOTA LLMs exhibiting this phenomena and using control experiments to rule out other causes.

There are works such "length generalisation of LLMs" that study generalisation of input length. And more recently also "What Makes Math Word Problems Challenging for LLMs?" that studies math problems and variations based on input. (Among other such findings).

---

2. Accuracy in rebuttal

This approach makes it possible to draw conclusions about generalization deficits, contrary to various previous anecdotal examples where such systematic measurements were not performed, findings were hardly reproducible and no clear conclusions about extent and nature of model failures were possible.

It would be good to be more precise when referring to previous work. (Which work was "hardly reproducible"?)

---

3. Size of the dataset

In the simplest form, a benchmark for measuring generalization can be constructed relying on already existing AIW problem variations. Using a robustness score computed over the shape of measured fluctuations distribution, models can be ranked in their generalization capability. A larger and more diverse dataset can be created by procedurally generating further AIW versions, where further variations can be introduced, e.g. varying names of entities, relational structure of the problem, and so on.

We believe that this would be required for this work to be a meaningful contribution to our field. At this stage we think that it is too narrow of a study and dataset and a bigger more useful dataset would be interesting - especially since in the next update of LLMs this benchmarks might be already obsolete (due to its limited size and breadth).

Controlling for low-level issues, contrast to reports of anecdotal model failures

We also would like to stress that one of the motivations behind our work was to depart from anecdotal reports on various failures of SOTA LLMs done in the past. To gather evidence for model breakdown and postulate existing generalization deficit, we performed systematic evaluation study across a large variety of SOTA LLMs with robust statistics accumulated over multiple trials (> 30 trials for each AIW variation 1-4 and each of 3 prompt types) under various conditions while controlling accurately for source of variation (Overview: Fig. K https://hackmd.io/_uploads/rkPA9H-myg.png). This is in contrast to previous observations attempting to showcase model breakdown on simple problems. For instance, the reports on the problems like “What is larger, 9.11 or 9.9” or “Count letters “r” in the word “Strawberry” ” were suffering from lack of systematic evaluation, reporting single cases of failure without executing multiple trials and estimating failure rates and without controlling for various conditions, eg prompt types. Often, such problems suffer from ambiguous character (eg, adding “real numbers” may disambiguate the question above avoiding interpretation as dates) that makes it unclear whether model failure is due to ill posed problem specification or indeed due to core function deficit. In contrast, AIW problem was made intentionally to be simple to handle, without any ambiguities or quirks in the formulation.

Another issue common to anecdotal reports are confounds in form of low level issues, eg inability to parse the input properly due to tokenization issues, like it might be the case for “Strawberry”, or in general, inability to access and deal with highly specific problem knowledge, which might cause model function breakdown although actual generalization and basic reasoning might still be intact. In our study, we made an effort to rule out such low level issues causing observed model breakdown on AIW by conducting control experiments using AIW Light problems. AIW Light problems are constructed to test operations that are also required to solve AIW original, by keeping the problem template unmodified and altering the question for checking specific operations (Fig. E https://hackmd.io/_uploads/ByCpjM9M1x.png, Sec 2.1, Sec 3.1.1). By observing that models handle AIW Light problems successfully, we prove that tested models are able to handle well all "low-level" operation and specific knowledge required for AIW, like parsing natural language, numbers, tokenization in general, grasping basic family relations, binding attributes to entities (eg female attribute via she pronomen to Alice) or performing arithmetic operations. Importantly, fluctuations on AIW Light variations disappear almost completely (Fig. E; paper Fig. 3,4,5).

Thus, contrary to anecdotal examples, our control experiments allow us to rule out low level trivial issues or other quirks behind strong fluctuations and performance breakdown observed on AIW (Fig. 1,2,6), strengthening evidence in favor of the hypothesis stating generic generalization deficits behind the fluctuations.

Additional experiments with evidence for generalization deficits on further problem examples

Per reviewers request, we executed a number of additional experiments that we think further strengthen evidence pointing to severe generalization deficits in SOTA LLMs.

One experiment concerns an AIW version where numbers for brothers and sisters are instantiated to be in an exaggerated range not realistic for a typical family scenario. In this AIW version, offset 60 is added to numbers in AIW original problem. We observe the same pattern as in AIW original (Fig. H https://hackmd.io/_uploads/H1sLdj2Mye.png) - low correct response rates and strong fluctuations across variations. We also see slightly lower correct response rates on average compared to AIW original. This might point to further generalization deficits becoming apparent when dealing with numbers outside the expected problem specific range, despite problem structure left unchanged.

Further, to confirm that the same observations hold for other simple problems of related kind, we conducted experiments with AIW versions with modified problem templates that differ from AIW Original. For instance, we either introduce Alice and Bob as two entities in the problem structure to deal with, or we replace brothers and sisters entities with male/female friends, abandoning family specific frame. Using same experimental procedure to create variations of these problem versions, we observe the same pattern as for the AIW original, especially the strong fluctuations across variations, confirming the existence of the same generalization deficits for further problem examples (Fig. I https://hackmd.io/_uploads/BJ1nqj3MJx.png)