RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning

We thank Reviewer 8kkH for commending our “methodology is well-designed”, our claims are “largely supported by empirical evidence”, and that our experimental design and analyses are “thoughtfully constructed, with several mechanisms to ensure validity”. We further appreciate and agree with their assessment that our paper is “well-situated within the broader scientific literature” and “contributes to multiple strands of prior research”.

Weakness/Question: “finding well-studied by previous works” [1, 2]

The only weakness raised by the Reviewer may stem from a misunderstanding as the objective, methodology, and findings of these two works are fundamentally different from ours. In light of the clarifications to follow and the Reviewer’s positive comments above, we respectfully ask the Reviewer to consider whether an adjustment to their initial score may be appropriate.

We kindly refer to Appendix A (lines 829-833), where we differentiate our work from Lampinen et al. (2024) [1] and the comparison table below. We did not initially include Wu et al. (2024) [2] because their methodology (applying a counterfactual ontology to logical templates) and findings have largely been established in Saparov and He (2023) [3], which we also already distinguished in Appendix A (lines 846-852). We are grateful for the suggestion and will add more discussion in the final version to make it clear.

As shown in the table above, our work differs significantly from [1] and [2] in that (a) we do not use a counterfactual ontology, (b) we do not introduce premises or conclusions that are untrue in the real world; and (c) our findings are not concerned with models’ performance in purely logical reasoning tasks, but with their ability to reason in general with natural language.

As the same two papers are mentioned in the Reviewer's question, we avoid repeating our response here for clarity.

Suggestion: alternative terminology

We thank the Reviewer for suggesting alternative terminology to connect our paper to other related work. We will incorporate this by describing rulebreaker cases as “factual contradictions arising from over-rigid reasoning with formal logic” in the final version. Additionally, we will make clear the connection and implications of our findings for LLM applications in knowledge-intensive tasks as we discuss in our S1 response to Reviewer twqU.

[1] Wu et al. https://aclanthology.org/2024.naacl-long.102

[2] Lampinen et al. https://doi.org/10.1093/pnasnexus/pgae233

[3] Saparov and He. https://openreview.net/pdf?id=qFVVBzXxR2V

We thank Reviewer fmAM for their helpful feedback. We appreciate their positive comments that our evaluation is “well executed and broad”, our analyses are “convincing” and that our paper is “clearly written, well presented and easy to follow”.

Q1: “How can GPT-4o be in Fig 6, if calculation relies on token probability?”

We can only access probabilities of the top 20 most likely tokens predicted by GPT-4o. The Reviewer is correct that Fig 6 is based on probability of the first token in the country name. By design, our experiment’s control for factual knowledge (lines 257-278) guarantees that the first token in the country name is the most likely token predicted by the model at that timestep. We can therefore access its predicted probability. By contrast, our method for model confidence (lines 251-267) requires access to probability of tokens which may not be in the top 20 predictions.

Suggestion 1 (S1): “claim that humans recognize 'rulebreakers', but no evaluation data provided”/Q2: “Did you consider providing a baseline based on humans solving these questions?”

Our dataset is based on rulebreakers used in prior cognitive science studies (see table below), which were all manually crafted by expert psychologists and tested on human participants under controlled settings. These studies already establish a baseline, finding that people generally handle these rulebreakers as we expected. Our key contribution is in systematically scaling these examples by identifying recurring patterns, creating templates and generating controlled variations that preserve the structure of these initial examples.

Also, the ability to recognize rulebreakers is contingent on reasoners having the relevant knowledge (e.g. that “Stockholm is in Sweden”). Since LLMs have often been observed to possess broader factual knowledge, directly comparing their performance on RULEBREAKERS against humans would not be valid. As such, we consider that an additional human baseline would add limited value. We are happy to conduct a small-scale human annotation but, given limited time to hire annotators, will include it in the final version.

Weakness 1 (W1): “idea is rather simple and only based on a few templates”

Our approach is intentionally grounded on existing cognitive science studies: we ensure the four templates (from which we generated 25,600 instances) are directly based on examples carefully crafted and validated in prior work with human participants.

Expanding the set of templates is a promising direction for future work but, as the Reviewer rightly implies, such new templates will also need to be based on examples that are validated empirically with human participants. We hope our novel methodology and findings will inspire efforts in this direction from both NLP and cognitive science communities.

W2: “only single step reasoning”

As the Reviewer rightly points out, models exhibit poor to mediocre performance even in single-step settings. We believe that a targeted and rigorous study of single-step inferences is critical in exposing blind spots that lead to failures in multi-step settings.

While we specifically focus on inherent patterns of a model’s reasoning process independent of any prompt engineering, future work could design prompting techniques to steer model behaviour and address the blind spot we identified of LLMs over-rigidly applying logical rules to accept and draw factually inconsistent conclusions.

S2: typos

Thank you for spotting these. We will incorporate the corrections in our final version.

[1] Quelhas et al. https://doi.org/10.1080/17470210903536902

[2] Quelhas and Johnson-Laird. https://doi.org/10.1080/17470218.2016.1154079

[3] Johnson-Laird and Byrne. https://doi.org/10.1037/0033-295X.109.4.646

We thank Reviewer QkHy for their helpful feedback and recognizing that our study was “extensive” in having “compared multiple advanced LLMs”.

Weakness 1 (W1): “human reasoning is undefined”/Question 1 (Q1): “how to define human-reasoning and why?”

We will replace references to “human reasoning” with “knowledge-informed reasoning”.

We define reasoning to be “knowledge-informed” if a reasoner incorporates factual and commonsense knowledge in their reasoning process, and does not draw or accept conclusions that factually contradict any premises. Accordingly, a model that accepts the conclusions in our rulebreaker cases would fail to be “knowledge-informed”.

Our definition is motivated by arguments in existing cognitive science literature that humans typically incorporate factual and commonsense knowledge to interpret natural language premises, and avoid drawing conclusions that, to their knowledge, factually contradict any premises. We discussed these in lines 45-63 and 96-116 and will add more justifications for our definition.

This term also highlights our findings' implications for LLM use in knowledge-intensive tasks, as discussed in our S1 response to Reviewer twqU.

W2: “example in Fig 1” is not counterintuitive

The example is commonly used in cognitive science works (e.g. [1], [2]) to demonstrate reasoning problems with “relevance conditional” statements. We will replace this with an example from our own dataset to better represent our rulebreakers.

W3: “dataset...was neither created nor validated by humans”

We kindly refer to our S1/Q2 response to Reviewer fmAM.

W4: “Limited exploration into internal model mechanisms that cause failure”

Please see Appendix M: we analyzed neuron activations in feedforward layers, complementing two potential causes diagnosed in Section 6. We welcome further suggestions on specific investigation methods.

Suggestion 1 (S1): “Potential effects of prompt phrasing…not deeply explored.”

We refer to Appendices D and F: we analyzed the breakdown of our results by different phrasings, and tested more re-phrasing and additions to confirm that our findings are robust against prompt variations.

S2: “hypothesis regarding model over-generalization …could benefit from further empirical validation”

See Appendices F and G: we validated our main results with further experiments, in addition to existing controls for models’ factual knowledge and prompt sensitivity (as above).

Q2: “Can you provide additional details on how model familiarity with entities impacts performance?”

To clarify lines 396-416, we found that a model generally performs worse on prompts containing unfamiliar entities, as compared to prompts containing familiar ones. This is supported by our qualitative analysis in Appendix I. This trend mirrors findings from cognitive science [3] that when participants are familiar with entities in the premises, they are more likely to recognize and avoid factually contradicting conclusions, as compared to when they are not.

Separately, when comparing across models, we found that simply being familiar with entities does not guarantee good performance on RULEBREAKERS: while Gemma-2-27b-it is overall highly familiar with entities in our dataset, it scores among the poorest in paired accuracy. This shows that a model can excel at recalling factual knowledge yet still struggle to apply that knowledge effectively in reasoning. We will add this expanded explanation in the final version.

Q3: “Have experiments been considered with models specifically fine-tuned for logical reasoning tasks?”

We did not do so, as our work initially aimed to highlight a blind spot that exists even in general-purpose models. We expect the behaviour of over-rigidly applying logic to be even more pronounced in such fine-tuned models.

To test this, we evaluate a Llama-3.1-8B-Instruct on Hugging Face [4] fine-tuned on a dataset for propositional logic reasoning, and compare its performance on RULEBREAKERS against the baseline model before fine-tuning.

As expected, the fine-tuned model performs better on non-rulebreakers but substantially worse on rulebreakers compared to the baseline. We will include a detailed discussion of these results in the final version.

[1] Johnson-Laird. Mental models. Cambridge University Press, 1983.

[2] Quelhas et al. https://doi.org/10.1080/17470210903536902

[3] Johnson-Laird and Byrne. https://doi.org/10.1037/0033-295X.109.4.646

[4] huggingface.co/ergotts/llama_3.1_8b_prop_logic_ft

We thank Reviewer twqU for commending that our “methods are reliable”, our “dataset represents a valid addition to the LLM community”, our experiments are “sound”, and results “highlight an open challenge”. We are glad they found our paper “primarily well-written”, echoing Reviewer fmAM.

Suggestion 1 (S1): “mention how failure could impact LLM real-world applications”

LLMs’ ability to detect and handle knowledge conflicts is crucial to ensure robustness and guard against misinformation. Suppose a model is given this context, like DS-type rulebreakers in our dataset:“The patient was informed that the treatment for their illness is available either in Berlin or somewhere in Germany. However, as it turns out, the treatment is not available anywhere in Germany.”

Models rigidly applying logic would incorrectly conclude that “the treatment is available in Berlin”, contradicting the fact in context. Instead, an ideal model should recognize that, since Berlin is in Germany, the treatment is not available in Berlin either.

To demonstrate, we add the bold phrases to premises in our prompts ("Suppose we are told that Anne is either in Stockholm or somewhere in Sweden. However, as a matter of fact, Anne is not in Sweden"), but found that models' performances on RULEBREAKERS remain mediocre (paired accuracy ranging 0.29 to 0.69). Due to space, we will add and discuss these results in the final version.

S2/Weakness 1 (W1): “Some findings not clearly presented: [Section 5.2, Section 6, Fig 5]”

We will replace Fig 5 with a table and revise Section 5.2 to improve clarity. Specifically, we will make clear that we use a model’s output probabilities as a measure of its confidence. We will also add to Section 6 the clarification we set out in our Q2 response to Reviewer QkHy.

W2: “selected only small/medium-sized LLMs”/Question 1 (Q1): “what would authors expect evaluating larger models?”

We included a 70B model and the large model GPT-4o. We do not observe a correlation between model size and performance on RULEBREAKERS: GPT-4o underperforms most models, and Llama-3-70B-Instruct underperforms its 8B variant. Thus, we consider that our results with selected model sizes already make a strong point regarding limitations of current LLMs, including state-of-the-art GPT-4o.

Q2: “Did authors consider impact of temperature when prompting?”/S3: “add experiments [re] temperature”

In our main experiment, we run only one forward pass to extract the most probable token without randomly sampling, so temperature is not used.

In Appendix G, we prompt LLMs to generate conclusions by greedy decoding. Our results already make a strong case, but we follow the Reviewer’s suggestion and test a subset of models. As Table 1 shows, random sampling with temperature yielded mixed effects for different models but did not alter our findings. We will add these results in the final version.

Q3: “performance of same LLM varies across rulebreaker types...no correlation between performance of different models on different types...any insight on why?”/S4: “add more insights on why these behaviours emerge”

We appreciate and agree with these observations. We expect models may differ in specific traits: some may be better at reasoning with geographical entities; others worse at recognizing some forms of negated premises. However, the fact that performance varies unpredictably shows that models do not handle these reasoning problems robustly or consistently. Rather, they may be affected by biases or quirks in the training data.

Q4: “additional experiments implementing hybrid LLM approaches?”/W3: “do not check how these hybrid models perform”

Our study aimed to expose a blind spot even in general-purpose models. We add a model specifically fine-tuned for logical reasoning (see our Q3 response to Reviewer QkHy), which enhances our findings and reflects our motivation for introducing “rulebreakers”.

Hybrid systems like LogicLM [1] use LLMs to parse natural language statements into logical form. However, each rulebreaker and non-rulebreaker in a pair share the same surface form and are both valid in propositional logic. Thus, even if the parsing is correct, the system would theoretically accept the conclusion in both rulebreakers and non-rulebreakers, hence scoring 0 in paired accuracy. We will add this discussion to the final version.

S5: “better representation of Fig 6”

Per the Reviewer's suggestion, we will replace Fig 6 with a table of mean familiarity values (and std) for clarity.

[1] Pan et al. https://aclanthology.org/2023.findings-emnlp.248