On the Paradox of Generalizable Logical Reasoning in Large Language Models

Q2. concern about closed-world setting In our experimental setting, Symbolic Tree is a closed-world dataset, meaning that any facts not presented or inferable in the dataset are assumed to be false. In other words, when the language model (LLM) outputs an "Unknown" answer, we interpret it as "False".

Consider the following false case example:

Statement: $r27$(patrick, marko)
Answer: We can use logical rule L24, which states that if there is a relationship of type 3 between B and A, and a relationship of type 3 between C and B, and A has property 1, then A has relationship 27 with C. 
From the facts, we know that there is a relationship of type 3 between Patrick and Theodor (F43), and a relationship of type 3 between Theodor and Marko (F44). However, we do not have any direct information about a relationship of type 1 involving Patrick. Therefore, we cannot use L24 to prove $r27$(patrick, marko). 
Thus, we cannot determine whether the statement $r27$(patrick, marko) is True or False using the given logical rules and facts.

Q3. the complexity and "gold proof" of induction tasks

In our induction tasks, we utilized the Symbolic Tree dataset, which consists of approximately 99 basic facts including gender information ("r43" and "r44"), "parentOf" ("r1"), and "inverse_parentOf" ("r45") relationships. Additionally, there are about 5-15 inferred facts related to the unknown (to be predicted) rule.

Note that a single rule can be equivalent to multiple rules. For example, the rule $\forall x,y,z: \text{parentOf}(x, y) \land \text{parentOf}(y, z) \land \text{gender}(x, \text{female}) \rightarrow \text{GrandmotherOf}(x,z)$ can be represented as two separate rules: $\forall x,y,z: \text{parentOf}(x, y) \land \text{parentOf}(y, z) \rightarrow \text{GrandparentOf}(x,z)$ and $\forall x,y,z: \text{GrandparentOf}(x,z) \land \text{gender}(x, \text{female}) \rightarrow \text{GrandmotherOf}(x,z)$. To simplify the induction evaluation process, we rewrite all logical rules by including only the "parentOf" and "gender" relations in the rule body. Doing so ensures that each inferred relation is implied by a single logical rule. We provide a rule template to define the rule search space, as detailed in Appendix A.2, and specific examples can be found in Appendix E.3.

Q4. Can you explain why In Table 1, for ChatGPT Zero-Shot-CoT is better than Zero-Shot for deduction, but in Table 2, it is worse for all depths? Does CoT help?

Indeed, in Table 1, we observe that the performance of ChatGPT in the Zero-Shot-CoT setting does not show a marked improvement over the Zero-Shot setting, especially when considering error bars. This suggests that the addition of CoT does not significantly enhance the model's ability to solve deduction tasks.

Table 2 presents an even more pronounced discrepancy, where CoT settings perform worse than the Zero-Shot setting. We hypothesize two main reasons for this observed trend:

1. The CoT settings perform worse than the zero-shot setting with a higher frequency of "Cannot be determined" answers. We speculate that this may arise from the LLMs struggling to generate explicit reasoning traces effectively.
2. Utilizing a step-by-step explicit reasoning trace might lead to error propagation, resulting in worse performance than directly obtaining the answer. In addition, this effect seems to be more pronounced in tasks with semantic contexts, as seen in Table 2. The discrepancy between Zero-Shot-CoT and CoT is greater. This observation might indicate that step-by-step reasoning could amplify the distracting effect of weird semantics, such as "The squirrel needs the dog."

Consequently, CoT might not always be helpful for reasoning tasks that involve in-context new knowledge for models with weaker reasoning abilities like ChatGPT.

Thank the reviewer for their comments.

W1. concern about the experimental results and conclusion

We have added error bars to the ChatGPT results in the revised version. Due to the cost of GPT-4, we do not provide its error bars. As seen in Table 1, the semantic performance under error bars is still significantly worse than symbolic performance.

As for Table 2, we observed that the symbol version performs comparably to the semantic version, primarily because the semantics in ProofWriter are irrelevant to commonsense. Essentially, tasks which require less intricate semantic understanding or minimal dependence on common sense knowledge are not significantly impacted by the presence or absence of semantics in the language models' performance, as highlighted in blue.

To further validate our finding that Language Learning Models serve as in-context semantic reasoners rather than symbolic reasoners, we introduced a counter-commonsense setting and conducted more ablation studies, as shown in Table 3. In addition to deduction tasks, we executed comprehensive experiments in induction and abduction, to provide a holistic illustration of our conclusions.

W2. concern about the results of human study

Piaget's theory of formal operations (https://en.wikipedia.org/wiki/Piaget' s_theory_of_cognitive_development) points out that individuals in early to middle adolescence are capable of hypothetical and deductive reasoning that involves assumptions with no necessary relation to reality, also including counterfactual thinking. While not all individuals use formal reasoning in their daily lives [1], most humans possess this skill to some degree. And they possess the cognitive ability to engage in formal reasoning when required. In addition to Piaget's theories, it has been suggested by citations [1][2][3] that humans possess symbolic reasoning abilities. To rigorously verify this claim, we will conduct a human study to measure human performance in these tasks.

[1] https://cbmm.mit.edu/research/2013-2018-research/exploring-future-directions/learning-and-reasoning-symbolic-domains [2] https://www.frontiersin.org/articles/10.3389/fpsyg.2014.00275/full [3] https://tigerprints.clemson.edu/all_dissertations/2932/

W3. definition of abduction

Abduction in logic traditionally involves generating plausible explanations for a given observation. Indeed, our approach to abductive reasoning aligns with this definition, particularly within the framework of first-order logic. Based on the definition of Logic-based abduction (https://en.wikipedia.org/wiki/Abductive_reasoning), a proof-theoretical abduction method for first-order logic have been proposed for finding explanations. In first-order logic contexts, abduction typically involves reasoning that is more definitive and less speculative than in commonsense reasoning scenarios. Most abductive reasoning in this domain tends to yield plausible conclusions.

Moreover, Regarding the specific task settings, our methodology follow the approach "Explanation Generation with Theory" described in the paper https://arxiv.org/pdf/2303.12023.pdf. This approach involves generating a new hypothetical fact h such that, when combined with a given theory C, can prove an observation O, i.e., $C \cup {h} |=O$. We also reference "Explanation Classification," where the task is to choose the most plausible hypothesis from given alternatives to explain sequential observations. Considering the complexity of generating new facts and the necessity of a feasible evaluation framework, we adapted our task to involve selecting the most plausible facts from existing ones. This adaptation allows us to maintain the rigors of first-order logic abduction while ensuring that our tasks are feasible and effectively evaluative.

Q1. the details of human study

We apologize for any confusion regarding Appendix I, which was designed to examine the influence of irrelevant information on logical reasoning. We will provide more details about the human study in the revised paper.

We recruited 11 participants from diverse science and engineering backgrounds, including computer science, electronics, artificial intelligence, and automation. Although they have basic understanding of simple logic concepts, they are not experts in logical reasoning. Therefore, we provided them with task instructions that explained the concepts of deduction, induction, and abduction, aligned with the illustrations and definitions of logical reasoning presented in Section 3 of our paper.

We then presented them with 18 specific tasks, including six tasks for each deductive, inductive, and abductive reasoning type. Each task closely resembled the zero-shot prompts given to LLMs. We refer to this setting as "zero-shot" because we did not provide any further specific examples to help participants understand the tasks. The examples can be found in Appendix I and detailed materials have been uploaded in supplementary.

Thank you for the reviewer's comments.

W1. The first part of the claims are made by direct prompt ChatGPT/GPT4. However, some gaps between the performances are not significant.

In Table 1, considering error bars, semantic performance is still notably worse than symbolic performance.

As for Table 2, we observed that the symbol version performs comparably to the semantic version, primarily because the semantics in ProofWriter are irrelevant to commonsense. Essentially, tasks which require less intricate semantic understanding or minimal dependence on common sense knowledge are not significantly impacted by the presence or absence of semantics in the language models' performance, as highlighted in blue.

To further validate our finding that Language Learning Models serve as in-context semantic reasoners rather than symbolic reasoners, we introduced a counter-commonsense setting and conducted more ablation studies, as shown in Table 3. In addition to deduction tasks, we executed comprehensive experiments in induction and abduction, to provide a holistic illustration of our conclusions.

W2. Some claims

Re W2 (1): The results of symbolic reasoning evaluation are provided in Appendix J.2. In this section, we perform deductive and inductive reasoning experiments while providing the models with only relevant facts. The findings demonstrate that LLMs are easily disrupted by irrelevant information, especially when confronted with longer contexts.

Re W2 (2): We suppose the reviewer is concerned about "rep_all_with_counter" setting presented in Table 4. We would like to emphasize that even though fine-tuned LLMs can leverage template matching to make predictions, it is still inevitable that they may become distracted by prior knowledge.

Re W2 (3): Our paper focuses on first-order logical reasoning abilities. Unlike the inference of machine learning which usually generalizes to the same distribution, an agent with “true” logical reasoning abilities can reason with any given information to solve new, unfamiliar problems, regardless of any prior knowledge or different knowledge representations. In other words, it can generalize across novel rules (Sec 1 Line 13). For instance, consider training on $r1(A, B) \land r2(B) \rightarrow r3(B, A)$ and testing on $r1(A, B) \land r2(B, C) \rightarrow r3(C, B)$, as depicted in Fig 3. In our experimental setting, RuDaS is the dataset featuring rules with similar forms as $\text{xx} \land \text{xx} \cdots \rightarrow \text{xx}$. FOLIO is a complex and diverse dataset for logical reasoning expressed in natural language. Through our comprehensive experiment results, we find fine-tuned LLMs on symbolic logic datasets cannot generalize to novel rules, highlighting that fine-tuning LLMs fails to achieve generalizable logic reasoning.

Thank you for recognizing our contribution.

additional interpretability analyses about failure cases.

We provided some failure cases and discussions in Appendix E. These failures can likely be attributed to interference from irrelevant information and the inclusion of hallucinated information.

the version of ChatGPT and GPT-4

We used the gpt-3.5-turbo-0301 and gpt-4-0314 versions for this study. We will provide this detail in the revised paper.

Thank the reviewer for their comments.

W1. The motivation behind the study is weak due to the discrepancy in baseline performance on logical reasoning tasks in question.

Indeed, Large Language Models have achieved impressive performance on a variety of reasoning tasks[1][2], such as arithmetic and commonsense reasoning. However, the extent to which LLMs have mastered formal logical reasoning remains unclear. Previous studies have shed light on LLMs’ reasoning abilities to some extent, but a deeper exploration of their underlying reasoning mechanisms is still necessary. Our work aims to fill this gap by providing a novel perspective, going beyond mere performance metrics to understand the essence of LLMs' reasoning processes.

The performance on the Symbolic Trees semantics version has achieved commendable results (91% accuracy on GPT-4). However, performance on reasoning datasets with semantics does not necessarily reflect a deep understanding or mastery of formal reasoning. When the semantics are decoupled, LLMs tend to perform much worse. Thus, our primary goal is to investigate whether LLMs possess genuine formal reasoning abilities or simply exploit superficial semantic associations between questions and answers to make intuitive predictions. This investigation is of great importance for advancing the development of artificial general intelligence (AGI).

Furthermore, we also experiment with fine-tuning Llama-2 on pure symbolic reasoning tasks to bridge the gap. Although it appears to close the gap, this is achieved through template matching rather than invoking the fundamental formal reasoning abilities. We identify this phenomenon to provide a clear understanding of the boundaries within which LLMs currently solve logical reasoning tasks.

Therefore, the interesting findings aim to motivate further investigation into unveiling the inner workings of black-box LLMs.

[1] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in Neural Information Processing Systems 35 (2022): 24824-24837. [2] OpenAI. Gpt-4 technical report, 2023. 1

W2. the use of the word "paradox" is confusing and unclear.

We would like to explore and highlight two key paradoxical observations in the capabilities of Large Language Models (LLMs) with respect to logical reasoning.

Paradox 1 revolves around the unexpected discrepancy in LLMs' performance in logical reasoning tasks. While LLMs like GPT-4 show high accuracy (91.1%) in tasks rich in semantics, their performance significantly declines when semantic cues are replaced with symbolic representations. This observation is paradoxical because, according to the formal definition of logical reasoning, LLMs' reasoning ability should rely purely on the form or syntax of premises and conclusions, independent of semantic content. Yet, our findings indicate that LLMs heavily rely on semantics, suggesting that current logical reasoning tasks may not fully reflect the true extent of LLMs' formal reasoning capabilities.

Paradox 2 emerges from our fine-tuning experiments on symbolic reasoning tasks. While supervised fine-tuning narrows the performance gap between semantic and symbolic settings, enabling LLMs to perform logical reasoning seemingly agnostic to semantics, their ability to generalize to unseen logical rules remains limited. This indicates that LLMs primarily use template matching mechanisms for reasoning, rather than engaging in genuine formal reasoning processes. This paradox challenges the common perception that fine-tuning effectively enhances LLMs' logical reasoning skills, suggesting that the improvement may be superficial rather than indicative of true reasoning abilities.

In summary, our paper identifies two paradoxes: 1) LLMs' reliance on semantics rather than engaging in formal reasoning, despite seeming logical reasoning proficiency, and 2) the superficial nature of improvements in logical reasoning through fine-tuning, which fails to achieve deep generalization to new rules. These paradoxes question the true logical reasoning capabilities of LLMs and emphasize the need for more nuanced understanding and development in this area.

Thank the reviewer for their comments.

Results with prompting

W1. similar question about content effect in LLMs' reasoning has been studied before.

While they conclude that "Language models show human-like content effects on reasoning", this is only a subordinate conclusion that suggests LLMs are in-context semantic reasoners influenced by semantic associations. However, their conclusion does not directly imply that "LLMs are not symbolic reasoners," as suggested by Paradox 1: "LLMs are in-context semantic reasoners rather than symbolic reasoners".

According to the definition of deductive reasoning from Wikipedia (https://en.wikipedia.org/wiki/Deductive_reasoning), logical reasoning should be formal and depends only on the form or the syntax of the premises and the conclusion, irrespective of the specific contents of this argument. To this end, our task decouples semantics and focuses on pure symbolic reasoning (only provided syntax). When we feed these to LLMs, they show significantly worse performance compared to when normal semantic words are fed. This phenomenon indicates that LLMs fail to invoke the basic formal reasoning abilities of humans but instead rely on shallow semantic associations for prediction.

Moreover, we conduct comprehensive experiments on various logical reasoning, including deduction, induction and abduction tasks and find that they perform notably worse in these types of reasoning tasks compared to deduction, regardless of whether semantics or symbols are used. When semantics are decoupled, the drop in performance is even more significant. These findings highlight the considerable room for improvement in LLMs' reasoning abilities and suggest that relying solely on semantics to achieve symbolic reasoning is challenging.

W2. specific examples of GPT-4 failures in induction and abduction

We provided some failure cases and discussions in Appendix E. These failures can likely be attributed to interference from irrelevant information and the inclusion of hallucinated information.

Results with fine-tuning

W3. the main result found models can internalize rules during training but fail to generalize to novel rules, with a limited number of rules tested. To convincingly conclude fundamental limitations in Transformers, a much larger number of training rules should be used.

We appreciate the reviewer's insightful comment on the necessity of task diversity and the scale of training data for effective in-context learning in language models. However, we would like to emphasize that our Symbolic Tree dataset included 28 logical rules (not "5 in training, 3 in testing" as the reviewer mentioned). Moreover, in our extended dataset, RuDaS, we significantly increased the diversity with up to 1,000 different rules. Despite this expansion in rule variety, the models still struggled to generalize to novel rules.

Additionally, we attempted to scale up the training data and extend training time to enhance the models' logical reasoning capabilities. As shown in Figures 5 and 6 of our paper, these efforts did not yield substantial improvements in performance. This suggests that simply scaling data and training duration may not be sufficient to overcome certain inherent limitations in transformers regarding logical rule generalization.

In the "Limitations and Future Work" section of our paper, we discuss these points, acknowledging that while our findings strive to address the limitations of scalability and diversity to obtain the most convincing results possible, they might not fully realize the potential of training with large-scale and highly diverse data.

With these considerations in mind, we will carefully refine and moderate our statements to ensure the precision and reliability of our paper.

Q1: Table 1: Why does induction column miss some values?

This is because we did not perform a few-shot setting. Symbolic Tree dataset has a limited number of rules, and these rules exhibit a degree of interconnectedness. For instance, a rule like 'grandAuntOf' can be derived through the composition of rules like 'sisterOf' and 'grandparentOf'. The potential overlap and dependencies among the rules could skew the assessment of the model's true inductive reasoning capabilities. Consequently, we chose to only conduct a zero-shot setting where we provide facts to infer new rules without exposing the model to similar examples.

Q2: LLMs can perform simple arithmetics and it couldn't have seen all possible additions / multiplications etc. during training. Doesn't this show it have some ability to learn rules beyond sematics?

The review's observation about LLMs performing simple arithmetic tasks raises an intriguing point about their ability to learn rules beyond semantics. However, our research suggests that this ability may not be as robust as it appears. The performance in simple arithmetic tasks could be attributed to data leakage or the application of pre-learned arithmetic solution templates, rather than a genuine understanding or mastery of arithmetic rules.

For instance, when testing GPT-4 with a large number arithmetic problem, such as "128373 + 2879321873=", the model provided an incorrect answer (2879449246 instead of the correct answer 2879450246). This error indicates a limitation in the model's ability to process and apply arithmetic rules, especially in the context of large numbers. In other words, we think if it truly master arithmetics rules (carry or borrow operations), it should process large number. It suggests that while LLMs may display a superficial capability to perform arithmetic operations, they may not truly master the underlying rules.

Therefore, while LLMs exhibit some level of proficiency in arithmetic, this should not be mistaken for a deep or comprehensive understanding of arithmetic rules. Our findings point towards a need for further development in logical reasoning area to enhance the true rule-learning capabilities of LLMs beyond semantic associations.

Thank the reviewer for their comments.

W1. lack of novelty

We would like to address your concerns regarding the novelty of our work and stress how our findings on paradox 1 differ from the conclusions drawn in prior works. We will discuss paradox 2 subsequently.

While Saparov and He (2022) conclude that "Language models show human-like content effects on reasoning", this is only a subordinate conclusion that suggests LLMs are in-context semantic reasoners influenced by semantic associations. However, their conclusion does not directly imply that "LLMs are not symbolic reasoners," as suggested by Paradox 1: "LLMs are in-context semantic reasoners rather than symbolic reasoners".

According to the definition of deductive reasoning from Wikipedia (https://en.wikipedia.org/wiki/Deductive_reasoning), logical reasoning should be formal and depends only on the form or the syntax of the premises and the conclusion, irrespective of the specific contents of this argument. To this end, our task decouples semantics and focuses on pure symbolic reasoning (only provided syntax). When we feed these to LLMs, they show significantly worse performance compared to when normal semantic words are fed. This phenomenon indicates that LLMs fail to invoke the basic formal reasoning abilities of humans but instead rely on shallow semantic associations for prediction.

Moreover, we conduct comprehensive experiments on various logical reasoning, including deduction, induction and abduction tasks and find that they perform notably worse in these types of reasoning tasks compared to deduction, regardless of whether semantics or symbols are used. When semantics are decoupled, the drop in performance is even more significant. These findings highlight the considerable room for improvement in LLMs' reasoning abilities and suggest that relying solely on semantics to achieve symbolic reasoning is challenging.

[1]Saparov, Abulhair, and He He. "Language models are greedy reasoners: A systematic formal analysis of chain-of-thought." arXiv preprint arXiv:2210.01240 (2022).

W2. concern about the generalization

We would like to clarify that our paper focuses on first-order logical reasoning abilities, which will be emphasized in the revised version. Unlike the inference of machine learning which usually generalizes to the same distribution, an agent with "true" logical reasoning abilities can reason with any given information to solve new, unfamiliar problems, regardless of any prior knowledge or different knowledge representations. In other words, it can also generalize across facts, rules and representations (Sec 1 Line 13):

1. Facts: We test the same rules as in the training but with different facts (Sec 3.2.1 Line 4), as shown in the testing settings in Table 4.
2. Representations: Logical rule reasoning should be independent of different semantic content, including the rep_xx_with_xx examples in Table 4.
3. Rules: We test different first-order rules, such as training on $r1(A,B) \land r2(B) \rightarrow r3(B,A)$ and testing on $r1(A,B) \land r2(B,C) \rightarrow r3(C,B)$, as demonstrated in Fig 3. Different hop rules are also considered as different rules. Relevant settings include pf-sym-depth1, pf-sym-depth2, and Rudas in Table 5.

Our comprehensive experimental results show that LLMs fine-tuned on symbolic logic datasets can achieve generalization in (1) and (2) to some extent with the help of template matching. However, they struggle to generalize to novel rules, emphasizing that fine-tuning LLMs does not achieve generalizable logical reasoning.

W3. some minor issues: (1) All tables and plots: missing error bars (2) Tables 4, 5, 6 can have more informative row names. The current row names are hard to parse. (3) Table 6 is lacking context. What is the baseline we are comparing to?

Re W3 (1) and (2): We have included error bars and refined the row names in Tables 4, 5, and 6 to be more descriptive and concise in the revised version of our paper, with the error bars highlighted in red. Thank you for your suggestions.

Re W3 (3): Table 6 represents the performance of RuDaS and FOLIO of the non-fine-tuned LLaMA2-13B-chat model. The results reveal that, although FOLIO's performance surpasses random, it remains inferior to the non-fine-tuned LLaMA2-13B-chat. This further emphasizes our finding that fine-tuned LLMs struggle to generalize to novel rules.