LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models

R1: Clarification on why simpler versions were insufficient for ‘or’.

A1: Thank you for this comment. Yes, indeed we want to make an explicit distinction between ‘or’ and ‘xor’. We found that understanding the logical implication of 'or' when integrated into logical formulations posed challenges to both humans - our human evaluators, mostly graduate students - and models in terms of understanding it. So with the focus of the paper on the inference rules, we decided to make the desired meaning of `or’ explicit using the above construct, so that our human evaluations and annotations would be correct. See the below example from LogicBench(Aug) where the use of our template vs. the direct 'or' formulation:

Context: If Sam has not finished his homework, then he will not go to the party.

Contextual Template Question (from our dataset): "Based on the context, can we ascertain that at least one of the following must always be true: (a) Sam has finished his homework, and (b) He will not attend the party?"

Direct 'or' Question: "Can we conclude that Sam has finished his homework or he will not attend the party?"

From the example provided above, it becomes evident that a question using a direct 'or' can be challenging to interpret and may cause confusion. Hence, we choose to go with the more expressed formulation of ‘or’.

R2: Additional information on the metric and about meta-data

A2: Yes, your understanding is indeed correct. The question with the answer ‘yes’ has the correct application of the rule, and the question with the answer ‘no’ has a conclusion that is not supported by context (rule). Furthermore, we agree that different variation types can be added as metadata for a better understanding of data. Thanks for this valuable suggestion, and we will incorporate it in the revised version.

R3: Details Related to fine-tuning experiments presented in Sec. 4.3

A3: Thank you for this suggestion. With our exploration of augmenting data for training and showing that it is useful for generalizing to real-world logic datasets, we demonstrate an additional use case of LogicBench. Thus, we consider these experiments valuable. However, the core contribution remains to be systematical evaluation and highlighting flaws in the logical reasoning ability of LLMs.

Note: During the rebuttal period, we conducted a human evaluation on a subset of LogicBench(Eval). We will share the results earliest possible.

R4: Small-scale Human Evaluation on LogicBench(Eval)

A4: Thank you for this suggestion. Please note that, in Sec. 3.3, we performed a systematic evaluation to ensure that generated instances follow the intended logical structure. To further support the integrity and reliability of the benchmark, we hired three graduate student volunteers to manually check the quality of generated data instances. Due to the limited rebuttal period, we randomly selected 5 instances from each inference rule, resulting in a total of 125 instances across 25 reasoning patterns for human evaluation. In particular, annotators are asked to provide binary answers (yes/no) for “Validity of generated context”, and “Validity of generated question” to make sure they adhere to the intended logical structure. Each instance is annotated by three different annotators. The inter-annotator agreement, measured with raw/observed agreement is 0.956. When there was a disagreement between annotators, a majority class was preferred. We provided example task instructions at https://anonymous.4open.science/r/LogicBench-EEBB/human_eval/readme.md. Results reveal that, after majority voting, annotators provided ‘yes’ for all 125 instances showing that all given instances adhere to the intended logical structure which further supports the quality of data checked by authors. We added this study in Appendix L.

R1: Justification on why we use the prompting method to evaluate LLMs’ reasoning ability

A1: We note that the emergence of the prompt paradigm has shifted the assessment of LLMs, favoring zero-shot and few-shot evaluations over traditional supervised fine-tuning (SFT). Thus, our work also adapts a similar method and evaluates LLMs using Zero-shot-CoT, a standard practice for meaningful reasoning with LLMs. While evaluating LLMs such as GPT-3/4, we set the temperature hyper-parameter to 0 to control the variation in generation. Furthermore, we also conducted a manual qualitative analysis (section 4.3) to support our findings.

R2: What are the source of the sentences used to generate the pairs in Figure 2?

A2: We prompt the model (GPT-3) to generate source sentences (propositions) as described in Sec 3.2.1. The prompt template is presented in Figure 3, and the comprehensive prompt is presented in Appendix A.

R3: Are Yes and No questions balanced in the benchmark?

A3: No, they are not balanced. Thus, we presented a separate accuracy in Table 6, A(Yes) and A(No) to evaluate model performance. We updated this in the paper (Section 3.3).

R4: The authors indicate that the LogicQA and ReClor include richer reasoning formats, but will it necessarily undermine the impacts of finetuning on LogicBench?

A4: LogicBench is designed to evaluate LLMs on mainly single-step reasoning across a range of inference rules and show that they struggle with even single-step reasoning for complex inference rules. Furthermore, errors made during the initial single-step reasoning can be propagated through subsequent reasoning steps, potentially leading to wrong conclusions in multi-step reasoning scenarios. Thus, teaching models first to do better single-step reasoning via different approaches (fine-tuning in our case) and then utilizing them to perform reasoning over complex and longer formats (LogiQA, ReClor) can improve their performance (as shown in Section 4.3). Hence, we believe that indicating LogicQA and ReClor include richer reasoning formats will not necessarily undermine the impacts of finetuning on LogicBench.

R1: “being more correct in NO answers might be due to having bias to saying more NOs than Yes’s and not more than that”

A1: Please note that we presented our results separately for A(No) and A(Yes). Moreover, we used CoT to evaluate the model which is the standard practice for meaningful reasoning with LLMs, and also make sure that the final answer is based on logical reasoning steps but not simple heuristics/shortcuts. Hence, we believe that incorrect/correct answers (Yes/No) produced by LLMs are not simply because of bias but because of the underlying reasoning process. To further support this, we also conducted a manual qualitative analysis (section 4.3) to present our findings.

R2: Justification of lower performance of GPT-4 on PL as compared to FOL

A2: Thank you for raising this point. Please note that the results in Table 6 show an average improvement in FOL results because of LLMs’ high accuracy on two axioms, EI and UI. However, when we compare the performance across the six common inference rules between PL and FOL (MT, HS, DS, BD, CD, DD), GPT-4 achieves an average of 49.16% A(Yes) for PL and 39.54% A(Yes) for FOL which shows that our results show the expected behavior.

The high accuracy of GPT-4 in handling EI and UI can be attributed to their simplicity. While from human experience and complexity theory, FOL is harder than PL in general; in the LLM context, the crucial factor becomes what kind of logical sentences LLMs are pre-trained on. It seems that LLMs are pre-trained more on simple FOL sentences than on simple PL sentences. To get an indication of this we gave GPT-4 prompt_1 - “Give twenty statements that have ‘if’ and ‘then’ in them” and prompt_2 - “Give twenty statements that have material implication "or" in them”. From the results, we can observe that 11 of the 20 sentences in response to the first prompt were FOL kind and only 9 were propositional kind. On the other hand, all 20 statements in the response to the second prompt were FOL sentences. This shows that LLMs’ comprehend simple FOL sentences, thus, showing high accuracy on simpler FOL axioms, i.e., EI and UI. However, LLMs exhibit consistently low performance across other complex/longer inference rules that are common to both PL and FOL (e.g., CD, BD, etc.).

Thanks for the author's response, I read the responses and the new analysis provided in the rebuttal phase. I think the paper is a good one as mentioned in my initial review.

R1: Discussion on how well LogicBench can generalize to real-world logical reasoning tasks

A1: Thanks for this comment. Please note that the core contribution of our proposed work is the diagnostic benchmark for systematically evaluating the logical reasoning ability of LLM across 25 different reasoning inference rules. This benchmark goes beyond simple inference rules in classical logic and incorporates more complex inference rules including NM reasoning. Moreover, the evaluation of various LLMs on LogicBench(Eval) leads to interesting findings and shows shortcomings in their ability to accurately solve even single-step reasoning problems. Furthermore, with our exploration of augmenting data for training and showing that it is useful for generalizing to real-world logic datasets, we demonstrate an additional use case of LogicBench. However, the core contribution remains to be systematical evaluation and highlighting flaws in the logical reasoning ability of LLMs.

R2: Justification of high few-shot performance of GPT-4 on LogicBench

A2: In the few-shot setting, we provide exemplars corresponding to the inference rule required to answer the test question. Bigger models such as GPT-4 have been shown to be remarkably good at following the in-context exemplars and mimicking the process to reach the correct conclusions. Thus, leveraging the in-context exemplars, GPT-4 achieves high accuracy in a few-shot setting (Table 13). In contrast, the zero-shot setting evaluates the true capabilities of these models in carrying out logical reasoning, as we can not expect that oracle exemplars corresponding to particular logical inference rules will always be available as part of prompts. We also believe that something as fundamental as logical reasoning ability should be an inherent capability of the model which is evaluated in a zero-shot setting. We discuss this point in Section 4.1.

R3: Including results of open-source LLaMa-2-7B on LogicBench

A3: Thanks for this suggestion. We added an evaluation of LLaMa-2-7B on LogicBench(Eval) in Appendix G. Similar to the other LLMs such as GPT-3/4 in zero-shot setting, LLaMa-2 does not fair well on LogicBench(Eval).

R4: Supervised fine-tuning results on LogicBench using T5-large

A4: As rightly mentioned by the reviewer, LogicT5 indeed achieves high accuracy (∼97%) on LogicBench(Eval).

R5: Whether high/low performance on LogicBench truly indicate high/low real-world logical reasoning capabilities for LLMs?

A5: While we agree that real-world logical reasoning can be more complex and require multi-step reasoning, we would like to emphasize that our benchmark represents the first step in evaluating LLM across a spectrum of logical inference rules and shows that these LLMs struggle even with the single-step reasoning process when logical rules become complex. Furthermore, errors made during the initial single-step reasoning can propagate through subsequent reasoning steps, potentially leading to wrong conclusions in multi-step reasoning scenarios (which is more human-like reasoning). While our benchmark primarily addresses single-step reasoning and doesn't specifically delve into multi-step human-like reasoning, evaluation on LogicBench is a strong indicator of real-world logical reasoning capabilities for LLMs.

R6: Is there evidence to support the premise that training LLMs on synthetic logic data such as those in LogicBench genuinely enhances their general logical reasoning capabilities?

A6: Some recent works such as [1] show that training on synthetic data can improve the performance of out-of-domain logical reasoning datasets indicating enhanced general logical reasoning capabilities.

[1] https://arxiv.org/pdf/2310.00836.pdf

R7: Discussion on LLMs’ capability for cross-type generalization

A7: Thanks for this valuable suggestion. During the rebuttal phase, we conducted suggested experiments and included results in Appendix K in the revised version. Experimental results reveal that the model fine-tuned on PL performs fairly well on FOL (~ 86%), though it remains lower than the supervised model fine-tuned on FOL (~ 97%). A similar observation is made for the model fine-tuned on FOL which fairly does well on PL. However, both these models struggle to generalize on non-classical NM reasoning showing lower accuracy (~ 52%). Additionally, the model fine-tuned on NM reasoning struggles in generalizing to classical logic, PL, and FOL.

R1: Further Discussion on the role of LogicBench towards “human-like” multiple steps reasoning

A1: We note that our benchmark represents a crucial first step in evaluating a variety of LLMs across a spectrum of logical inference rules and shows that these LLMs struggle even with the single-step reasoning process when logical rules become complex. Furthermore, errors made during the initial single-step reasoning can be propagated through subsequent reasoning steps, potentially leading to wrong conclusions in multi-step reasoning scenarios (which is more human-like reasoning). While our benchmark primarily addresses single-step reasoning and doesn't specifically delve into multi-step human-like reasoning, it leads to valuable findings that LLMs struggle with even single-step reasoning for complex inference rules. Moreover, even in inference rules (e.g., HS, CD, DD, etc.), multiple sub-steps are involved in matching with the appropriate parts of the inference rule.

R2: Clarification on task and further justification of GPT-4 results on simpler PL problems

A2: Each task is a binary classification task with a Yes/No answer. Note that, the results in Table 6 show that GPT-4 demonstrates superior performance for simpler PL inference rules such as HS and MT as compared to others in PL which is expected. However, as we introduce more complex PL inference rules (i.e., longer inference rules that are not commonly used in human writing) such as DD, BD, and CD, GPT-4 starts to show a drop in performance. This is one of the findings in the paper as discussed in Section 4.3 (“Longer inference rules are still challenging”).

R3: Additional details on findings that are presented in Sec 4.3.

A3: Thanks for this comment. Please note that we performed qualitative analysis over selected samples (where the reasoning chain leads to incorrect predictions) to identify flaws in the reasoning process of LLMs, hence, low performance. We believe that different findings across various reasoning types are useful in terms of understanding mistake that is made by models in general. Moreover, this qualitative analysis supports the difficulty posed by negations and the challenges associated with longer inference rules.

R4: Use of ‘If’ as material implication in LogicBench

A4: Thanks for bringing up this interesting point. While the use of ‘if’ in natural language sentences in the wild could have other interpretations such as a subjunctive, we took care in our construction so that in our dataset ‘if’ meant material implication. Specifically, our data creation pipeline ensures the logical correctness of generated context and questions where ‘if’ is used as material implication, thus maintaining the reliability of our instances.

R5: Justification of lower performance of GPT-4 on PL as compared to NM

A5: Thank you for making this observation. Its explanation, as we give below, is illuminating. In the development of AI, non-monotonic logics were partly developed to formalize natural language constructs, such as “normally birds fly’’, that were not formalizable in a straightforward manner using classical mathematical logics. Thus, while it was difficult for researchers to come up with non-monotonic logics and formalize non-monotonic reasoning, the fact that they were usually motivated by natural language examples, suggests that many of the non-monotonic reasoning aspects are present in the NL text in the wild that is used in the pre-training of the ultra-large LLMs such as GPT4. On the other hand, some of the PL features are counterintuitive to humans such as if we have contradiction (a and ~a) then everything (even unrelated) is entailed. Also, some PL features are perhaps less prevalent in human writing (on which LLMs are trained) - such as Modes Tollens. Thus your observation is not a surprise and we will add this point to the paper.

R6: Is LogicT5 pre-trained from scratch using the T5 model recipe or from a pretrained T5-Large checkpoint?

A6: We train T5 on LogicBench(Aug) starting from a pre-trained T5-Large checkpoint.

We continued the discussion in the next comment. Thank you!

R6: Small-scale Human Evaluation on LogicBench(Eval)

A6: Thank you for this suggestion. Please note that, in Sec. 3.3, we performed a systematic evaluation to ensure that generated instances follow the intended logical structure. To further support the integrity and reliability of the benchmark, we hired three graduate student volunteers to manually check the quality of generated data instances. Due to the limited rebuttal period, we randomly selected 5 instances from each inference rule, resulting in a total of 125 instances across 25 reasoning patterns for human evaluation. In particular, annotators are asked to provide binary answers (yes/no) for “Validity of generated context”, and “Validity of generated question” to make sure they adhere to the intended logical structure. Each instance is annotated by three different annotators. The inter-annotator agreement, measured with raw/observed agreement is 0.956. When there was a disagreement between annotators, a majority class was preferred. We provided example task instructions at https://anonymous.4open.science/r/LogicBench-EEBB/human_eval/readme.md. Results reveal that, after majority voting, annotators provided ‘yes’ for all 125 instances showing that all given instances adhere to the intended logical structure which further supports the quality of data checked by authors. We added this study in Appendix L.

Thank you for your response.

R1: How were prompts chosen?

A1: Please note that prompts presented in Appendix H.2 (used in our study) are Zero-shot-CoT prompts and we use the original paper for Zero-shot-CoT [1] as a reference to create those prompts manually.

[1] https://arxiv.org/pdf/2205.11916.pdf

R2: Does Flan-T5 even respond to CoT?

A2: Yes, FLAN generates basic explanations for the answer when given a CoT prompt instead of generating direct predictions. Thus, we also included FLAN-T5-3B (along with GPT-3/4, ChatGPT, and Tk-instruct) in our study for comprehensive evaluation. Please see the below example response generated by FLAN-T5-3B using our prompt 3 (Appendix H.2):

Context: If Liam finishes his work early, then he will order pizza for dinner.
Question: If he won't order pizza for dinner, does this imply that Liam didn't finish his work early?
FLAN (prediction): The answer is yes because if he didn't order pizza which means that he didn't finish his work early.

From the response, we can see that FLAN generates basic explanations given the CoT prompt, not a detailed step-by-step process.

R3: Analysis of results by "Length" of inference rules

R3: Thank you for this suggestion. Context in the LogicBench(Eval) uses templates developed from formal expressions of inference rules, hence we use token length of context to measure the length of inference rules. The below table represents the performance of GPT-4 w.r.t. token length of context. Since inference rules from classical logic, PL and FOL, are formally expressed, we evaluate the effect of context length on the performance of GPT-4 on these inference rules.

From the results, please note that the performance of GPT-4 decreases with the increasing token length of context. Specifically, GPT-4 often shows low performance on inference rules with longer context (e.g., CD, DD, BD, etc.).

R4: How did we ensure ‘if’ as material implication in prompts?

R4: Our prompts consist of <sentences, context, question> triplets as exemplars, and we (authors) created those examples manually where we ensure the examples always contain ‘if’ as material implication. Furthermore, we manually validated LogicBench(Eval) to make sure that the generated context and question adhere to the intended structure of inference rules, and the ‘if’ in the inference rules that we use is the material implication ‘if’.