JustLogic: A benchmark for natural language deductive reasoning

Thank you for your detailed and constructive reviews! We’re glad that you appreciate the significance of the flaws in current reasoning benchmarks and how JustLogic addresses them.

W1: …mostly a source of confusion for both models and human labelers…However, this can be accomplished without contradicting world knowledge - see for example MuSR (Sprague et al., 2024)...

We conducted an additional empirical study showing that overriding the truth value of statements does not lead to confusion for both models and human labelers.

For models, we conducted an empirical study during the discussion period that proves statements whose truth values are unrelated to the real world do not lead to confusion. The full methodology and results are shown in Appendix C. If such statements lead to confusion, we should find that they exhibit lower accuracies.

However, our results show that when reasoning depth is 1, factually inaccurate conclusions in fact exhibit higher performance than factually accurate ones! When reasoning depth is 7 or less, factually inaccurate conclusions exhibit lower performance for Llama3-8B, Llama3-70B, GPT-4, and GPT-4o, but the opposite is true for OpenAI o1-preview. Moreover, the difference in accuracy is typically marginal.

These results show that the factual accuracy of conclusions has no clear or meaningful impact on accuracy, which suggests such conclusions do not lead to confusion.

For humans, we find that participants of our human evaluation are able to reason about the truth value of the statement while ignoring its truth value in the real world. First, our instructions explicitly ask participants to “assume everything in the paragraph is true” and to “only use information found in the paragraph”. We also anecdotally confirmed with several participants that our instructions were clear. Second, the perfect human ceiling performance of 100% and strong average human performance of 73.0% indicate that participants are not confused by such statements.

Third, the ability to deductively reason without drawing on the truth value in the real world is a fundamental human skill that is commonly used in daily life. For example, to evaluate whether a debater’s speech or journalist’s article supports their position, one must first figure out if their argument contains any logical flaws independent of one’s world knowledge. Given the prevalence of reasoning independent of world knowledge, we posit that while humans may find statements whose truth values do not align with the real world to be odd, they are not confused by them.

With regards to MuSR (Sprague et al., 2024), its approach is indeed interesting and relevant to our paper. A brief discussion is included in the Related Works section (L141). We find that its dataset construction method is currently unsuitable for deductive reasoning.

MuSR’s dataset construction method can be summarized as follows: First, a domain and gold set of facts is manually curated. Second, a reasoning tree is recursively generated using domain-specific prompts on an LLM. Third, a story is generated with GPT-4 and the reasoning tree.

We agree that MuSR achieves prior knowledge independence without contradicting world knowledge because (i) a fictional scenario is manually curated with no connection to the real world, e.g. a murder mystery story, and (ii) GPT-4 is used to generate a complete fictional story.

However, MuSR’s method cannot be applied to deductive reasoning tasks, such as JustLogic. Beyond deductive reasoning, MuSR also tests for inductive reasoning, generalization, and commonsense knowledge (Appendix F, Listing 4 of the MuSR paper). GPT-4 will undeniably make some mistakes during the story generation; the authors of MuSR analyze this in Appendix D. Mistakes include ignoring instructions, hallucination, and invalid logic. Despite the mistakes, the stories are acceptable in MuSR’s task because models can still generalize and inductively reason. However, mistakes will be highly problematic for JustLogic because any imprecision in the premises will make deductive reasoning impossible, therefore rendering the instance useless. If GPT-4 is used to generate JustLogic’s premises, it will no longer be 100% reliable.

Moreover, MuSR’s method is not without downsides. Specifically, MuSR has significantly poorer scalability and diversity. MuSR relies heavily on the human curation of domain, gold facts, and task format, prior to LLM generation. It thus only has 3 domains in the original paper. JustLogic, on the other hand, includes a significantly larger domain due to its use of GenericsKB sentences. With regards to scalability, JustLogic can trivially generate more instances with greater difficulty and domain diversity by running its code. MuSR, on the other hand, requires manual curation to expand into more domains.

(Further discussion on MuSR is continued in Part 2)

Nevertheless, thank you for bringing up this paper! MuSR’s approach deserves further exploration as LLMs become more reliable. In our future works, we will study the introduction of LLMs to Step 2 of JustLogic’s dataset construction, while taking deliberate measures to ensure that the validity of each instance is maintained.

To summarize, given that (i) factually inaccurate statements do not lead to confusion in models and humans, and (ii) alternative methods to ensure prior knowledge independence, e.g. MuSR, are flawed for deductive reasoning, we maintain that our use of factually inaccurate statements is an effective solution to prior knowledge independence.

W2: High "natural language complexity", by which the authors mean vocabulary size. I think this is a straightforward overclaim…In my opinion this would also imply moving beyond propositional logic, as natural language largely revolves around predicate-argument structure.

We agree that moving beyond propositional logic and the predicate-argument structure will bring natural language complexity to the next level. However, while we strive towards this target, this may be too challenging for deductive reasoning benchmarks currently.

Deductive reasoning benchmarks are typically backed by a formal logic system; the natural language texts are generated or curated based on it. Thus, some level of linguistic rigidity is expected. To illustrate this, below are two sample texts from FOLIO, a human-curated deductive reasoning benchmark, and CLUTRR, a synthetic one. The newly added Appendix D shows more examples and analysis from other benchmarks.

• FOLIO [1]: “All people who regularly drink coffee are dependent on caffeine. People regularly drink coffee, or they don't want to be addicted to caffeine, or both. No one who doesn't want to be addicted to caffeine is unaware that caffeine is a drug…”
• CLUTRR [2]: “The bald eagle is not rough. The bear does not need the bald eagle. The dog needs the bear. If someone is rough then they chase the bald eagle…”

Evidently, both examples do not reach the natural language complexity of news articles or fiction books. Nonetheless, FOLIO is noticeably more linguistically diverse than CLUTRR. Therefore, in line with the existing literature on deductive reasoning benchmarks [1], we measure natural language complexity based on vocabulary size and the number of domains.

Given that complexity is evaluated in comparison with other deductive reasoning benchmarks, the claim that JustLogic has high natural language complexity is justified.

Nonetheless, thank you for drawing attention to this! Improving natural language complexity will be a significant contribution to the deductive reasoning benchmark literature. A discussion regarding this has been added to the Future Works section (Appendix E).

W3: The 'future-proofness' of the dataset is called into question by the fact that o1-preview outperforms the human average…the authors claim that the dataset can be scaled to match stronger models but do not actually demonstrate this for the strongest model.

We posit the o1-preview’s superior performance to the human average does not call into question the dataset’s ‘future-proofness’.

First, while OpenAI o1-preview outperforms the human average, which includes non-expert participants, it still significantly underperforms the human ceiling (100%). Thus, the model still has substantial margin for improvement.

Second, we demonstrate the JustLogic can be scaled to match stronger models in the error analysis (Section 5.3). In Figure 3 (right), we show that OpenAI o1-preview’s accuracy generally decreases as reasoning depth increases. Llama3-70B and Llama3-8B exhibit similar trends. Thus, to future-proof JustLogic, we can increase reasoning depths to raise the dataset’s difficulty level.

Suggestion: It would be helpful for clarity if you could use a term other than 'context-independence'...

Thank you for the suggestion! We agree that the term “context” could be confusing because it can be used in two different contexts. The term has been changed to “prior knowledge independence” for better clarity.

Q1: How is the human ceiling computed?

Human ceiling is measured by the highest accuracy of a single human participant.

[1] Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq Joty, Alexander R. Fabbri, Wojciech Kryscinski, Xi Victoria Lin, Caiming Xiong, and Dragomir Radev. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840, 2022. URL https://arxiv.org/abs/2209.00840.

[2] Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint arXiv:1908.06177, 2019.

Thanks for addressing my question and suggestion! Your points on prior knowledge independence are well taken. I'm keeping my score the same, as I still think the vocabulary diversity in JustLogic doesn't constitute meaningful linguistic complexity -- I feel existing benchmarks with purely synthetic language i.e. SimpleLogic (Zhang et al., 2022) cover the same ground semantically.

W3: Error analysis is somewhat superficial…A more interesting investigation would be regarding at which step the model starts to fail and why, as it would be a direct guidance of steering the model towards the correct reasoning path and avoid straying from the subject.

We have considered conducting such an investigation. However, this requires considerable effort because different models structure their CoT responses differently. In fact, different questions within the same LLM can result in vastly different forms of CoT. Thus, every response must be manually and individually parsed to identify where exactly the model failed.

Moreover, based on our examination of several LLM responses, there is a wide and complex array of potential errors, including hallucinated premises, not using relevant premises, incorrect usage of argument forms, and even responses that are simply incoherent. Such an analysis requires careful and extensive study, which is beyond the scope of this paper.

The error analysis (Section 5.3), where we study how reasoning depth and argument form affect accuracy for various models, is the first step toward a more thorough investigation of where models fail and why.

We agree that this is an interesting and promising direction, which we intend to explore in the future. A discussion on this has been added to the Future Works section (Appendix E). Thank you for the suggestion!

W4: It would be better to add another section to prove the robustness of the benchmark by finetuning a model on the synthetic deductive data, showing whether the benchmark can be hacked by training on test data.

As observed in other deductive reasoning datasets, finetuning on the same dataset always leads to some degree of ‘hackability’. Fine-tuned models exhibit significant performance improvements. In FOLIO [3], a human-curated dataset, the fine-tuned Flan-T5-Large (783M parameters) outperforms GPT-4. In CLUTRR [4], a synthetic dataset, models were finetuned on a dataset with depth=2 and 3. Graph Attention Network achieved near-perfect performance on these depths, but performed significantly worse as depth is increased.

Therefore, we expect models to perform better after being finetuned on the JustLogic dataset. In line with other deductive reasoning benchmarks, this does not challenge the robustness of JustLogic.

Moreover, two features of JustLogic reduce its hackability. First, JustLogic’s reasoning depth can be increased. Just like CLUTRR, if the model is finetuned on depth=2 and 3, then the model can be tested on depths > 3, in the event that the model memorizes all the possible permutations of argument forms at lower depths.

Second, JustLogic’s training and test data can have completely different linguistic structures. Specifically, GenericsKB sentences that appear in the train data will not appear in the test data. Moreover, the expressions of each logical form (Table 3) used in the train data will not be used in the test data. This can be trivially implemented and verified using JustLogic’s program.

[1] Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3621–3634, Online. Association for Computational Linguistics.

[2] Sprague, Z., Ye, X., Bostrom, K., Chaudhuri, S. and Durrett, G. 2023. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. arXiv preprint arXiv:2310.16049.

[3] Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq Joty, Alexander R. Fabbri, Wojciech Kryscinski, Xi Victoria Lin, Caiming Xiong, and Dragomir Radev. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840, 2022. URL https://arxiv.org/abs/2209.00840.

[4] Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint arXiv:1908.06177, 2019.

Thank you for your detailed and constructive reviews! We’re glad that you appreciate how JustLogic addresses the limitations of existing deductive reasoning benchmarks.

W1: It does not make much sense to “intentionally” choose factually inaccurate statements…For example, if a model is already pretty sure about the fact that “Japan is in Asia”, then this degree of certainty would likely to affect the reasoning about the question that leads to “Japan is not in Asia”, so the influence of prior knowledge is not “eliminated”...A more reasonable principle of choosing the statements would be complete irrelevance of real-world knowledge, e.g. “My cat Fluffy likes paper bags”.

You are indeed right that the effects of prior knowledge can be on both positive and negative sides. However, it is not the case that if a conclusion is factually inaccurate, models may use prior knowledge and therefore lead to artificially low accuracies.

First, our prompt explicitly instructs models to answer the question only using the paragraph provided, and not to use prior knowledge (Appendix B). Moreover, in few-shot prompts, the examples provided include conclusions where their factual accuracy does not match the correct answer. These measures encourage models to ignore prior knowledge even when conclusions are factually accurate.

Second, we conducted an additional empirical study showing that the factual accuracy of conclusions has no clear impact on model accuracy. The full methodology and results are shown in Appendix C. If prior knowledge interferes with the deduction of factually inaccurate conclusions, then such questions have lower accuracies.

However, our results show that when reasoning depth is 1, factually inaccurate conclusions in fact exhibit higher performance than factually accurate ones! When reasoning depth is 7 or less, factually inaccurate conclusions exhibit lower performance for Llama3-8B, Llama3-70B, GPT-4, and GPT-4o, but the opposite is true for OpenAI o1-preview. Moreover, the difference in accuracy is typically marginal. Further explanations of these observations can be found in Appendix C.

These results show that the factual accuracy of conclusions has no clear or meaningful impact on accuracy, which suggests prior knowledge independence is upheld regardless. Therefore, factually inaccurate conclusions do not confuse models due to contradicting prior knowledge.

With regard to using statements irrelevant to real-world knowledge, the existing literature suggests that this approach exhibits significant flaws. In order to generate fictional stories or arguments, every sentence must be constructed from scratch, rather than derived from a natural language text database like GenericsKB. There are currently two ways of accomplishing this: (i) programmatically and (ii) using an LLM.

ProofWriter [1] programmatically generates fictional sentences using limited grammar. The following is a sample text: “The bald eagle is not rough. The bear does not need the bald eagle. The dog needs the bear…”. More examples from CLUTRR and ProntoQA can be found in Appendix D. Evidently, programmatic generations significantly compromise natural language complexity. Sentences no longer represent real-world scenarios and domains. Such datasets are significantly less realistic than JustLogic.

MuSR [2] uses GPT-4 to generate fictional stories. While it does have high natural language complexity, the stories sometimes contain mistakes, e.g. ignoring instructions, hallucination, and invalid logic. MuSR’s tasks also test for inductive reasoning and commonsense knowledge, so minor inaccuracies in the stories do not affect the viability of the dataset. However, JustLogic solely tests for deductive reasoning, so any imprecision in the premises will make deductive reasoning impossible, therefore rendering the instance useless. If GPT-4 is used to generate JustLogic’s premises, the dataset will no longer be 100% reliable. Therefore, MuSR's method of using GPT-4 is not viable for JustLogic's purpose.

To summarize, the intentional use of factually inaccurate conclusions does not cause confusion in models, therefore making JustLogic’s approach justified for ensuring prior knowledge independence. Moreover, JustLogic’s approach is superior to alternative approaches due to their significant flaws.

W2: One of the concerns would be: Is it assured that any two statements in the dataset are semantically independent with each other?

Regarding the example of contradictory statements, this is not possible because all sentences in GenericsKB are true.

It is possible that some sentences are semantically related. However, the chances are extremely small, considering there are 3.4M unique sentences across 129,987 topics.

To further mitigate this, we will edit the JustLogic program such that every GenericsKB sentence in a paragraph belongs to a different topic, thus further reducing the chance of accidental semantic dependence.

Thank you for your feedback. I appreciate your efforts, but your feedback does not fully convince me.

"You tell te model not to use prior knowledge" does not mean it genuinely does not use prior knowledge. For example, in system prompt I could say "do not say any words that might be harmful to human society", but I can still easily trigger the model to say something like this. Also, "However, the chances are extremely small, considering there are 3.4M unique sentences across 129,987 topics.", I don't think 3M / 129k is a small number. You need to further justify this in your newest draft if you have not done so yet.

" Thus, every response must be manually and individually parsed to identify where exactly the model failed." I don't think this should be left to future work. You should at least give an example of one popular model to show how your proposed task is challenging to it. Your error analysis is interesting but not somewhat lack of depth.

Regarding the natural language complexity of JustLogic: We hold that GenericsKB sentences and our templates are sufficiently complex. GenericsKB appears simple but actually contains highly diverse vocabulary and sentence structures. It extracts 1.7B sentences from Waterloo, SimpleWiki, and ARC, then selects 3.4M generic, standalone statements using lexicosyntactic rules and a BERT classifier. Moreover, many other datasets utilize sentences from GenericsKB [1,2,3], which suggests it has sufficient linguistic complexity to be used in datasets.

As for our templates, we believe you are referring to the expressions of various logical forms (Table 3), of which there are 50 in total. This introduces significant diversity because many statements in JustLogic are compositions of multiple logical forms. For example, the logical form x → ¬y contains a conditional and negation, which has 11 and 15 unique expressions respectively. Thus, this logical form has 11x15=165 possible expressions. Moreover, x and y are randomly chosen from GenericsKB’s 3.4M sentences.

Therefore, the JustLogic dataset has high natural language complexity.

L333: This may not be that simple. One may be able to prove the negation in a smaller set of steps. Is that considered as well? Or, the dataset does not have such examples?

The solutions provided by JustLogic are already the fastest way to arrive at the conclusion. This is because of how the argument structure is constructed (L252-259): the LLM must reproduce exactly the same argument forms in the same order to derive the correct solution. Therefore, the reasoning depth of each instance is well-defined.

L350-353: Very interesting paragraph. Again how do authors ensure the factual incorrectness of the conclusions?

Please refer to our response to W1, where we show that the factual correctness of the conclusions does not affect prior knowledge’s influence on the results.

L442:Why is the performance on CLUTTRR so low compared to JustLogic?

That is because CLUTRR has 16 answer options while JustLogic only has 3, therefore making it harder for the model to answer correctly. However, JustLogic’s performance is in fact better because its accuracy is lower than random probability which indicates that prior knowledge is not useful for answering the question. CLUTRR, on the other hand, has a higher accuracy than random.

L514: For the apparently long tail argument forms, have you qualitatively evaluated the CoT or some other ways to actually see what reasoning is being done by the LLMs?

We have considered qualitatively evaluating the LLMs’ reasoning. However, this requires considerable effort because different models structure their CoT responses differently. In fact, different questions within the same LLM can result in vastly different forms of CoT. Thus, every response must be manually and individually parsed to identify where exactly the model failed.

Moreover, based on our examination of several LLM responses, there is a wide and complex array of potential errors, including hallucinated premises, not using relevant premises, incorrect usage of argument forms, and even responses that are simply incoherent. Such an analysis requires careful and extensive study, which is beyond the scope of this paper.

The error analysis (Section 5.3), where we study how reasoning depth and argument form affect accuracy for various models, is the first step toward a more thorough investigation of how exactly LLMs reason.

Nevertheless, this is an interesting and promising direction, which we intend to explore in the future. A discussion on this has been added to the Future Works section (Appendix E). Thank you for the suggestion!

[1] Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862–6868, Online. Association for Computational Linguistics.

[2] Hao Peng, Xiaozhi Wang, Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, and Qun Liu. 2022. COPEN: Probing Conceptual Knowledge in Pre-trained Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5015–5035, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

[3] Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022. e-CARE: a New Dataset for Exploring Explainable Causal Reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432–446, Dublin, Ireland. Association for Computational Linguistics.

Thank you for your detailed and constructive reviews! We’re glad that you appreciate how JustLogic addresses the flaws of current reasoning benchmarks.

W1: …No effort have been made to enforce factual inaccuracies…I believe, such a process is required, if you really want to ensure that prior knowledge will not affect the results.

You are indeed right that some conclusions are factually accurate. However, it is not true if a conclusion is factually accurate, models may use prior knowledge and therefore lead to artificially high accuracies.

First, our prompt explicitly instructs models to answer the question only using the paragraph provided, and not to use prior knowledge (Appendix B). Moreover, in few-shot prompts, the examples provided include conclusions where their factual accuracy does not match the correct answer. These measures encourage models to ignore prior knowledge even when conclusions are factually accurate.

Second, we conducted an additional empirical study showing that enforcing factual accuracy is not necessary. The full methodology and results are shown in Appendix C. If factually accurate conclusions cause prior knowledge to affect results, we should find that such questions have higher accuracies.

However, our results show that when reasoning depth is 1, factually inaccurate conclusions in fact exhibit higher performance than factually accurate ones! When reasoning depth is 7 or less, factually inaccurate conclusions exhibit lower performance for Llama3-8B, Llama3-70B, GPT-4, and GPT-4o, but the opposite is true for OpenAI o1-preview. Moreover, the difference in accuracy is typically marginal.

These results show that the factual accuracy of conclusions has no clear or meaningful impact on accuracy, which suggests prior knowledge independence is upheld regardless. Therefore, enforcing factual inaccuracy is unnecessary to ensure prior knowledge will not affect results.

With regards to ProntoQA: It is indeed an important work in deductive reasoning evaluation. We have included the paper in our related works and the newly added Appendix D, which shows sample texts from various reasoning benchmarks and discusses their linguistic complexity.

Its “fictional” ontology guarantees factual inaccuracy. However, ProntoQA’s method leads to significantly lower natural language complexity because each sentence must be programmatically generated rather than extracted from a natural language text database. Moreover, sentences no longer represent real-world scenarios and domains, thus making ProntoQA significantly less realistic than JustLogic.

Here is a sample text from ProntoQA, which illustrates our point: “Lempuses are bitter. Every lempus is a lorpus. Brimpuses are vumpuses. Tumpuses are impuses. Each impus is not hot. Every numpus is a sterpus. Each shumpus is brown. Sterpuses are fast. Every vumpus is not small…”

W2: Overall the conclusion that every LLM except o1 is lagging -- seems to not add much to the ongoing body of knowledge. Why are some of the derivations difficult vs easy…

Thank you for raising this concern! We agree that it is important to extract insights beyond just the lagging performance of LLMs. As such, in our experiment results and error analysis, we show that…

1. LLMs continue to increase in size in hopes of improving their capabilities, including logical reasoning. However, our results show that increasing model size offers diminishing returns on performance (L459-461). Therefore, we expect that the performance gains in logical reasoning for SOTA models will start to plateau. Alternative methods for improving reasoning should be explored.

2. Performance improvements offered by increasing model size pale in comparison to those offered by better prompting methods, such as chain-of-thought (L462-465). For example, Llama3-8B with CoT prompting outperforms Llama3-70B with zero-shot prompting. This shows that prompting has an outsized effect on reasoning capabilities and that the specific mechanisms that cause this deserve further study.

3. OpenAI o1-preview outperforms the second-best model, GPT-4o, by 15 percentage points, which is a significant improvement. This proves that the former’s approach of using reinforcement learning and CoT prompting leads to substantially better deductive reasoning in LLMs (L466-468). To the best of our knowledge, this paper is the first to evaluate and highlight OpenAI o1-preview’s deductive reasoning abilities.

(Points 4 and 5 are found in Part 2)