From Indeterminacy to Determinacy: Augmenting Logical Reasoning Capabilities with Large Language Models

We sincerely thank you for your constructive suggestions and valuable comments! We will answer the questions in detail and will appreciate very much if you could kindly raise your score if your concerns are addressed.

Q: technical contribution

A: Thanks for your constructive comments! We would like to clarify that considering the critical research question of augmenting the reasoning capabilities of LLMs, we were motivated by three important challenges in bridging the gap between LLM-based reasoning human reasoning. The proposed method provides a solution to this research question in terms of three key factors, which cannot be considered and achieved by baseline methods including CR. Our contributions can be summarized as follows:

1. We propose to formulate the reasoning process as the transition from indeterminacy to determinacy, which is a novel perspective to describe the essence of solving reasoning tasks;

2. We employ two-stage quantitative measurements (relevance and supplement) for premise selection and exploration, which can explore new insights by prioritizing given premises that are conducive to deriving conclusions;

3. We introduce a reasoning memory module to automate storage and extraction of available premises and reasoning paths, ensuring the consideration of essential historical reasoning details during the iterative reasoning process.

Q: systematic premise identification

The implementation of premise identification is to formulate detailed division rules through carefully designed instructions and demonstrations, which are available in revised Appendix. Compared with the identification results, the more important significance of this process is to enable the entire reasoning process to evolve from indeterminacy to determinacy. Considering your valuable comments, we will remove the adjective "systematic" to avoid misunderstandings.

Q: the definitions of relevance, supplement

A: All these functions are implemented by instructing LLMs and the prompt templates are detailed in revised Appendix. We would like to clarify that all three functions are innovations of DetermLR. relevance and supplement can prioritize better premises to explore new propositions, unlike randomly selecting premises in CR. verify consider multi-aspect verification checks, whereas CR only includes a "logical validity" check.

The main prompts and examples are provided below for easier understanding.

relevance and supplement

Prompt:

Please follow these steps:
1.From the determinate premise, select the "Most relevant premise" which has the same subject with "Hypothesis", and give a score from 0 to 1.
2.You need to assess how the "Most relevant premise" relates to all the other "determinate premise" and "indeterminate premise", based on Relevance scoring rules.
3.The "determinate premise" and "indeterminate premise" with scores higher than 0.25 will be used as the final supplement results, along with Most relevant premise.
Relevance scoring rules:
1. When scoring relevance, 0.25 added for each noun or 0.25 added for each adjective that is the same between two sentences.
2. Scores start to accumulate from 0 points, and the upper limit is 1 point.
3. If sentence p1 is a hypothetical premise of sentence p2, then add 0.25 to p2. for example: measure "if A then B." and "A is true." Then add 0.25 points to "if A then B".

E.g.:

Target: bear is nice and red (True/False/Unknown). 
Determinate premises: 
d1) The bear is big. 
d2) Billy is rough. 
d2) The tiger is rough.
Indeterminate premises: 
i1) If bear is big, then bear is red.
i2) If someone is big, then they are nice.

### After premise priority scoring: 
Most relevant premise $p*$: The bear is big.
Other Premises:  
i1) If bear is big, then bear is red.(0.75)
i2) The tiger is rough.(0.0) 
i3) If someone is big, then they are nice.(0.5) 
d2) Billy is rough.(0.0)
Supplementary premises *$\mathbf{p}_s$*:
i1) If bear is big, then bear is red.(0.75)
i3) If someone is big, then they are nice.(0.5)

So the selected premises for exploration: The bear is big. If someone is big, then they are nice. If bear is big, then bear is red.

New proposition $\widehat{p}$: The bear is nice and red.

Conclusion: True.

{{#system}}Suppose you are one of the greatest AI scientists, logicians and mathematicians. Let us think step by step. 
Please use the Logical Reasoning Rules(LRR) to determine whether the deduction of the given "Premises" to a "Proposition" is valid or not, reply with True or False.
Logical Reasoning Rules(LRR):
1. "Two premises": "If A,then B. A is true." then "Proposition": "B is true."
2. "Two premises": "If A,then B. If B,then C." then "Proposition": "If A, then C."
3. "Two premises": "If A,then B. B is not true." then "Proposition": "A is not true"
4. "Two premises": "A is either C or D. A is not C." then "Proposition": "A is D."
----{{/system}}
{{~#each examples}}
{{#user}}
---
"Premises": "{{this.premises}}"
"Proposition": "{{this.proposition}}"
{{/user}}

{{#assistant}}"Judgement": "Is this deduction valid? {{this.validation}}"{{/assistant}}
{{~/each}}

{{#user}}
---
"Premises": "{{premises}}"
"Proposition": "{{proposition}}"
{{/user}}

{{#assistant}}"Judgement": "Is this deduction valid? {{/assistant}}
{{#assistant}}{{select "validation" options=valid_validation}}{{/assistant}}

Selected premises: All eels are fish. No fish are plants.
New proposition: No eels are plants.
Validation: True.

Selected premises: Nothing that breathes is paper. All animals breathe.
New proposition: All animals are paper.',
Validation: False.

{{#system}}Suppose you are one of the greatest AI scientists, logicians and mathematicians. Let us think step by step. 
Please determine whether there is a new useful "Proposition". Reply with True or False.
----{{/system}}

{{#user}}
---
"Proposition": "There is no new proposition that can be deduced from the given premises to determine the correctness of the hypothesis."
{{/user}}
{{#assistant}}False{{/assistant}}

{{#user}}
---
"Proposition": "A Czech person wrote a book in 1946."
{{/user}}
{{#assistant}}True{{/assistant}}

{{#user}}
---
"Proposition": "There is no new proposition that can be deduced from the given premises that would be useful in determining the correctness of the given hypothesis."
{{/user}}
{{#assistant}}False{{/assistant}}

{{#user}}
---
"Proposition": "None of the premises provide information to deduce a proposition related to a Czech person writing a book in 1946."
{{/user}}
{{#assistant}}False{{/assistant}}

{{#user}}
---
"Proposition": "{{proposition}}"
{{/user}}
{{#assistant}}{{select "is_something" options=valid_something}}{{/assistant}}

New proposition: No new premises can be derived here.
Validation: False.

New proposition: I cannot derive new premises.
Validation: False.

{{#system}}Suppose you are one of the greatest AI scientists, logicians and mathematicians. Let us think step by step. 
Can this "proposition" can be derived using only one "premise"?Please reply with True or False.
----{{/system}}

{{~#each examples}}
{{#user}}
---
"Premises": "{{this.premises}}"
"Proposition": "{{this.proposition}}"
Can this "proposition" can be derived using only one "premise"?
{{/user}}

{{#assistant}}"Judgement": "{{this.duplicated}}"{{/assistant}}
{{#assistant}}"explanation": "this.explanation"{{/assistant}}
{{~/each}}

{{#user}}
---
"Premises": "{{premises}}"
"Proposition": "{{proposition}}"
Can this "proposition" can be derived using only one "premise"?
{{/user}}

{{#assistant}}"Judgement": " {{/assistant}}
{{#assistant}}{{select "duplicated" options=valid_duplicated}}{{/assistant}}

Existing premises: Miroslav Venhoda was a Czech choral conductor who specialized in the performance of Renaissance and Baroque music. Any choral conductor is a musician. Some musicians love music. Miroslav Venhoda published a book in 1946 called Method of Studying Gregorian Chant.

New proposition: If someone is a choral conductor, then he is a musician.
Duplicated: True.

Existing premises: All eels are fish. No fish are plants. A thing is either a plant or animal. Nothing that breathes is paper. All animals breathe. If a sea eel is either an eel or a plant, then a sea eel is an eel or an animal.

New proposition: No eels are plants.
Duplicated: False.

We sincerely thank you for your constructive suggestions and valuable comments! We will try our best to solve your issues and will appreciate very much if you could kindly raise your score if your concerns are addressed.

Q: lacks much necessary information about framework

A: Thanks for your reminding! We have reorganized the structure of Method Section. We add a Problem Formulation Subsection to provide more explanations and revise the notations to make the implementation easier to understand.

Q: how to score relevance and supplement

A: We implement the relevance and supplement scorer by prompting LLMs. We provide prompt templates and an example below for easier understanding.

Prompt:

Please follow these steps:
1.From the determinate premise, select the "Most relevant premise" which has the same subject with "Hypothesis", and give a score from 0 to 1.
2.You need to assess how the "Most relevant premise" relates to all the other "determinate premise" and "indeterminate premise", based on Relevance scoring rules.
3.The "determinate premise" and "indeterminate premise" with scores higher than 0.25 will be used as the final supplement results, along with Most relevant premise.
Relevance scoring rules:
1. When scoring relevance, 0.25 added for each noun or 0.25 added for each adjective that is the same between two sentences.
2. Scores start to accumulate from 0 points, and the upper limit is 1 point.
3. If sentence p1 is a hypothetical premise of sentence p2, then add 0.25 to p2. for example: measure "if A then B." and "A is true." Then add 0.25 points to "if A then B".

E.g.:

Target: bear is nice and red (True/False/Unknown). 
Determinate premises: 
d1) The bear is big. 
d2) Billy is rough. 
d2) The tiger is rough.
Indeterminate premises: 
i1) If bear is big, then bear is red.
i2) If someone is big, then they are nice.

### After premise priority scoring: 
Most relevant premise $p*$: The bear is big.
Other Premises:  
i1) If bear is big, then bear is red.(0.75)
i2) The tiger is rough.(0.0) 
i3) If someone is big, then they are nice.(0.5) 
d2) Billy is rough.(0.0)
Supplementary premises *$\mathbf{p}_s$*:
i1) If bear is big, then bear is red.(0.75)
i3) If someone is big, then they are nice.(0.5)

So the selected premises for exploration: The bear is big. If someone is big, then they are nice. If bear is big, then bear is red.

New proposition $\widehat{p}$: The bear is nice and red.

Conclusion: True.

Q: how to filter out determinate premises

A:

1. Determinate premises are defined as simple statements directly related to the target conclusion, requiring to instruct LLMs to understand the target rather than simply filtering it by word matching. Taking Case B in our paper as an example, the determinate premise is like "Someone is male/female bachelor/PhD degree", the partial solution to the problem.

2. For your given example, the premise If Erin is round, then Gary is quiet (denoted as $i_7$) is identified as indeterminate premises since its "if-then" structure cannot directly derive new determinate facts without interacting with other premises. But different indeterminate premises also have different priorities for deriving conclusions, which is exactly the meaning of supplement scoring in the second part of our premise prioritization. Since the target revolves around "Gary" and "round", $i_7$ may have a higher supplementary score than other premises without "Gary", and is more likely to be used to generate new propositions to close to the target.

Q: how to exploit history reasoning paths information

A: The historical reasoning path is mainly used at the beginning of each reasoning iteration. We use memory extraction to merge the successful and failed reasoning steps of previous iterations.

Successful reasoning step:

In the NO:x round, we use these “premises”: “If something chases the cat then the cat chases the dog. The bald eagle chases the cat.” and got a “New Determinate Premise”: “The cat chases the dog.”

Failed reasoning step:

In the NO:y round, we use this "most relevant premise": "Bonnie is either a student who attends the school and is very engaged with school events, or she is not a student who attends the school and is not engaged with school events." and got a "false Proposition": "Bonnie is either a student who attends the school and is very engaged with school events, or she is not a student who attends the school and is not engaged with school events."

Thank you for your time and efforts in providing the rebuttal. I appreciate the response to the framework details. For the comparison with SI, I cannot see substantial differences. Specifically,

We propose to formulate the reasoning process as the transition from indeterminacy to determinacy

The perspective is new. However, the reasoning process is pretty much similar.

We employ two-stage quantitative measurements (relevance and supplement) for premise selection and exploration

This part of the contribution is acknowledged.

We introduce a reasoning memory module to automate storage and extraction of available premises and reasoning paths

This is also a widely applied trick in many frameworks, such as langchain, babyAGI, etc.

In summary, I stand with my concerns on the incremental novelty of this work.

We sincerely thank you for your constructive suggestions and valuable comments! We will try our best tackle your concerns and will appreciate very much if you could kindly raise your score.

Q: how each module in the proposed framework is implemented, how the scorers are implemented and what is the threshold for supplementary premises filtering

A: Thanks for your reminding!

1. The essential process of the proposed method including premise identification, premise prioritization, verification checking, and reasoning memory are all implemented by prompting LLMs with carefully designed instructions and demonstrations.

2. For premise priority scoring, the prompt templates and examples are included as follows, with the threshold set to 0.25. More prompt templates are available in revised Appendix.

Prompt:

Please follow these steps:
1.From the determinate premise, select the "Most relevant premise" which has the same subject with "Hypothesis", and give a score from 0 to 1.
2.You need to assess how the "Most relevant premise" relates to all the other "determinate premise" and "indeterminate premise", based on Relevance scoring rules.
3.The "determinate premise" and "indeterminate premise" with scores higher than 0.25 will be used as the final supplement results, along with Most relevant premise.
Relevance scoring rules:
1. When scoring relevance, 0.25 added for each noun or 0.25 added for each adjective that is the same between two sentences.
2. Scores start to accumulate from 0 points, and the upper limit is 1 point.
3. If sentence p1 is a hypothetical premise of sentence p2, then add 0.25 to p2. for example: measure "if A then B." and "A is true." Then add 0.25 points to "if A then B".

E.g.:

Target: bear is nice and red (True/False/Unknown). 
Determinate premises: 
d1) The bear is big. 
d2) Billy is rough. 
d2) The tiger is rough.
Indeterminate premises: 
i1) If bear is big, then bear is red.
i2) If someone is big, then they are nice.

### After premise priority scoring: 
Most relevant premise $p*$: The bear is big.
Other Premises:  
i1) If bear is big, then bear is red.(0.75)
i2) The tiger is rough.(0.0) 
i3) If someone is big, then they are nice.(0.5) 
d2) Billy is rough.(0.0)
Supplementary premises *$\mathbf{p}_s$*:
i1) If bear is big, then bear is red.(0.75)
i3) If someone is big, then they are nice.(0.5)

So the selected premises for exploration: The bear is big. If someone is big, then they are nice. If bear is big, then bear is red.

New proposition $\widehat{p}$: The bear is nice and red.

Conclusion: True.

Q: why the proposed method can select a reasoning structure

A: Thanks for your insightful comments! We would like to clarify that we does not focus on pre-defining the reasoning structure, while the structure can be derived by reviewing the connections between the corresponding states of each reasoning iteration. Pre-defining the reasoning structure before solving problems is quite different from the way of human reasoning. Humans actually do not preset a structure when facing a reasoning problem, while the so-called reasoning structure should be formed based on the reviewed reasoning results after solving the problem. Our experiments also show that compared to predefined thinking structures, the complexity of the reasoning structure reviewed by our method is more consistent with the problem difficulty itself.

Q: efficiency of the proposed method compared to the baselines

A: Thanks for your constructive suggestions! We have added complexity analysis for more detailed efficiency comparisons. We choose ToT [1] and CR [2] as baselines and randomly select 100 cases of FOLIO to compute the average inference time for each visited state. The results below show that since our DetermLR can substantially save average states with slight increment on inference time per state, the overall inference efficiency of DetermLR is still better than ToT and CR.

We sincerely thank you for your constructive suggestions and valuable comments! We will answer the questions in detail and will appreciate very much if you could kindly raise your score if your concerns are addressed.

Q: terminology, notations, and in general the explanation of the proposed approach

A: Thanks for your reminding! We have reorganized the method section to make notations clearer and add more explanations in the revision.

Q: results in selected datasets

A: We would like to clarify that the test data used in our comparative experiments is mostly based on the official repositories without any further post-preprocessing [1-3], which is also the most common way of using these datasets. For LogiQA, since it contains different types of examination questions, non-logical reasoning like common sense reasoning is not our focus. In order to accurately evaluate the ability of LLMs to perform logical reasoning using only given conditions without relying on any external knowledge, we carefully reviewed LogiQA, excluded cases with unclear conditional expressions, and finally remained 179 high-quality questions as a curated collection. With our effort of data proofreading, the processed data will be released to the community to facilitate further studies.

Q: The results were reported only on GPT4.

A: Thanks for your valuable comments! We agree that it is indeed necessary to add more base models to further compare the effectiveness of the proposed method. We plan to add GPT-3.5-turbo and LLaMA2 as engines for comparison. Currently we can provide the results of GPT-3.5-turbo on ProofWriter and FOLIO below, showing that all methods perform uniformly worse than GPT-4 (by about 0.1 accuracy). But DetermLR still outperforms all baselines on the same base model. We will complete these results and more discussions about the impact of base models soon in the future revision.

Q: why the term indeterminacy was chosen for some parts of the information

A: Thanks for your insightful comments! We will introduce in detail the necessity of the definitions of "indeterminacy" and "determinancy", and an example of the reasoning process from indeterminacy to determinancy.

1. We introduce indeterminacy and determinancy to distinguish whether the information in a given premise is clear for deriving the conclusion. Only simple statements directly related to the conclusion are treated as determinate premises. So "indeterminacy" can indicate that those premises must be combined with other conditions to deduce new determinate information by removing the indeterminate structures (if-then, or).

2. Taking Case B in our paper as an example, all original given premises are treated as indeterminate premises based on our definition. In this case, we just need to further infer each original premise to obtain potentially determinate information. From "$i_2$: The educational levels of A, B, and C are the same, while those of F and G are different", we know that: there are at least 4 people with education levels of A, B, and C. Then we can derive a determinate premise: "$d_1$: A, B, and C are Bachelors." based on the boundary condition: There are only 3 PhDs among the 7 people. Then more determinate premises can be deduced in the subsequent reasoning process. Therefore, it is consistent with our assumptions about the reasoning process: from indeterminacy to determinacy.

[1] https://allenai.org/data/proofwriter

[2] https://github.com/Yale-LILY/FOLIO

[3] https://github.com/suzgunmirac/BIG-Bench-Hard/blob/main/bbh/logical_deduction_seven_objects.json

{{#system}} Suppose you are one of the greatest AI scientists, logicians and mathematicians. 
Let us think step by step. 
First, read and analyze the following definition:
Determinate premise: The premise contains the same noun or adjective as the Hypothesis, and the premise is not in the structure of ``if...'' or ``if...then...''.
Second, read and analyze the ``Premise'' and ``Hypothesis'' .Judge ``Premise'' is ``determinate premise'' or not.
Third, please make sure your classification decisions are derived directly from definitions, rather than unsourced common sense.
---
{{/system}}
{{~#each examples}}
{{#user}}
---
``Premise'': ``{{this.Premise}}''
``Hypothesis'': ``{{this.Hypothesis}}''
{{/user}}
\raggedright{{#assistant}}``Judgement'':``Is this ''Premise`` a ''determinate premise`` or not?
{{this.usefulness}}'' {{/assistant}}
{{#assistant}}``Explanation'': {{this.Explanation}}{{/assistant}}
{{~/each}}

Input target: bear is rough. (True/False/Unknown) 
Input Premises: 1) The bear is big. 2) Billy is rough. 3) The tiger is rough. 4) If bear is big, then bear is red. 5) If someone is big, then they are nice.

Determinate premises: d1) The bear is big. d2) Billy is rough. d3) The tiger is rough.
Indeterminate premises: i1) If bear is big, then bear is red. i2) If someone is big, then they are nice.

{{#system}}Suppose you are one of the greatest AI scientists, logicians and mathematicians. Let us think step by step. 
Please use the Logical Reasoning Rules(LRR) to determine whether the deduction of the given "Premises" to a "Proposition" is valid or not, reply with True or False.
Logical Reasoning Rules(LRR):
1. "Two premises": "If A,then B. A is true." then "Proposition": "B is true."
2. "Two premises": "If A,then B. If B,then C." then "Proposition": "If A, then C."
3. "Two premises": "If A,then B. B is not true." then "Proposition": "A is not true"
4. "Two premises": "A is either C or D. A is not C." then "Proposition": "A is D."
----{{/system}}
{{~#each examples}}
{{#user}}
---
"Premises": "{{this.premises}}"
"Proposition": "{{this.proposition}}"
{{/user}}

{{#assistant}}"Judgement": "Is this deduction valid? {{this.validation}}"{{/assistant}}
{{~/each}}

{{#user}}
---
"Premises": "{{premises}}"
"Proposition": "{{proposition}}"
{{/user}}

{{#assistant}}"Judgement": "Is this deduction valid? {{/assistant}}
{{#assistant}}{{select "validation" options=valid_validation}}{{/assistant}}

Selected premises: All eels are fish. No fish are plants.
New proposition: No eels are plants.
Validation: True.

Selected premises: Nothing that breathes is paper. All animals breathe.
New proposition: All animals are paper.',
Validation: False.

{{#system}}Suppose you are one of the greatest AI scientists, logicians and mathematicians. Let us think step by step. 
Please determine whether there is a new useful "Proposition". Reply with True or False.
----{{/system}}

{{#user}}
---
"Proposition": "There is no new proposition that can be deduced from the given premises to determine the correctness of the hypothesis."
{{/user}}
{{#assistant}}False{{/assistant}}

{{#user}}
---
"Proposition": "A Czech person wrote a book in 1946."
{{/user}}
{{#assistant}}True{{/assistant}}

{{#user}}
---
"Proposition": "There is no new proposition that can be deduced from the given premises that would be useful in determining the correctness of the given hypothesis."
{{/user}}
{{#assistant}}False{{/assistant}}

{{#user}}
---
"Proposition": "None of the premises provide information to deduce a proposition related to a Czech person writing a book in 1946."
{{/user}}
{{#assistant}}False{{/assistant}}

{{#user}}
---
"Proposition": "{{proposition}}"
{{/user}}
{{#assistant}}{{select "is_something" options=valid_something}}{{/assistant}}

New proposition: No new premises can be derived here.
Validation: False.

New proposition: I cannot derive new premises.
Validation: False.

{{#system}}Suppose you are one of the greatest AI scientists, logicians and mathematicians. Let us think step by step. 
Can this "proposition" can be derived using only one "premise"?Please reply with True or False.
----{{/system}}

{{~#each examples}}
{{#user}}
---
"Premises": "{{this.premises}}"
"Proposition": "{{this.proposition}}"
Can this "proposition" can be derived using only one "premise"?
{{/user}}

{{#assistant}}"Judgement": "{{this.duplicated}}"{{/assistant}}
{{#assistant}}"explanation": "this.explanation"{{/assistant}}
{{~/each}}

{{#user}}
---
"Premises": "{{premises}}"
"Proposition": "{{proposition}}"
Can this "proposition" can be derived using only one "premise"?
{{/user}}

{{#assistant}}"Judgement": " {{/assistant}}
{{#assistant}}{{select "duplicated" options=valid_duplicated}}{{/assistant}}

Existing premises: Miroslav Venhoda was a Czech choral conductor who specialized in the performance of Renaissance and Baroque music. Any choral conductor is a musician. Some musicians love music. Miroslav Venhoda published a book in 1946 called Method of Studying Gregorian Chant.

New proposition: If someone is a choral conductor, then he is a musician.
Duplicated: True.

Existing premises: All eels are fish. No fish are plants. A thing is either a plant or animal. Nothing that breathes is paper. All animals breathe. If a sea eel is either an eel or a plant, then a sea eel is an eel or an animal.

New proposition: No eels are plants.
Duplicated: False.