FLARE: Faithful Logic-Aided Reasoning and Exploration

We thank the reviewer for their feedback on our work and proceed to sequentially address the raised Weaknesses and Questions:

General comments on Weaknesses:

Expensive: Given that the LLMs have been used for planning, code generation and the execution of code via multi-hop search, instead of relying on symbolic solvers, this might increase the cost for computation.

The speed of the external solvers used for baseline methods (F-CoT) is highly dependent and sensitive towards the executing system and choice of the setup that runs it (F-CoT can be run with different versions of Python, PDDL etc). This makes a direct comparison with LLM inference impossible. However, per the reviewer's request, we compare the number of tokens and the inference speed for generating CoT vs F-CoT (without the external solver added time) vs FLARE.

Our results show that the total length of the trace is on average composed of ~250 tokens which corresponds to an approximately 20% increase compared to CoT and F-CoT. Despite this marginal increase, the impact on overall inference speed is negligible w.r.t. to Cot and F-CoT. In particular, the additional tokens are not a reason for a computational overhead as the computational cost of adding a small number of tokens (such as the additional ~250 tokens in FLARE traces) is minimal compared to the overall cost of processing the base sequence. This is specifically true when the sequence lengths remain well below the model's maximum context window.

Weak baselines: Experiments have been performed with relatively simpler datasets. Authors should experiment with more complex datasets that are already present in the literature for question answering such as LogiQA [1], PrOntoQA [2], ProofWriter [3]

As clarified in the paper, we used the benchmarks from Faithful-CoT paper, which are also widely present across the literature. Per the reviewers request, we add a direct comparison with to Logic-LM on the hardest sets of ProntoQA[2], AR-LSAT[3] and Logical Deduction [4]. These datasets are used in the original study of Logic-LM and are considered challenging logic inference benchmarks. We opted not to include PrOntoQA and ProofWriter, as they exhibit considerable overlap with the benchmarks already used in our study (e.g., CLUTTER, AQUA, and StrategyQA). Instead, we prioritised datasets that present unique challenges to demonstrate the efficacy of FLARE better.

Our findings show that FLARE achieves state-of-the-art results on $2$ out of $3$ logic inference benchmarks with $10$ percent increase over CoT and $~7$ percent increase over Logic-LM. These results highlight FLARE’s ability to handle challenging reasoning tasks compared to existing approaches.

I appreciate the detailed rebuttals by the authors but I would like to seek some clarifications.

1. Search Process: Please correct me if I have misunderstood something. Is the search process used in FLARE different from the dfs used in TOT? It is impressive that the overall cost for computation is low because only 1-3 dfs paths are explored but is this scalable? For example in LogicLM or in F-COT and the use of external solvers would mean that incase of problems having high search depths, these methods would require potentially same number of tokens as now because the number of tokens isn't dependent on search depth. For example in real world SAT problems where search depth is high, one could build a scalable system with LogicLM or F-COT but that would be difficult with FLARE. Can you give me a scenario in such cases where it would be prudent to use FLARE instead of F-COT or LogicLM?

2. Additional Experiments: I appreciate the fact that the authors have gone the extra mile to perform experiments on more challenging datasets however here are my concerns. Out of 3 datasets FLARE does well on PrOntoQA by a good margin but it does not do well at all on LogicalDeduction by a good margin too. Performance on AR-LSAT is almost close to LogicLM (since it is only a percentage increase) One thing where I would conced is the fact that LogicLM uses self refinement as does F-COT while FLARE doesn't. I understand the main contribution fo the paper is to provide an interpretable approach but can you in anyway quantify the effect in performance if FLARE used self refinement?

After considerable deliberation, I believe that while the increase in inference cost with greater search depth is indeed a drawback, as the authors have pointed out, correctly formalizing a query in the language supported by a solver remains a non-trivial task too.

Following additional experiments, including the integration of self-refinement into FLARE, I find the proposed method to be a noteworthy solver-free, LLM-only approach. The proposed method uniquely combines code generation and execution entirely within the LLM framework, demonstrating significant performance on both arithmetic and complex logical reasoning datasets. The proposed method remains interpretable while achieving results comparable to solver-based approaches like LogicLM.

Minor Suggestions:

1. Given ICLR's strict 10-page limit, pls move the reproducibility report to the appendix.
2. Pls mention the limitations and future work as it is neccessary.

In light of these considerations, I would recommend the acceptance of this paper.

Rebuttal for Reviewer CeSr We thank the reviewer for their thoughtful and constructive feedback on the paper. Below, we address the raised weaknesses and questions sequentially.

General comments on Weaknesses:

Although the method seems powerful for QA tasks, its adaptability to harder algorithmic and reasoning tasks seem difficult.

The benchmarks within the study are composed of Math Word Problems, Fuzzy, deductive and relational reasoning challenges like StrategyQA and Clutter. Per the reviewers request, we add a direct comparison with to Logic-LM on the hardest sets of ProntoQA[2], AR-LSAT[3] and Logical Deduction [4]. These datasets are used in the original study of Logic-LM and are considered challenging logic inference benchmarks. We opted not to include PrOntoQA and ProofWriter, as they exhibit considerable overlap with the benchmarks already used in our study (e.g., CLUTTER, AQUA, and StrategyQA). Instead, we prioritised datasets that present unique challenges to demonstrate the efficacy of FLARE better.

Our findings show that FLARE achieves state-of-the-art results on $2$ out of $3$ logic inference benchmarks with $10\%$ increase over CoT and $~7\%$ increase over Logic-LM. These results highlight FLARE’s ability to handle challenging reasoning tasks compared to existing approaches.

As logic programming (Prolog) is capable of producing complete algorithms of varying complexities (Anything NP-complete), continuous numbers and nested structures no direct adaptation is needed for more algorithmic tasks.

The authors run experiments with GPT3.5 rather than GPT4o or GPT4o-mini which makes applicability to SoTA models questionable.

The main goal of the research was to introduce an interpretable reasoning methodology that is foundational and model-agnostic. We have explored models of varying scales from 8B to 100B+, showing the consistent improvement that FLARE provides to each of them. Given the architectural similarities, the findings from GPT-3.5 are expected to generalize to more advanced models like GPT-4.

General comments on Questions:

Could FLARE’s framework extend to other tasks that require complex reasoning? Insights on its limitations in such settings would be helpful.

As mentioned priorly in the comments for weaknesses, FLARE is capable to be extended towards any algorithmic task. Some of the benchmarks already included in the paper require Fuzzy, deductive and relational reasoning. As mentioned, we also added more challenging logic inference benchmarks and showed that FLARE remains efficient in these types of setups as well. In general, we find that FLARE yields the most increase for tasks where formalisation is not straightforward, thus there is a requirement for fuzzy/soft reasoning.

The method seems to work more reliably on LLaMa 8b and CmDR rather than on GPT3.5. Is there a reason why?

GPT-3.5 achieves 4 /9 best results in our benchmarks, demonstrating that using FLARE expands the capabilities of the model. Furthermore, our method consistently outperforms CoT and F-CoT in 5/9 benchmarks. Notably, it surpasses F-CoT across all benchmarks.

The framework seems to fail on sports and date Q&A - some insight into why that happens would be useful.

The two benchmarks where CoT somewhat outperforms our method—GSM8k and Sports—are tasks that primarily require straightforward one-hop reasoning or retrieval from a commonsense knowledge base. These tasks do not significantly benefit from the autoformalization and search mechanisms central to our method, which may explain the observed differences.

Is it true that if the model outputs a Prolog code that is not executable (or wrong syntax), it's not possible do any error detection or faithfulness measure?

It is possible to detect underutilizations or hallucinations within the search w.r.t. the written code regardless if it is executable or not. It is also possible to detect if the code is correct and try to refine it if there is an error, however such approaches are rather computationally costly and are not a part of the original study. It is however not possible to measure the faithfulness of the sample without executable code, as the trace generated from the original code, which we compare to the LLM simulation, can only be generated in by running the code.

We hope that the clarifications and the questions addressed in our rebuttal have resolved any concerns, allowing the reviewer to consider increasing the support for the paper. If there are any remaining issues, we welcome further feedback for improvement.

We thank the reviewer for their thought-provoking feedback on the paper and proceed to sequentially address the raised Weaknesses and Questions:

General comments on Weaknesses:

The paper would benefit from more detailed examples to illustrate how the simulated search operates and how it benefits intermediate reasoning steps.

We have updated Section 3 and 5 with more explanations and demonstrations to clarify the simulated search process and the benefits that it offers. We also further refine Figure 1, which showcases an example of simulated search and the hallucination and underutilisation detection process. We also added a raw complete example in the Appendix for observation. We also include all the prompts and solved samples for each dataset within our anonymised repo: https://anonymous.4open.science/r/FLARE-D88B

There is no theoretical analysis of search complexity. If complexity analysis isn’t feasible for certain fuzzy problems, a time cost comparison between FLARE and baseline methods (e.g., CoT and F-CoT) would be a useful alternative.

The speed of the external solvers used for baseline methods (F-CoT) is highly dependent and sensitive towards the executing system and choice of the setup that runs it (F-CoT can be run with different versions of Python, PDDL etc). This makes a direct comparison with LLM inference impossible. However, per the reviewer's request, we compare the number of tokens and the inference speed for generating CoT vs F-CoT (without the external solver added time) vs FLARE.

Our results show that the total length of the trace is on average composed of ~250 tokens which corresponds to an approximately 20% increase compared to CoT and F-CoT. Despite this marginal increase, the impact on overall inference speed is negligible. In particular, the additional tokens are not a reason for a computational overhead as the computational cost of adding a small number of tokens (such as the additional ~250 tokens in FLARE traces) is minimal compared to the overall cost of processing the base sequence. This is specifically true when the sequence lengths remain well below the model's maximum context window.

General comments on Questions:

Considering that DFS can become computationally intensive as search depth increases, and that appending search traces to prompts may risk exceeding the LLMs' context length, how do you manage the generated Prolog program to avoid an unacceptable search space?

As shown above, the overall length of the average plan, code and search remains below 1000 tokens, and with added 4-8 in context samples, would still remain well under the context length limit (128K for most models, 16K for GPT3.5).

We also have provided the overall statistics for the completed search in Table 4 within the paper. Our results show that while the models are capable of performing 11 multi-hop steps within a single DFS branch/path, usually only 1-3 paths are explored for a total of 250 tokens on average. We see a similar distribution from the traces generated from actual code execution. We postulate that most of the queries do not require the depth of the search to be high and also show that models use fuzzy reasoning to skip failing paths and reasoning lines (Tables 4, 5,6), thus truncating the need for exhaustive search.

We thank the reviewer for their feedback and questions and proceed to sequentially address the raised Weaknesses and Questions:

General comments on Weaknesses:

The advantages of using LLMs to simulate Prolog execution are ambiguous.

Our results show that the models simulating logic programming execution yield an average of increase of 23% over CoT and ~60% on F-CoT and yield SOTA results across 7 of the 9 QA tasks that we tested on. We have further upon the reviewers request added 3 logic inference benchmarks ProntoQA, AR-LSAT, LogicalDeduction [1,2,3] on which the model shows an average 16% increase over CoT and 8% increase over Logic-LM on 2 out of 3 benchmarks. We show that the increase in results are due to the presence of those simulated code execution by directly comparing performance with plan-only and code approaches (Section 5.2).

The method also allows to explicitly measure the faithfulness of the final answer w.r.t. the performed multi-hop search. Furthermore, FLARE allows direct detection of hallucinations and underutilised knowledge during the search process, which is a unique feature of a reasoning method allowed through code execution simulation. We are able to confirm this way that the optimal searches exhibit, on average, a $18.1\%$ increase in unique emergent facts, a $8.6\%$ higher overlap between code-defined and execution-trace relations, and a $3.6\%$ reduction in unused code relations. We show that this behaviour is very specific

Some key design and implementation details are missing. Code and data are not provided for reproducibility.

We have added all of the implementation details, all of the used prompts and results in the submitted updated paper, supplementary materials and the code. All of these can also be found in our anonymously shared code repository: https://anonymous.4open.science/r/FLARE-D88B/README.md

Answer to Questions:

It would be helpful to include the notations for equations (Eq. (1)-(4)) and explanations for figures (e.g., Figure 1) to enhance readability and logical flow. Additionally, there are a few specific points that may benefit from clarification. For example, T as a single token in query Q is misused in Eq. (1)-(4). In Eq. (4), there might be a redundant comma or a missing variable.

We have updated the readability of the overall paper, including the notation, equations, their definitions, demonstrations and the overall flow of the paper. (See updated manuscript)

In Section 3.1, the Simulating Search section might benefit from additional details. Specifically, elaborating on how a problem space traversal trace is formed would be valuable. Including a detailed description, pseudocode, or examples could provide further clarity. It would also be helpful to include a detailed explanation of the backtracking mechanism.

We have added a detailed explanation of the whole search process in the section and refined the writing with additional details. We further improved Figure 1 which includes a direct demonstration of the whole search process with backtracking. We have also added a complete example of FLARE in the Appendix of the paper.

The complete process could be further clarified by explaining the steps following inconsistency detection and faithfulness measurement. For example, it would be valuable to know if a best-of-n selection, weighted voting or other strategies are used to decide the final answer. A pseudocode of the entire framework could also enhance understanding here. In addition, it's better to include some full examples for inconsistency detection and faithfulness measurement.

We want to clarify that faithfulness measurement and hallucination detection are done after inference on a benchmark is completed. This means that they do not impact the final answer-generation process. In FLARE, the final answer is generated by directly prompting the LLM as explained in Section 3.1. We further refined that sections 3.2 and 3.3 for a more clear description of the faithfulness measurement and hallucination detection. We also improved figure 1 to explicitly show how the matching for hallucinated and underutilised facts and relations are done for a given example.

The framework may benefit from adaptive capabilities. Currently, the plan generation is done in one pass without a feedback loop. Introducing a mechanism to refine the generated code if errors are detected during subsequent simulation could also improve adaptability.

While self-refinement and other techniques can be applied on top of FLARE it is not a part of the original motivation of the study as the goal was to create an interpretable method that allows for planning, fuzzy reasoning, and traversing the problem space with backtracking, exact task decomposition, and measuring faithfulness.

It must also be noted that the suggested self-refinement loops are highly computationally costly, but must not necessarily improve the over performance of the method, as FLARE does not rely on the correctness of the generated code.

For simulating Prolog execution, the choice to use LLMs over an external solver might benefit from further clarification. Understanding the reasoning behind using an LLM-based simulation instead of an external solver, particularly in terms of handling the self-bias of LLMs, would be insightful.

Directly using external solvers comes with a variety of limitations.

1. Not all natural language queries can be safely and correctly translated into a formal language (Autoformalization [1]), because queries leave off crucial commonsense, domain or deductive information, thus not allowing either to produce an executable code or code that contains expressive enough facts, relation or strategies for reasoning. This is discussed in Sections 5.1 and 5.2.

2. External solvers are also incapable of fuzzy reasoning and reasoning outside of the scope defined within the program, while LLM simulated code executions do not have such a limitation. This means that queries involving contextual ambiguity and requiring deductive or analytical reasoning not bounded by a predefined scope benefit from the use of LLMs for simulated code execution.

Since Prolog is tailored for rule-based logical reasoning, the specific adaptations for solving math word problems need to be clarified. Further implementation details and example outputs could illustrate the achieved improvements and highlight the framework's effectiveness.

Prolog directly supports mathematical operations and is capable of outputting continuous numbers in a program. In this sense no specific adaptation is needed for solving math world problems. We include all the prompts and solved samples for each dataset within our anonymized repo: https://anonymous.4open.science/r/FLARE-D88B

We also refined Section 3 for further clarifications.

It would be valuable to expand the evaluation to include other general methods, such as Tree of Thought, Program of Thought, and Self-Consistency, alongside comparisons to similar methods [1-3]. Additionally, for Llama-3.1-8B, CmDR, and CMDR+, the F-CoT results are zero. Could the authors provide more context on this outcome?

Per the reviewer's request, we add a direct comparison with Logic-LM on the hardest sets of ProntoQA[2], AR-LSAT[3] and Logical Deduction [4]. These datasets are used in the original study of Logic-LM and are considered challenging logic inference benchmarks. We do not use FOLIO and ProofWriter as they are rather similar to the currently existing benchmarks (Clutter and StrategyQA) in our study.

Table: Comparison of Direct Prompting, CoT, Logic-LM, and FLARE.

Our findings show that FLARE achieves state-of-the-art results on $2$ out of $3$ logic inference benchmarks with $10\%$ increase over CoT and $~7\%$ increase over Logic-LM.

As the Tree of Thoughts and Self-Consistency do not offer ways for measuring faithfulness and do not necessarily produce interpretable answers that follow a given search logic over a defined problem space (Facts, relation, search goal) while being extremely computationally costly, comparing to them would be outside of the scope of the current research and would not add any new revelations.

Code and data are not provided for reproducibility.

We provide the complete code and data for the reproduction on the experiments in an anonymised repository: https://anonymous.4open.science/r/FLARE-D88B

We respectfully request the reviewer to kindly reassess the scores and consider increasing their support for the paper unless they have remaining concerns with our work or the rebuttal, in which case we welcome any additional feedback or clarifications.

We would like to address all of the points made by the reviewer and point out several conceptual misunderstandings and missed/incorrect benchmark comparisons in the paper.

Novelty: The proposed framework is purely based on prompting with a straightforward design. The approach of performing reasoning without relying on external tools has been explored in related work [1,2,3].

The goal was to create an interpretable method that allows for planning, fuzzy reasoning, and traversing the problem space with backtracking, exact task decomposition, and measuring faithfulness.

None of the pointed-out studies explore logic program simulation with an LLM and address faithfulness measurement, hallucination and suboptimal reasoning detection. Also, all of these papers use vastly different methodologies and pursue different goals.

Let’s go through each of them:

1. [1] “Large Language Models can Learn Rules” - This paper proposes a straightforward LLM prompting inductive-to-deductive reasoning paradigm which extracts and learns a set of Natural Language rules given the query. The study does not attempt any type of task formalisation into formal logic and does not attempt logic search simulation of any capacity. The core methodologies, goals, and benchmarks of the study are significantly different from those proposed in our study, thus rendering this comparison unfair and uninformative.
2. [2] “Language Models can be Logical Solvers” - LogiPit creates a synthetic dataset of a specific solver (PyKe), which contains the query and the heuristically coded/formed iterative implicit fact enrichment process of the solver (In the paper, they create a hand-coded pipeline to have a specific type of an iterative output for the solver). It must be noted that LogiPit does not formalise the query into a program but rather tries to infer unknown/implicit facts given the context. They further use several conversational turns for this formulation instead of direct few-shot execution. The paper further trains a model on the generated dataset and evaluates it on two logical inference tasks. As no code simulation is present, the evaluated models are explicitly trained on data(instead of few-shot eval) and no presence of faithfulness and hallucination detection we consider it unfair and uninformative to compare the paper to our study.
3. [3] “Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach” - This paper has been submitted to Arxiv on 29 Oct 2024, thus making it impossible to compare to given the Full Paper Submission Deadline of Oct 1. Also the core methodologies, goals and benchmarks of the study are significantly different compared to the ones proposed in our study, thus rendering this comparison unfair and uninformative.

Moreover, numerous LLM-based agentic frameworks for reasoning already exist. Consequently, the novelty of this work appears to be limited.

We respectfully disagree with such an assessment because although LLM-based reasoning frameworks exist, they do not cover solver-free LLM-based search simulation and do not propose methods for measuring faithfulness and detecting hallucinations and sub-optimal reasoning. Consequently, this make the ideas in our paper novel and priorly unexplored.

Clarity: As noted in the initial review, the methodology section remains unclear. The simulating search described in Section 3.1 only covers the generation of a single trace, leaving it unclear how the authors handle path searching and backtracking—arguably the most critical aspects. No detailed pseudo-code is provided to clarify these processes. In contrast, other search-based algorithms, such as Tree of Thought (ToT) and RAP [4], typically include clear pseudo-code in their papers to aid understanding.

We think there might be a minor misunderstanding here. Section 3.1 covers the complete process of search trace generation. A trace includes all of the paths explored by a model, not a single path. A path can either be successful or lead to failure, in which case we backtrack to the closest sub-goal. As outlined in the benefits of Prolog, the solver completes an exhaustive search over the problem space, which includes several paths and backtracking for failed paths. The in-context samples in LLM demonstrations include traces like that (containing several/all paths and backtracking). We have added details to Section 3 for more clarity. Upon the reviewer's request, we have also added a pseud-code for our method in the Appendix A.3

Significance: The paper fails to justify the pure prompting-based design, as it essentially renders the approach into yet another fancy CoT or ToT method that could hallucinate during its reasoning.

We strongly disagree with such a sentiment as our motivations and justifications supported by the results show that the models using FLARE yield an average increase of 23% over CoT and ~60% on F-CoT and yield SOTA results across 7 of the 9 QA tasks that we tested on (original benchmark of Faithful CoT). We have further upon the reviewers request added 3 logic inference benchmarks ProntoQA, AR-LSAT, LogicalDeduction [1,2,3] on which the model shows an average 16% increase over CoT and 8% increase over Logic-LM on 2 out of 3 benchmarks. We show that the increase in results are due to the presence of those simulated code execution by directly comparing performance with plan-only and code approaches (Section 5.2).

The method also allows to explicitly measure the faithfulness of the final answer w.r.t. the performed multi-hop search. Furthermore, FLARE allows direct detection of hallucinations and underutilised knowledge during the search process, which is a unique feature of a reasoning method allowed through code execution simulation. We are able to confirm this way that the optimal searches exhibit, on average, a $18.1\%$ increase in unique emergent facts, a $8.6\%$ higher overlap between code-defined and execution-trace relations, and a $3.6\%$ reduction in unused code relations. This means that FLARE proposes novel interpretable reasoning capabilities to LLMs and query exploration and reasoning optimality through simulated code search.

Additionally, as noted in the initial review, language models (LMs) are generally more effective at handling natural language than symbolic representations. Integrating symbolic representation into the reasoning process of LMs solely through prompting is likely to degrade their performance, which is shown in [8].

We would respectfully disagree with such a statement as the use of formalisation for reasoning (through in-context samples or training) has been widely shown to be beneficial during LLM reasoning ([9,10,11]). We would also like to point out that the study [8] ("Let me speak freely? a study on the impact of format restrictions on performance of large language models.") does not in any form hint that reasoning with symbolic representations causes performance degradation. The study [8] only claims that by generating constrained formats such as JSON, YAML etc. the model might face discrepancies. Neither JSON nor a similar format can be compared to a symbolic representation such as a declarative logic programming code, thus rendering the conclusion/analogy/argument mentioned by the reviewer scientifically dubious.

We respectfully request the reviewer to kindly reassess their scores and consider increasing their support for the paper as all of the mentioned concerns, questions, and misunderstandings have been addressed. If there are remaining questions to our work or the rebuttal, we welcome any additional feedback or clarifications.

[9] Quan, X., Valentino, M., Dennis, L. A., & Freitas, A. (2024). Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving. arXiv preprint arXiv:2405.01379. (EMNLP 2024 Outstanding paper)

[10] Zhou, J. P., Staats, C., Li, W., Szegedy, C., Weinberger, K. Q., & Wu, Y. (2024). Don't Trust: Verify--Grounding LLM Quantitative Reasoning with Autoformalization. arXiv preprint arXiv:2403.18120. (ICLR 2024)

[11] Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., & Ji, H. (2024). Executable code actions elicit better llm agents. arXiv preprint arXiv:2402.01030.

The primary baseline used for comparison is F-CoT [7], but there is a scope mismatch. F-CoT primarily targets logical reasoning tasks (e.g., ProntoQA, ProofWriter, FOLIO, LogicalDeduction, AR-LSAT), whereas the benchmarks in this paper focus on math problems and multi-hop QA. It is unclear whether the authors have carefully adapted F-CoT to these benchmarks, raising concerns about the validity of the comparisons.

We would like to make a clarification here that the “Faithful CoT” that we compare to in the paper is not the [7] (Faithful Logical Reasoning via Symbolic Chain-of-Thought.), which is known/referred to as SymbCoT. Faithful CoT is the fundamental method from the paper “Faithful Chain-of-Thought Reasoning”, which is referred to in the community as Faithful CoT (F-CoT). We had one of the references accidentally misplaced which we have corrected in our revision. We explicitly use all of the benchmarks from the real “Faithful CoT” (“Faithful Chain-of-Thought Reasoning”) study, which does not use logic reasoning benchmarks mentioned by the reviewer. This makes the comparison with Faithful-CoT one-to-one.

Per the reviewers request, we add a direct comparison with to Logic-LM on the hardest sets of ProntoQA[2], AR-LSAT[3] and Logical Deduction [4]. These datasets are used in the original study of Logic-LM and are considered challenging logic inference benchmarks. We opted not to include PrOntoQA and ProofWriter, as they exhibit considerable overlap with the benchmarks already used in our study (e.g., CLUTTER, AQUA, and StrategyQA). Instead, we prioritised datasets that present unique challenges to demonstrate the efficacy of FLARE better.

Our findings show that FLARE achieves state-of-the-art results on $2$ out of $3$ logic inference benchmarks with $10\%$ increase over CoT and $~7\%$ increase over Logic-LM. These results highlight FLARE’s ability to handle challenging reasoning tasks compared to existing approaches.

Comparison to related work: As noted in the initial review, the comparison to related work remains very limited. The proposed method simulates program execution by searching in the problem space, yet it does not include comparisons with classic search algorithms such as Tree of Thought (ToT) and Self-Consistency (SC). Comparing a search-based approach solely with one-pass CoT is insufficient and unfair.

As mentioned above we directly added comparison to Logic-LM another pioneering study for LLM reasoning. As the Tree of Thoughts and Self-Consistency do not offer ways for measuring faithfulness and do not necessarily produce interpretable answers that follow a given search logic over a defined problem space (Facts, relation, search goal) while being extremely computationally costly, comparing to them would be outside of the scope of the current research and would not add any new revelations.

Performance discrepancies: The reported results for Llama-3.1-8B CoT are significantly lower than those presented in the official Llama paper [5] and other studies [6]. For example, its performance on GSM8k is reported as 74.5 in [6] but only 59.2 in this paper; on SVAMP, it is 89.2 in [6] but 58.6 here; and on StrategyQA, it is 68.4 in [6] but 2.9 in this paper. A similar issue is observed with GPT-3.5 CoT. These discrepancies raise concerns about the validity of the evaluation conducted by the authors, leaving the claimed superiority of the proposed method over CoT unclear.

For CoT results we either directly used the results reported in Faithful CoT(“Faithful Chain-of-Thought Reasoning”) (GPT 3.5) or simply reran the benchmarks using their codebase with prompts, in-context examples without any changes made. This makes sure that we have a 1-1 fair comparison with the original study when we compare FLARE with F-CoT. This would mean that as we either directly report the numbers from the F-CoT paper or run their codebase with only a change of a model that the evaluations are completely valid and the improvement over CoT is fairly measured.

In addition, the original Faithful CoT code is designed for GPT-3.5, where it demonstrates high performance. However, the authors directly applied it to other models, such as Llama3.1 and CmDR, without proper adaptation, resulting in 0 accuracy. This comparison is unfair due to the inappropriate experimental setup. I recommend that the authors re-evaluate the results for Faithful CoT after making appropriate adaptations to ensure a fair comparison.

The Faithful CoT paper does not mention explicit limitations of the model use. Our results show the brittlness of code generation with models that are not explicitly tuned for coding or fail to produce code in a strict parsable format. This shows the explicit limitation of approaches like F-CoT and Logic-LM that if the model fails to produce executable or explicitly formatted output than the execution of the generated code becomes impossible and massively degenerates the results. FLARE is not dependent upon such limitations which is a clear advantage for a solver free LLM based method.

We would like to ask for clarifications regarding the proper adaptations mentioned by the reviwer as we find no natural changes that can be made to the F-CoT approach for adapting to other models. Per the reviewer's request, we explicitly tested a change in instructions and the formatting of the in-context samples, yet the F-CoT results still remain consistent. We consider this a fair comparison as we 1-1 produce the results of the F-CoT with the suggested hyperparameters, instructions and in-context samples.

Comparison to related work:
From my understanding, the central claim of the paper is that faithfulness and the incorporation of formal logic enhance reasoning performance. If that is the case, why not compare the proposed method to an approach that excludes these elements (similar to an ablation study)? If all baselines are required to include faithfulness and logic representations, how can the effects of faithfulness and logic representations be effectively investigated?

In addition, the agentic framework typically involves substantial token usage. Could the authors provide a comparison of average token usage/accuracy trade-off between the proposed method, CoT, Faithful CoT, Tree of Thoughts and Self-Consistency?

It must be noted that in order to produce ToT or Self-Consistency results, we must set the decoding temperature to a number bigger than zero, i.e. greedy decoding is not possible (temperature usually >= 1). However, in our study, we used greedy decoding (for CoT, F-CoT and FLARE) for complete reproducibility and fair comparison. Nonetheless, per the reviewer's request, given the tight timeframe, we produced a direct comparison with ToT on two challenging logical inference datasets. We will submit the completed benchmark with ToT for the final version of the paper.

Our results show that using FLARE allows to achieve significant improvements over ToT.

We thank the reviewer for their comments.

The comparison with CoT maybe not entirely fair. FLARE is essentially an agentic framework augmented with formal logic. To accurately assess its effectiveness, it would be more appropriate to compare it against other agentic frameworks without formal logic. The reason is that agentic frameworks are typically better than CoT (particularly for weaker LLMs).

What kind of experiments would the reviewer like to see? Does the reviewer have any references for agentic frameworks we can include that have not already been included in our comparisons?

We followed the evaluation protocols in, e.g. the F-CoT paper (https://arxiv.org/abs/2301.13379; where authors compare with vanilla CoT and another prompting-based baseline) and added more baselines (Logic-LM and ToT per the reviewers request), but we are happy to include more comparisons if that helps the quality of the paper!

We, as mentioned priorly, also include a rigorous analysis of our framework compared to CoT and F-CoT and explicitly highlight the strengths of the proposed method (Section 5.2-5.4).