CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long-Range Reasoning Problems using LLMs

Thank you very much for taking the time to review our manuscript and provide valuable feedback.

W:

We will reply in the Question section.

Q1:

Example: Banerjee S, Agarwal A, Singla S. Llms will always hallucinate, and we need to live with this[J]. arXiv preprint arXiv:2409.05746, 2024.

Q2:

Specifically, it refers to the enhancement achieved through the introduction of ITE.

Q3:

State Transitions refers to the change in the model at different inference steps.

Unidirectional reasoning refers to Monte Carlo-based reasoning methods as opposed to two-way simultaneous reasoning such as DES.

Causal probability trees refers to the probability tree expansion based on simultaneous consideration of node causality.

Q4:

Specifically, our approach is to use ITE to enhance causal significance and consistency.

Q5:

We present concrete algorithms for causal reasoning in discrete probability trees that cover the entire causal hierarchy.

Q6:

Our experimental results are the results of multiple repeated experiments. In the meantime, we will replace guarantee with enhance.

Q7:

A 4-steps reasoning task, for example, is detailed reasoning as opposed to a 12-steps reasoning task.

Q8:

Reasoning. We’re very sorry for the misunderstanding. We will use the same term as' long range reasoning '.

Q9:

Yes, we believe that this accuracy rate demonstrates the ability of the large model to handle multi-step multiplication, which requires multiple steps of reasoning to accomplish.

Q10:

We just want to make it clear that this struggle has been ongoing for a long time in the field of long-range reasoning problems. This paper marks the beginning of this field, and in recent years, LLMs related methods have been working hard to address this issue. Long range reasoning refers to the problem shown in lower part of Figure 1.

Q11:

Refers to the elements which are included: initial state, OSR, state transition, next state, and goal state, as shown in Fig. 2.

Q12:

ATE is the mean value of ITE, and the mathematical nature of the two is the same. Yao L, Chu Z, Li S, et al. A survey on causal inference[J]. ACM Transactions on Knowledge Discovery from Data (TKDD), 2021, 15(5): 1-46.

Q13:

In brief:

E(ITE): Causal Significancy.

Var(ITE):Causal Consistency.

This is because our experiment found that the mean of ITE reflects the significance of the results, while the variance reflects the consistency of the results.

After reading the appendix, you will have a clearer understanding of this.

Q14:

Definition of Variables: Variables Y and W are binary variables, taking values of 0 or 1. A value of 1 indicates correctness, while a value of 0 indicates an incorrect state. These binary variables are used to represent the correctness of the OSR and the state transition, respectively. For example, W can refer to the action of moving a block of objects, such as put down and pick up. Y can refer to the correct position and stacking order of the object block.

The next state refers to the next state in which the stacking order of objects is changed after action/intervention.

Q15:

Probability trees are one of the simplest models of causal generative processes.

You can refer to:

Genewein T, McGrath T, Delétang G, et al. Algorithms for causal reasoning in probability trees[J]. arXiv preprint arXiv:2010.12237, 2020.

Q16:

We will delete “ to summarize”.

Q17:

Thanks, we'll replace ensure with enhance.

Q18:

The experimental results indicate that as the number of steps increases, the inference accuracy decreases. This is limited by the model's own capabilities, pay attention to the comparison between the DES group and the CRE only group. This is certainly a limitation, there is no contradiction.

Q19:

Thank you, we have adjusted the formatting of all citations.

Q20:

Thanks, we have changed it in the revised version.

Q21:

Thanks, we have changed it in the revised version. Also the reference is in line 231 of the original text.

Q22：

https://huggingface.co/docs/transformers/perplexity

Note, however, that this is the PPL definition of the original case, in which we modified to introduce ITE.

Q23:

Thanks. We will revise it in the revised version.

Q24:

We’re afraid not. See L390-L395. For example, stacking order.

Thank you for reviewing the revised version of our manuscript and for your continued constructive comments.

Thank you for your detailed response and for providing further clarification on your perspective. We fully understand your concerns regarding the presentation of the manuscript and the significant effort required to make the paper readable and publishable. 

Given the scale of revisions required, we agree that the discussion phase alone may not be sufficient to address all the issues. We are committed to working on a more thorough revision and will use the time during the waiting period and beyond to further refine the manuscript. We are dedicated to improving the clarity of the presentation, ensuring that the contributions are clearly communicated, and addressing any lingering concerns about causal consistency and empirical validation. 

We also appreciate your recognition of the potential in the high-level idea. Despite the challenges in engaging with the manuscript, we are grateful for the feedback provided, which will guide our efforts in restructuring the paper to improve its readability and overall quality. 

Once again, thank you for your constructive input. We will continue working on this revision and ensure that future versions of the manuscript address the concerns you and the other reviewers have raised.

Thank you very much for taking the time to review our manuscript and provide valuable feedback.

W1:

Can be more general: the method section ties in with the experiment section. For example, the CRE loss function is based on numerical supports, which leaves a gap (need to test both for new cases thus more training cost) for more general applications.

The method section is closely linked to the experiments. We aim to refine the causal enhancement of the large model reasoning process, and actually decouple our method from the experiments, for example in the task of knowledge enhancement you mentioned.

Causality Comments: Here the causal definition seems more like "Improve the possibility of what should happen of an action by minimizing the CRE loss thus we can reduce hallucination", from which I do not see the role of "treatment for control" part, i.e. W = 0. Also, the claims between lines 275 - 278 are incorrect. Given the condition "X and Y are correlated", we cannot simultaneously have "Y can be predicted using X" and "intervening in X would not lead to any changes in the distribution of Y".

We statistically calculate ITE from the model test outputs for each epoch of the model and adjust the inputs to the model for the next epoch of training, measuring the values at the model output.

W2:

1. Clarification on the Role of $W = 0$:
   The role of $W = 0$ (representing the absence of OSR correctness) is critical in establishing the treatment-control framework within our causality analysis. When $W = 0$, it reflects scenarios where the OSR output is incorrect, serving as the control condition in our intervention framework. By comparing the outcomes of $Y$ under $W = 1$ (treatment) and $W = 0$ (control), we quantify the causal effect of OSR correctness on state transitions. This is encapsulated in the ITE (Individual Treatment Effect), which forms the basis of the CRE loss. The minimization of the CRE loss ensures that the causal relationship between OSR and state transitions is strengthened, thereby reducing hallucinations.

2. Claims Between Lines 275-278:
   We agree that the statement as presented could lead to confusion. Let us clarify:

   • If $X$ and $Y$ are correlated, it is indeed possible to predict $Y$ using $X$.
   • However, correlation alone does not imply causation. When $X$ and $Y$ are only correlated but not causally linked, intervening on $X$ (e.g., manipulating $X$) would not alter the distribution of $$Y$).
     For example: People are running during earthquakes, which is what the sample can see, so we think there is a correlation between earthquakes and running. But what the sample cannot see are people who did not run away from the earthquake because they could have been buried, so the causality of earthquakes and running does not necessarily hold. These two observations are consistent and do not contradict one another. To improve clarity, we will revise this section to explicitly state that while correlation allows for prediction, it does not guarantee that interventions will have an effect unless a causal link exists.

3. Improving Causal Definitions:
   To address your concerns about the causality definition, we will refine the discussion to better emphasize the causal reasoning framework. The CRE loss explicitly embeds causality by reducing the ITE variance ($(\text{Var(ITE)}$) and the expectation of absolute ITE ($|E(\text{ITE})|$$). This minimizes the occurrence of samples where OSR and state transitions are non-causally related. The causality is thus operationalized not just as "improving the possibility of what should happen," but as establishing and stabilizing a causal alignment between OSR correctness and state transitions.

Q1:

Of course. Our prompt is the same pattern. The accuracy of reasoning results.

Q2:

State refers to the stacking state of all object blocks at a given step.

Treatment can be an alternative to OSR, or intervention, referring to the action taken by the LLM.

The treatment effect refers to the NEXT state, i.e., the new stacking order of the object blocks after the LLM action.

Sorry for the confusion caused. As shown in the figure, we have multiple alternative paths (interventions) and generated corresponding inference results (treatment effects).

Q3:

Considering the different dimensions of these two variables, we have roughly determined the initial values based on preliminary experimental data. Then continuously try different values during the training process, observe the changes in training effectiveness under different combinations of values, and fit them to a suitable value.

Thank you for reviewing the revised version of our manuscript and for your continued constructive comments.

Thank you very much for taking the time to review our manuscript and provide valuable feedback.

W1:

Please see Appendix A.6.

W2:

We think we have selected the best performing current benchmarks for comparison, these data already represent the results and we will refer to your suggestions for future work.

W3:

Our experimental data is already the average of multiple repeated experiments. After our experiments, the upper and lower errors do not exceed 5%. The reasons for this error include seed values, graphics card models, and the values of other hyperparameters.

W4:

CRE is a method for training models, DES is a search method, both can work independently on their own, and when CRE and DES are used together, the two together form CreDes.

Q1:

We intervene to change the inputs to the model and observe the changes in the outputs, from which we calculate the OSR and the ITE prior to the next state, and use them in the loss function to train the model.

Q2:

Yes. Similar problems to the model presented in this article are theoretically feasible, but we have not conducted any relevant experiments yet. Knowledge-related work requires the use of reasoning, and we are reasoning rigor-bound work, so we can easily migrate applications.

Q3:

We believe that this may be the case, but our current experimental results do not reveal it. However, our validity determination is based on the PDDL, so this scenario would be considered correct.

Thank you for reviewing the revised version of our manuscript and for your continued constructive comments.

Thank you very much for taking the time to review our manuscript and provide valuable feedback.

W1: Due to the length limitation of the main text, we have placed some background information explanations in the appendix section, which you can refer to. If necessary, we will post the experimental results in the revised version and swap them with the theoretical section in the appendix.

W2: We statistically calculate ITE from the model test outputs for each epoch of the model and adjust the inputs to the model for the next epoch of training, measuring the values at the model output.

W3: We provide concise examples in Figure 1, Figure 4, and Equation 1. Sorry for the inconvenience caused. We provide further examples of tree structures in the appendix section .

W4: The direction in which the tree unfolds depends on the distance D. Generally, we calculate the distance between any two nodes by calculating the positional difference of each block in these two nodes, that is, calculating the Euclidean distance of all block positional differences.

W5: See Answer4 and L307. The distance is calculated as the sum of the Euclidean distances between the current position of each object block and its target position.

W6: Because not using DES is sufficient to achieve a high level of accuracy, the addition of DES only increases computational complexity. The purpose of introducing DES is to simplify complex problems.

Thank you for reviewing the revised version of our manuscript and for your continued constructive comments.

Thank you very much for taking the time to review our manuscript and provide valuable feedback.

W1:

Definition of Variables: Variables Y and W are binary variables, taking values of 0 or 1. A value of 1 indicates correctness, while a value of 0 indicates an incorrect state. These binary variables are used to represent the correctness of the OSR and the state transition, respectively. For example, W can refer to the action of moving a block of objects, such as put down and pick up. Y can refer to the correct position and stacking order of the object block.

ITE Calculation: The ITE (Individual Treatment Effect) is derived from the confusion matrix, and its purpose is to measure the causal relationship between Y (state transition correctness) and W (OSR correctness). This calculation is detailed in Appendix A.8 of the paper, which explains how the elements of the confusion matrix are used to compute ITE. The ITE is introduced into the loss function to strengthen the causal connection between the OSR and the state transition.

Context of ITE Values: Since ITE is based on probabilities derived from the confusion matrix, it cannot exceed 1 in certain cases, reflecting strong causal effects. This is consistent with its theoretical definition and statistical properties, as outlined in the appendix.

W2:

We adjusted the definition of the PPL calculation for the usual case by adding the expectation and variance of the ITE, a function that will measure and adjust the training inputs for the next epoch after the model outputs its results in order to achieve the effect of the constraint.

Regarding $D_{\text{CRE}} = \ln(\text{PPL})$ in Equation 3: Generally, $\ln(\text{PPL})$ is calculated solely based on the cross-entropy loss. However, in our approach, we extend the calculation to include the ITE (Individual Treatment Effect), combining it with the cross-entropy loss to form a new loss function. This modified loss function is then used to compute the perplexity $($\text{PPL}$)$. The inclusion of ITE aims to enhance the causal alignment between OSR and state transitions, making $\text{PPL}$ not just a measure of prediction quality but also a reflection of causal consistency. We will update the text to make this modification more explicit and ensure it aligns with the rest of the formulation.

Clarification on "Coordinates of Node $i$" in Equation 4: The "coordinates of node $i$" refers to the same $i$ as in $N_i$ in the subsequent text, ensuring consistency. Both $i$ and $N_i$ represent the same node in the reasoning path. We will make this explicit in the paper to eliminate ambiguity and ensure that readers clearly understand the relationship between these terms.

W3:

Focus on Comparison with Prior Work in This Domain: We prioritize comparisons with prior work specifically within this research domain. This ensures that the evaluations remain relevant and meaningful, as our approach directly builds upon and extends existing methodologies in this area. Our results demonstrate a significant performance improvement through targeted training techniques, establishing the efficacy of our contributions.

Distinctive Training Approach Beyond Fine-Tuning: Our method involves a comprehensive training process where no parameters are frozen, distinguishing it from typical fine-tuning approaches. While fine-tuning methods are highly diverse, our approach systematically optimizes the entire model to enhance causal reasoning and reduce hallucinations. This is a deliberate design choice to achieve significant gains in model performance, as reflected in our experiments.

The basis of our method is obtained by changing the loss function by the fine-tune method, in order to address your concerns, we removed the additions in the loss function and supplemented the ablation experiments, considering the time-consuming nature, we only obtained a portion of the test data, and the complete data will be continuously added to the appendix in the subsequent versions of the article. Currently, based on LLama-2-7B, the success rates for 2-steps to 6-steps under the blocksworld dataset are 83%, 74% and 44%, respectively. A sharp decrease is evident for longer tasks.

Q:

The explanation has already been provided earlier.

Thank you for reviewing the revised version of our manuscript and for your continued constructive comments.