Reasoning Elicitation in Language Models via Counterfactual Feedback

Thank you for your thoughtful comments and suggestions! We give answers to each in turn, as well as pointing out corresponding updates we have made to the paper (changes are highlighted in blue in the revised paper).

(1) Non-binary Causes and Effects

The math benchmarking domain (detailed in Appendix C.4) is a good example of how non-binary causes and effects can be handled within our formulation. In this domain, we consider a particular math problem within the GSM8K dataset, which is about Carla downloading a file with a certain size (denote it with $N_X$) and how long it would take to download this file in minutes (denote it with $N_Y$). Note that both of these variables would be non-binary. As an effect, we consider whether the download takes longer than 120 minutes, that is $Y=1$ if $N\_Y>120$ and $Y=0$ otherwise. As an intervention, we consider what would have happened if Carla was downloading a file that is twice the size. For this, we defined $X$ such that $X=1$ if the intervention has occurred and $X=0$ if it has not; and the post-intervention file size is given by $N'\_X = X\cdot (2 N\_X) + (1 - X) \cdot N_X$.

Generally, when interested in a non-binary outcome $N_Y$, one can consider binary events dependent on that outcome instead. Similarly, when interested in interventions on a non-binary target $N_X$, one can decide on a particular intervention and let the cause $X$ be the binary indicator of whether that intervention has occurred or not. We mainly focused on binary causes and effects because the concepts of necessity and sufficiency are the most meaningful for binary variables. However, it is worth mentioning there is also work that aims to extend these concepts to categorical variables (Li & Pearl, 2022, https://arxiv.org/abs/2208.09568).

Update: We have now included this discussion in the newly added Appendix H.1.

(2) Organization of the Paper

Update: We have now moved the related work to the end of the paper, just before the conclusion. A more direct transition from the description of our algorithms to the experiments can be easier to follow. We thank the reviewer for this suggestion! We have also fixed the typo mentioned in the review.

I am happy with the answers to my questions and will keep my score.

Thank you for your thoughtful comments and suggestions! We give answers to each in turn, as well as pointing out corresponding updates we have made to the paper (changes are highlighted in blue in the revised paper).

(1) Formal Representations and Dedicated Solvers

While the reasoning problems we consider are straightforward to solve once translated into a formal representation (like the structural equations in Appendix C), extracting such formal representations from the natural language descriptions of the problems remains a significant bottleneck for performance. Off-the-shelf language models are not effective at this task. For instance, OpenMathInstruct-1 (https://arxiv.org/pdf/2402.10176) achieves an accuracy of 84.6% on GSM8K (competitive but still far from 100% accuracy) by generating “code-interpreter” solutions, which list the computations performed to arrive at the final answer in a formal language. However, this required fine-tuning CodeLlama-70B to generate responses in the style of these code-interpreter solutions using a custom dataset with 18 million problem-to-code examples.

In short, extracting formal representations (like code-interpreter solutions) could indeed enable exact computation of answers thereafter but fine-tuning a language model to accurately extract such representations is likely a more complex task than fine-tuning the model to provide direct answers. This challenge is especially pronounced for small language models like Phi-3-Mini, which has 3.8B parameters compared to the 70B of OpenMathInstruct-1. Given all this, our approach is well motivated, particularly within the domain of small language models.

Additionally, our setting poses an even further complication for the formal-representation route: The reasoning problems we consider are not simple forward computations as in math problems but rather involve causal systems with multiple possible interventions, each intervention altering the forward computation necessary to obtain the final answer.

Update: We have now included this discussion in a newly added Appendix D.2.

(2) Originality of the Reasoning Problems Studied

With the exception of the math benchmarking domain, which is based on the GSM8K dataset, all of the reasoning problems in our paper are original contributions. Even the math benchmarking domain required significant modifications to the original GSM8K problems to adapt them to a causal framework. Specifically:

1. The logic problems in Figures 5 and 6 were purpose-built to explore the different generalization modes we define in Section 2.

2. The healthcare domain is based on a real-life breast cancer treatment guideline. We converted this guideline into a causal model using statistics gathered from various medical journals (details provided in Appendix C.2) and crafted prompts that align with this causal model (also detailed in Appendix C.2).

3. The engineering domain follows a similar process: It is based on a real-life algorithm for fault detection in transmission lines. We converted this algorithm into a causal model and created the associated prompts (details in Appendix C.3).

4. Lastly, the math benchmarking environment is based on a particular problem within the GSM8K database. However, the original problem does not involve any interventions. We adapted the text of the problem to allow some of its story elements to be intervened on (sample prompts can be found in Appendix C.4).

We are planning to release all these problems along with our code upon acceptance.

(3) Relationship between Reasoning and Generalization

Generalizing from a few examples to a larger input space, potentially involving never-seen-before inputs as in the case of 2-4 digit multiplication to 5-7 digit multiplication, is a useful and non-trivial property for a language model to have. This type of generalization remains a challenge within what we term the ‘in-domain’ problem. In fact, the results in Figure 5 show how DPO+CCF outperforms other baselines in this regard.

While there are many notions of generalization, what we argue is that reasoning fundamentally involves breaking a problem down to its basic components and synthesizing them in novel ways that enable generalization to entirely new problems. In a causal graph, this may correspond to an understanding of individual edge relations in a way that would enable one to make estimations for any composite relationship. This is the type of generalization we are interested in. Here, sample efficiency in learning to solve a fixed problem would not necessarily be indicative of reasoning ability.

Update: We have now clarified that our focus is on a specific type of generalization that aims to isolate reasoning, rather than alternative notions of generalization such as sample efficiency (Footnote 3 and the newly added Appendix E in the revised version).

Results for Engineering

Scenario:

The type of fault on a transmission line is determined through three factors X, Y, and Z. These factors are 'close to zero' if they are less than 0.1. (1) If only one of the factors is close to zero, it is a line-to-line fault. When there is a line-to-line fault, it is BC fault if factor X is close to zero, AC fault if factor Y is close to zero, and AB fault if factor Z is close to zero. (2) If exactly two of the factors are close to zero, it is a line-to-ground fault. When there is a line-to-ground fault, it is AG fault if factors Y and Z are both close to zero, BG fault if factors X and Z are both close to zero, and CG fault if factors X and Y are both close to zero.

Response of Phi-3-Mini:

XC = (X < 0.1) and (Y >= 0.1) and (Z >= 0.1)
YC = (X >= 0.1) and (Y < 0.1) and (Z >= 0.1)
ZC = (X >= 0.1) and (Y >= 0.1) and (Z < 0.1)
BC_fault = XC
AC_fault = YC
AB_fault = ZC
YZ_close = (Y < 0.1) and (Z < 0.1)
XZ_close = (X < 0.1) and (Z < 0.1)
XY_close = (X < 0.1) and (Y < 0.1)
AG_fault = YZ_close
BG_fault = XZ_close
CG_fault = XY_close

This response is accurate except that it marginalizes out the general fault types, namely line-to-line and line-to-ground. While it is possible to compute factual values for specific fault types using these equations, the same is not true for the general fault types. Additionally, this limitation prevents us from performing any interventions on the general fault types.

Results for Math Benchmarking (A Modified GSM8K Problem)

Scenario:

Carla is downloading a 250 GB file. Normally she can download 2 GB/minute, but in 100 minutes, Windows will force a restart to install updates, which takes 20 minutes. After the restart, Carla can resume her download. Consider how long it will take to download the file.

Response of Phi-3-Mini:

NA: The amount of data Carla downloads before the restart
NR: The amount of data Carla downloads after the restart
T: Total time to download the file
NA = 2 GB/minute * 100 minutes
NR = (250 GB - NA) / 2 GB/minute
T = 100 minutes + 20 minutes + NR/2 minutes

This response underlines the additional challenges involved in extracting causal equations. First of all, the calculations provided in the response are not entirely accurate, even for the specific instance of the scenario presented. Setting aside these small inaccuracies, the response assumes that a restart will necessarily occur. With these equations, it is not possible to reason about outcomes under interventions such as “What if Windows had not forced a restart?” or “What if Carla had been downloading a file half the size?” — both of which are types of interventions considered in our experiments.

Even if we were to modify these equations to explicitly include whether a restart occurs or not as a variable, it may not always be possible to anticipate what other kinds of interventions will be queried. For instance, these equations also fail to account for potential interventions on the download speed.

We sincerely thank you for engaging deeply with both our work and our rebuttal, and for highlighting this alternative approach to eliciting reasoning. While the equations extracted using our one-shot prompt may not be sufficient to support causal reasoning, the result for the Engineering domain was promising. Certainly, further research could explore improving the accuracy of these equations through prompt engineering or fine-tuning.

Most importantly, these responses definitely helped us better understand the challenges associated with general causal reasoning, especially the response for Math Benchmarking. Accordingly, we will include these interactions with Phi-3-Mini as part of the discussion in Appendix D.2.

The “Math Benchmarking” domain in our paper is based on the GSM8K dataset, which is why we referenced OpenMathInstruct-1 in our response; their results are relevant to our setting.

That being said, your suggestion to evaluate our model of interest, Phi-3-Mini, directly on our own reasoning problems in terms of its ability to extract formal representations is an excellent one. For this purpose, we designed the following one-shot prompt using the problem in Figure 5 as an example. We then used this prompt to extract structural equations for our other problem domains: Healthcare, Engineering, and Math Benchmarking. The results are detailed further below.

Prompt:

I want you to generate structural equations from a causal scenario described in English. Respond only with the structural equations. Here is an example:

Scenario:
Anna, Bill, Cory, and Dave are going to a party, where the host is going to distribute candies. Anna will be happy if she gets at least 4 candies. Bill will be happy if he gets at least 6 candies. Cory will be happy if Anna and Bill are both happy or if he gets at least 8 candies. Dave will be happy if Anna and Bill are both happy or if he gets at least 10 candies.

Structural Equations:
NA: The number of candies Anna gets
NB: The number of candies Bill gets
NC: The number of candies Cory gets
ND: The number of candies Dave gets
HA: Happiness of Anna
HB: Happiness of Bill
HC: Happiness of Cory
HD: Happiness of Dave
HA = (NA >= 4)
HB = (NB >= 6)
HC = (HA and HB) or (NC >= 8)
HD = (HA and HB) or (ND >= 10)

Scenario:
{Test Scenario}

Structural Equations:

Results for Healthcare

Scenario:

There are four types of breast cancer patients (based on their ERPR and HER2 indicators): (1) If a patient is ERPR positive and HER2 negative, they are 'Luminal A'. All luminal A patients should undergo surgery. (2) If a patient is ERPR positive and HER2 positive, they are 'Luminal B'. Luminal B patients should undergo surgery if their tumor is smaller than 1 cm and there is no nodal involvement. Luminal B patients should undergo therapy if their tumor is larger than 1 cm or if there is nodal involvement. (3) If a patient is ERPR negative and HER2 positive, they are 'Enriched'. Enriched patients should undergo surgery if their tumor is smaller than 1 cm and there is no nodal involvement. Enriched patients should undergo therapy only if their tumor is larger than 1 cm (even if there is nodal involvement). (4) If a patient is ERPR negative and HER2 negative, they are 'Basal'. Basal patients should undergo surgery if their tumor is smaller than 1 cm and there is no nodal involvement. Basal patients should undergo therapy only if their tumor is larger than 1 cm (even if there is nodal involvement).

Response of Phi-3-Mini:

NA: ERPR status of the patient
NH: HER2 status of the patient
TA: Tumor size of the patient
TN: Nodal involvement of the patient
SA: Surgery for the patient
TH: Therapy for the patient
LA: Luminal A patient
LB: Luminal B patient
EN: Enriched patient
B: Basal patient
SA = (NA = 'ERPR positive' and NH = 'HER2 negative' and TA < 1 cm and TN = 'No nodal involvement')
LB = (NA = 'ERPR positive' and NH = 'HER2 positive' and (TA < 1 cm and TN = 'No nodal involvement' or TA > 1 cm or TN = 'Nodal involvement'))
EN = (NA = 'ERPR negative' and NH = 'HER2 positive' and (TA < 1 cm and TN = 'No nodal involvement' or TA > 1 cm))
B = (NA = 'ERPR negative' and NH = 'HER2 negative' and (TA < 1 cm and TN = 'No nodal involvement' or TA > 1 cm))
LA = (NA = 'ERPR positive' and NH = 'HER2 negative')
LB = (NA = 'ERPR positive' and NH = 'HER2 positive')
EN = (NA = 'ERPR negative' and NH = 'HER2 positive')
B = (NA = 'ERPR negative' and NH = 'HER2 negative')
SA = (LA or LB or EN or B)
TH = (not SA)

Beyond whether it is accurate or not, this response lacks a clear formal interpretation, as many of the variables are defined multiple times (causing ambiguity).

(5) Relationship to Chain-of-Thought Prompting

Responses of a language model can be improved in different ways, two of which are fine-tuning and prompt engineering. When eliciting reasoning, our focus has been fine-tuning while chain-of-thought prompting is a form of prompt engineering. These two approaches mainly differ in terms of what point they intervene on: Prompt engineering modifies the inputs passed to the language model whereas fine-tuning modifies the language model itself. Importantly, this means that the use of these two techniques are largely orthogonal to each other and definitely not mutually exclusive. In our case, chain-of-thought prompting is not a competing method against ours but rather a separate avenue for further improvement that could be pursued in conjunction with fine-tuning. Hence why we did not consider it as a baseline in our experiments.

Given this context, the purpose of our footnote was to highlight that chain-of-thought prompting is subject to the same evaluation challenge that our fine-tuning approach faces: Examples provided in a chain-of-thought prompt might lead to more accurate answers for “in-domain” questions, however, this does not mean the examples was successful in eliciting reasoning unless the accuracy generalizes to new problems (that require forming novel chains using the same concepts).

Update: We have now included this more detailed discussion in a newly added Appendix D.1. We have also clarified the corresponding footnote.

Thank you for your thoughtful comments and suggestions! We give answers to each in turn, as well as pointing out corresponding updates we have made to the paper (changes are highlighted in blue in the revised paper).

(1) Choice of Phi-3-Mini

We chose a small language model like Phi-3-Mini, with limited reasoning capabilities, because it offers significant room for improvement — making it ideal for demonstrating the effectiveness of our methodology as well as highlighting the differences between various approaches more clearly.

(2) Architecture and Reasoning Performance of Phi-3-Mini

In all our experiments, we specifically used the version of Phi-3-Mini available on https://huggingface.co/microsoft/Phi-3-mini-128k-instruct. This version has a context length of 128k and 3.8B parameters, and has been fine-tuned for instruction following. Below is a summary of its performance on various reasoning benchmarks compared to other language models (copied from the link above). Overall, Phi-3-Mini demonstrates competitive performance.

Update: We have now included this information in Appendix C.

(3) Intuitive Explanation of Inconsistency Rates

An alternative interpretation of the inconsistency rates we introduce is to frame reasoning about necessity or sufficiency as a classification problem. For instance, focusing on necessity, each context variable $U$, along with the cause $X$ and the potential effect $Y,Y_{X’}$ induced by $U$, needs to be assigned to one of three classes:

1. $\mathbb{N}$: The cause and the effect both occurred together, and the cause was necessary for the effect to occur, meaning the effect would not have occurred otherwise if the cause was prevented. In a case like this, one might say “For the observed effect, the cause was a necessary condition.”

2. $\mathbb{N}’$: The cause and the effect both occurred together, but the cause was not necessary for the effect to occur, meaning the effect would have occurred anyways even if the cause was prevented. In a case like this, one might say “For the observed effect, the cause was not a necessary condition.”

3. $\emptyset$: It is not meaningful to talk about necessity as either the cause or the effect did not occur. Note that the previous cases take the occurrence of the cause and the effect as a given, and make statements about what would have happened to the effect if the cause were to be prevented. This is consistent with Pearl’s definition of PN as a conditional probability with the condition being $X=x$ and $Y=y$.

According to the underlying causal model, each context has a corresponding ground-truth “label” as defined in Equation 6. Given a language model, its answers provide estimates for the unknown potential outcomes, which in turn imply a prediction of this ground-truth “label”. The necessity inconsistency rate (N-IR) is the error rate for this classification task (how often the predicted label differs from the ground-truth label).

Update: We have now included this alternative explanation of the inconsistency rate metrics in the newly added Appendix F.

(4) Detailed Analysis of Generalization Results

Update: We have now included a more detailed analysis of the generalization results in the newly added Appendix G. There, we not only expand upon the findings presented in the main paper (in “Results”, Section 6.2, now 5.2 in the revised version) but also highlight additional observations. In particular, we explain why cause-based deduction, under no direct effect (“NDE”), results in larger improvement in S-IR while effect-based deduction leads to larger improvement in N-IR. In short, this is a consequence of $A$ being sufficient for $B$ and $B$ being sufficient for $C$ in our constructed example, illustrating how the underlying problem structure can influence the effectiveness of different modes of reasoning generalization.

Thank you for your thoughtful comments and suggestions! We give answers to each in turn, as well as pointing out corresponding updates we have made to the paper (changes are highlighted in blue in the revised paper).

(1) Balance of Training Data between Different Methods

For the results in Figure 5, OnlyF and OnlyCF has the same number of training examples as F&CF. We ensure this by sampling twice the number of context variables $U$ for OnlyF and OnlyCF compared with F&CF. In other words, F&CF has access to 100 context samples with two question-answer examples for each context sample (one factual, one counterfactual). Meanwhile, OnlyF and OnlyCF have access to 200 context samples with a single question-answer example for each context (either factual or counterfactual). So, the experimental setup for Figure 5 already matches the suggested setup in the review.

Update: We now mention this detail regarding the experimental setup at the end of “Experimental Setup” in Section 6.1 (now 5.1 in the revised version) as well as in Appendix C.

However, the experimental setup for the results in Table 1/2 is different. There, we sampled the same number of context variables for each method. This means F&CF has more question-answer examples in its training dataset, but crucially, each method still sees the same number of problem instances. For example, in the healthcare domain, OnlyF and F&CF sees the same set of breast cancer patients (each patient corresponding to a context); the training data for F&CF does not include any new patients; and the additional CF feedback is provided only for the existing patients. Therefore, the comparison between OnlyF and F&CF remains fair. Note that this would not have been the case if the context variables for the factual dataset and the counterfactual dataset were sampled independently (which is not the case in our experiments).

In Figure 5, we preferred the setup with the same number of question-answer examples across methods to be absolutely certain that any improvement we see is due to the type of feedback being provided (like the reviewer has rightly raised as a concern). In Table 1/2, we preferred the setup with the same number of context variables across methods because this is likely to be the case in practice (for instance, if we apply these methods in a healthcare setting, the number of patients would be constant for each method).

Update: We have now included this discussion regarding the two different approaches one could take when benchmarking OnlyF and F&CF in Appendix C.

(4) Previous Work and Novelty

The related work mentioned in the review aims to evaluate reasoning capabilities of language models. Meanwhile, our primary goal is to elicit reasoning, that is to tune language models to directly optimize their reasoning ability. As emphasized in our related work, this is the novel aspect of our work. Achieving this goal first requires us to define a precise optimization target. We do this by defining various inconsistency rates and generalization modes. On top of these contributions, we also propose novel algorithms for generating fine-tuning data that targets the metrics we formulate.

We disagree that the conclusion of our paper can be reduced to “more data can improve reasoning.” In fact, we keep the size of the available training data constant in all our experiments. Instead, we investigate the type of data provided (factual vs. counterfactual) and the form in which it is provided (independently as in F&CF vs. pairwise as in CCF). Our conclusion is that having both factual and counterfactual examples during fine-tuning is essential, and that pairing factual and counterfactual examples related through a shared context can lead to further improvements.

Update: We have now included Wu et al. (2023) and Frohberg & Binder (2022) as examples of reasoning evaluation in our related work. We have also clarified our conclusion at the end of the paper.

(5) Details of Real-World Problems

Details regarding the real-world problems in Section 6.3 are provided in Appendix C. We would like to clarify two points: (i) Although the problems in Section 6.3 are based on real-world processes and statistics, they are still fully simulated environments and do not involve the use of real-world datasets. (ii) Generalization modes that can be tested for each problem are fundamentally restricted by the causal graph underlying the problem. For instance, if a problem does not feature two variables colliding, then it would not be possible to test common-cause and common-effect generalization for that problem. All the generalization modes we tested for each problem domain are listed in the “Scenario” column of Table 2 (which is a detailed breakdown of Table 1 in Section 6.3).

Update: We have now added Table 3, which summarizes all of our problem settings, available generalization modes for each setting, and the number of context variables sampled for each generalization mode in one place. Thank you for this suggestion!

(2) Detailed Analysis of the Results

We would like to highlight that our analysis already covers different failure modes of fine-tuning for reasoning, in particular it forms connections between these failures and error types as well as problem structures. We emphasize two examples:

1. In Figure 6a, we see that common-effect generalization leads to an asymmetric improvement in N-IR and S-IR. This is because, in common-effect generalization, the effect of interest is fixed between training and testing, hence the factual questions regarding this fixed effect remain the same as well (factual questions do not depend on the cause as there are no interventions involved). Meanwhile, the cause that is being intervened on changes, hence the fine-tuned model now needs to answer a different set of counterfactual questions during test time than the ones seen during training time. Therefore, we would expect common-effect generalization to be more challenging in terms of CF-ER as opposed to F-ER. Since in our constructed examples, causes tend to be sufficient for effects rather than necessary, performing well in terms of S-iR is more dependent on counterfactual question answering while N-IR is more dependent on factual question answering. Most contexts can be dismissed as irrelevant as far as necessity is concerned (i.e. $\mathcal{N}=\emptyset$) by estimating the factual outcome to be negative, whereas determining sufficiency requires determining whether the counterfactual outcome is positive or it is also negative. Therefore, we see a greater improvement in terms of N-IR.

2. Across all generalization modes presented in Figure 6, by far the most challenging one is cause-based deduction with direct effect (“WDE”), where no method strictly improves the base model. This is due to the problem structure: When $A$ affects $C$ both directly as well as indirectly through $B$, it is not possible to separate the two types of effects without seeing any interventions on the intermediary $B$. Notice how this stops being an issue for cause-based deduction with no direct effect (“NDE”). Interestingly, we still see an improvement in terms of N-IR (at the cost of worse performance in S-IR). This is similar to the previous example: Seeing demonstrations for $A\to C$ at least improves the model's ability to answer factual questions regarding $C$, which translates to better F-ER when reasoning about $B\to C$, which in turn translates to better N-IR as before ($B$ is sufficient for $C$ hence most contexts can readily be dismissed as irrelevant as far as necessity is concerned just by estimating the factual outcome correctly).

Update: We have now added a more detailed analysis of the generalization results in the newly added Appendix G, which includes the discussion above as well as additional observations. In particular, we explain why cause-based deduction, under no direct effect (“NDE”), results in larger improvement in S-IR while effect-based deduction leads to larger improvement in N-IR. In short, this is a consequence of $A$ being sufficient for $B$ and $B$ being sufficient for $C$ in our constructed example, illustrating how the underlying problem structure can influence the effectiveness of different modes of reasoning generalization.

(3) False Necessity and False Sufficiency

Breakdown of necessity and sufficiency inconsistency rates into finer components such as false necessity and false sufficiency is a very interesting idea. For instance, in our framework, false necessity and false sufficiency can be interpreted as the events $\hat{\mathcal{N}}=\mathbb{N}\wedge\mathcal{N}\neq\mathbb{N}$ and $\hat{\mathcal{S}}=\mathbb{S}\wedge\mathcal{S}\neq\mathbb{S}$ respectively. However, notice that both of these events are the same event as $\hat{Y}\neq Y\vee\hat{Y}\_{X’}\neq Y\_{X’}$, that is either a factual or a counterfactual error has been made. These failures are already covered by factual and counterfactual error rates.

Overall, the individual inconsistency rates we introduce, N-IR, S-IR, AN-IR, and AS-IR, together with the individual error rates, F-ER and CF-ER, are already a finer breakdown of aggregated metrics like Avg-IR and Avg-ER, both of which represent general accuracy (the former accounts for correlations between factual and counterfactual errors whereas the latter looks at them independently). They offer a more nuanced insight into fine-tuning for reasoning as exemplified above.

Update: We have now included this discussion in the newly added Appendix H.2.