Measuring the Reliability of Causal Probing Methods: Tradeoffs, Limitations, and the Plight of Nullifying Interventions

(Continued from rebuttal part 1/N.)

C3: Notation

Thank you for noticing two typos in our notation: $Z$ was incorrectly removed as a subscript for $\hat{P}_Z$ in equations 1, 2, 3, and some of the surrounding text; and $i$ should be replaced with $k$ in the superscript of line 172. We will correct both typos in our next draft.

As for computing min/max over distributions: we take the maximum probability in each corresponding distribution (i.e., max, not argmax). E.g., for $P(Z = z_1) = 0.3, P(Z = z_2) = 0.7$, these operations would yield $min(P) =0.3$ and $max(P) = 0.7$. The only reason this is required in Equation 3 is to normalize the selectivity metric such that it is bounded by [0, 1] (see line 200).

Regarding $\hat{h}^l$ and $h^{l^*}$: as described in Section 3, $\hat{h}^l$ is the embedding obtained following the application of an intervention method that is being evaluated, and $h^{l^*}$ is a hypothetical embedding that would be obtained following the application of an idealized “perfect” intervention (which we do not have access to in practice). We agree with your suggestion that $h^{l^*}$ would be better denoted as $h^{*^l}$, and we will modify our notation correspondingly.

C4: Unclear training objective of approximated oracle probe

We explain how approximated oracle probes are trained in Section 4.3 (lines 282-290) and Appendix C.2: specifically, the training objective is to minimize the cross-entropy loss over the value taken by the property given an embedding input. Please let us know if you have any additional questions about the content in these sections, or if there are specific details we have not included that you would like us to add.

C5: Interventions on earlier layers

First, we will clarify why we only experimented on final-layer embeddings in our paper: as described in Section 4.2 (lines 265-269), we only consider interventions over final-layer embeddings in our experiments to ensure that our results are not impacted by the recoverability phenomenon observed by Elazar et al. (2021), allowing for direct analysis of the impact on LM task accuracy (Sections 4.5 and 5.2).

However, our framework can easily be applied to any layer. Thus, as requested, we have performed an additional followup experiment over a few earlier layers $l \in \{ 1, 4, 7, 10\}$ and the two top-performing counterfactual interventions (AlterRep and FGSM GBIs) to compare with the existing results for the final layer $l = 12$ and study the difference between linear (AlterRep) and nonlinear (FGSM GBIs) methods.

The results are visible at this anonymized link, where we plot the reliability (solid line), completeness (dashed line), and selectivity (dotted line) of FGSM and AlterRep by layer, using the intervention hyperparameter values for each layer that yield the maximum reliability (cf. Table 1). These results are obtained by following the same procedure as all experiments in the main paper, where the only element that has changed is the layer under investigation.

These results show that, as hypothesized (see lines 460-462), the nonlinear counterfactual method (FGSM) performs better in earlier layers, and the linear counterfactual method (AlterRep) performs better in later layers. This indicates that the representation of grammatical number ($Z_c$) is nonlinearly encoded in earlier layers, but becomes increasingly linear in later layers (cf. Alain & Bengio, 2017).

C6: Provide an example where non-linear causal probing is better than linear causal probing

Theoretically, in any case where properties are encoded nonlinearly – as may be expected of highly nonlinear objects such as deep neural networks – nonlinear probes will be better able to recognize these representations, as is often observed empirically (Pimentel et al., 2020; Belinkov et al., 2022; Davies et al., 2023). In such cases, causal probing interventions operating over nonlinear probes would be able to perform more complete interventions on a neural network’s representation of the target property, as we observe in the results reported for C5 above. However, some linear causal probing methods like AlterRep or INLP come with, e.g., stronger theoretical guarantees on “collateral damage”, which can lead to a more favorable tradeoff in favor of selectivity even if interventions are not fully complete (see Table 1).

C7: Include a figure denoting overall framework and contributions bullets in the intro

We agree that these suggestions would improve the readability of our paper and the clarity of our contribution. We will make both of the recommended additions. Likewise, we will implement the suggestion to add some citations on line 64 that are currently not cited until Section 2.

Conclusion

We hope we have resolved all of your concerns. If not, we remain available for further discussion.

Thank you for your detailed read of our paper and providing this comprehensive and thoughtful review. We appreciate that you found our framework to be novel, our results to be clear and justified, our findings to be potentially significant to the community, and our experiments to be comprehensive.

We address each of the concerns you raised as follows:

C1: Target distribution of oracle probes

You suggested that if “the distribution learnt by oracle probe is not uniform, that might bias the completeness metric for nullifying interventions [...] Hence, the authors should verify the oracle distribution in the case of nullifying interventions and check how similar it is to the uniform distribution.”

In our paper, we use oracle probes to evaluate interventions by first observing the distribution emitted by oracle probes for intervened embeddings, then assessing how similar this distribution is to the target distribution (as defined in Section 3). However, if – as you suggested – this evaluation process was further “verified” by measuring how similar the oracle probe distribution is to the target (uniform) distribution, this would amount to “verifying” the evaluation process based on how high a score it assigns to the interventions it is evaluating, which would be circular reasoning.

Please let us know if we have misunderstood your suggestion, and we would be happy to discuss this matter further.

C2: Standard deviation and random seed

Below, we report the standard error for each result in Table 1:

As for the figures in the main paper, which are all scatterplots, it is not clear how we would report standard deviation in such plots. However, these figures are based on results that can also be visualized as line plots (across a greater number of figures) that are included in the appendix, where Figure 1 corresponds to Figures 3-5.

Thus, we have recomputed these figures with standard error visualized as a semi-transparent region around each line in the plot, and make them available at the following anonymized links for GBIs, AlterRep, and nullifying methods (corresponding to Figures 3, 4, and 5, respectively).

As observed in the included table and all figures, the standard error is very small, and is barely visible in visualizations. This is due to the very large size of the test set, which includes 50K instances in total. We will add these error values to Table 1 and Figures 3-5.

Regarding the random seed: we split the full train/val/test set from the LGD subject-verb agreement task data only once using a single random seed, as is standard; but we do train several different randomly-initialized interventional probes and approximated oracle probes across various hyperparameter settings ($n = 36$) for each experiment, each time selecting the probes with the highest validation-set accuracy. (See Appendix C.2 for further details.)

(Continued in rebuttal part 2/N.)

Thanks a lot for your clear and detailed response! A lot of concerns were addressed, but I am still not sure about the distribution of oracle probe. I don't understand the "circular argument" pointed by the authors regarding my concern about the oracle probe for nullifying interventions. From my understanding, the oracle probe gives a distribution over whether the property $Z=z$ holds after an intervention. So authors have access to $p_Z(z|h_{Z}^l)$, and they can test whether it is a uniform distribution or not. As the completeness metric for nullifying interventions (eq 2) assumes it to be uniform and then checks how much does $\hat p_Z(z|h_{Z}^l)$ differs from it. But if $p_Z(z|h_{Z}^l)$ is not uniform to being with, then this metric might be biased. Please let me know if you need more clarification on this.

Thank you for your thoughtful and comprehensive review. We appreciate that you found our approach to be simple, versatile, and novel.

We address each of the concerns you raised as follows:

C1: Must enumerate all probed properties

While it is indeed necessary to enumerate the set of properties $Z_1, …, Z_k$ to be used in computing selectivity, each corresponding approximated oracle probe $o_1, …, o_k$ can both be trained and applied to evaluate interventions at test-time fully independently of the others. This means that it is possible to add new properties post-hoc, train oracles on only the new properties, and easily recompute the aggregated selectivity without having to re-train existing oracle probes.

C2: How should oracle probes be evaluated?

It is true that our framework crucially relies on high-quality oracle probe approximations, and the question of how best to evaluate such approximations is indeed an open one. In our experiments, we simply perform standard hyperparameter grid search over possible oracle probes and select the MLP architecture and learning rate that yields the highest validation-set accuracy (see Section 4.3 and Appendix C.2). We opted for this approach in accordance with Occam’s razor: as we are the first to define and experiment with oracle probes, we selected from the range of possible oracle probes in the way that one would select any other model or probe (i.e., the one with the best performance on the validation set).

However, we also experimented with linear oracle probes (see Appendix D.3). These linear oracles yielded similar results to those obtained using the MLP oracles whose results are listed in the main paper – e.g., all major findings are the same – which gives us some confidence that different types of oracle probe approximations generally yield similar results in the experimental setting explored in this work. However, we acknowledge that this is a purely empirical observation made on the basis of a limited set of experiments, and it is possible that the matter of oracle probe selection could be much more consequential in more complex experimental scenarios. This is an important consideration for future work, and we will add further discussion on this point to the paper.

C3: Experiments consider only binary properties

It is true that our experiments consider only binary properties, which – as you noted – is a standard setting explored in the literature, and thus is a natural place to begin validating the proposed causal probing evaluation framework. However, our framework (as provided in Section 3) is explicitly defined to accommodate properties with more than two possible values (see lines 178-180).

C4: Are there additional implicit assumptions for oracle probes?

No, we enumerate the full set of experimental assumptions for oracle probe approximations in Section 4.3 (lines 282-290) – i.e., that they should be trained on a disjoint set from interventional probes, that the target property should be conditionally independent of the other properties, and that the label distribution of the target property should be preserved while enforcing conditional independence. However, please note that these are not theoretical assumptions for defining oracle probes – rather, they are empirical measures to obtain more reliable oracle probe approximations (see lines 285-290).

We will better clarify this point in Section 4.3 – specifically, that these experimental assumptions are introduced as initial “best practices”, and that future work could experiment with additional measures to improve such approximations.

Conclusion

We hope we have resolved all of your concerns. If not, we remain available for further discussion.

Thank you for your thoughtful review. We appreciate that you found our framework to be novel and consider our analysis to be detailed and comprehensive across multiple methods and experimental settings.

We address each of the concerns you raised as follows:

C1: Limited scope of models and tasks: how can the framework be adapted to more general/complex scenarios?

It is important to distinguish between our framework and our experiments. Our framework, as presented in Section 3, is general with respect to arbitrary language models and tasks – i.e.:

• Our framework does not make any assumptions regarding language model architecture, parameter count, training distribution, etc.; nor does it make any specific assumptions about task structure. For example, the “causal vs environmental properties” we discuss in Section 4.2 are not required to apply our framework – we select a task where this division is clear for our experiments only because it better motivates our study of task accuracy.
• Our framework does make one important assumption regarding properties: that is, we assume that there is a finite set of properties (i.e., discrete random variables $\mathbf{Z} = Z_1, …, Z_k$ corresponding to latent features of text inputs) that each take a specific value $Z_i = z$ for any given input $x$. However, this is simply the definition of probing (Belinkov, 2022), meaning that all works in the probing literature share this assumption. As such, the framework can be applied without modification to any setting that can be studied via causal probing (as discussed in Appendix A).

However, we agree that the specific experimental instantiation of our framework, as explored in this work, is somewhat limited in that we only experiment with a single well-studied language model, a simple linguistic task, and two binary variables denoting surface-level syntactic properties. We begin with this simple setting because, despite the abundance of interpretability research on this model and task, there is still no consensus regarding what causal probing methods are appropriate for studying this setting, let alone how to compare and evaluate such methods. Thus, we argue that it is important to begin by comprehensively studying existing causal probing methods using the proposed framework in this simple setting, rather than prematurely scaling our analysis to more complex models and tasks (see Appendix A). Note that this choice also enables us to study existing methods in contexts for which they were originally designed, rather than applying them in new scenarios for which they were never intended (as doing so might have yielded unfair results).

C2: Challenging in practice: approximating oracle probes

You note that “some ideas seem challenging to achieve in practice (line 145)”, and the challenge described in this part of the paper is that “a perfect oracle probe does not exist in practice, so any practical implementation must approximate it.”

It is true that our framework crucially relies on high-quality oracle probe approximations, and the question of how best to evaluate such approximations is indeed an open one. In our experiments, we simply perform standard hyperparameter grid search over possible oracle probes and select the MLP architecture and learning rate that yields the highest validation-set accuracy (see Section 4.3 and Appendix C.2). We opted for this approach in accordance with Occam’s razor: as we are the first to define and experiment with oracle probes, we selected from the range of possible oracle probes in the way that one would select any other model or probe (i.e., the one with the best performance on the validation set).

However, we also experimented with linear oracle probes (see Appendix D.3). These linear oracles yielded similar results to those obtained using the MLP oracles whose results are listed in the main paper – e.g., all major findings are the same – which gives us some confidence that different types of oracle probe approximations generally yield similar results in the experimental setting explored in this work. However, we acknowledge that this is a purely empirical observation made on the basis of a limited set of experiments, and it is possible that the matter of oracle probe selection could be much more consequential in more complex experimental scenarios. This is an important consideration for future work, and we will add further discussion on this point to the paper.

C3: Redundant descriptions

Thank you for pointing out that there is some overlapping and redundant information in Sections 4 and 5, and that Section 5 contains some excessive experimental details. We agree, and will remove redundancies and port some minor details to the appendix.

(Continued in rebuttal post 2/N.)

(Continued from rebuttal post 1/N.)

C4: Experiments assume full determinism and binary variables

If, by “full determinism”, you mean access to ground-truth labels for properties, where properties are discrete random variables taking a specific (deterministic) value for any given token or token sequence, then it is true that this is an assumption of our framework, but this assumption is common to all probing research, as discussed in our response to C1. (Please let us know if you meant something else by “full determinism”, and we will be happy to discuss this matter further.)

Next, while it is true that our experiments include only binary variables, our framework is explicitly defined to accommodate properties with more than two possible variables – see lines 178-180.

Finally, our framework is also general to any number of properties – see, e.g., line 200-203, which defines selectivity when more than two properties are being considered (the definition for completeness and reliability does not change). Thus, while our experiments only consider a scenario with two properties, $Z_c$ and $Z_e$ in the context of the subject-verb agreement task, our framework can still be applied to any other language-modeling task, and also does not require that properties be categorized as “causal” or “spurious” (as noted in our response to the first concern).

We will update Sections 3 and 4 to more explicitly differentiate between the “probing assumption” required by our framework versus the features of our experimental setting that are not assumed by the framework.

Conclusion

We hope we have resolved all of your concerns. If not, we remain available for further discussion.

Thank you for your response. However, considering the gap between the proposed framework and the experimental setting, I will keep my socre.

Dear Reviewer,

Thank you for responding to our rebuttal. We respectfully disagree with your characterization of our experimental setting: it is a valid instantiation of the proposed framework, with no "gap" between the framework and experiments.

Rather, as you expressed in your initial review, the potential concern here is whether the experiments are sufficiently broad to validate our general framework. We responded to this point in detail in our rebuttal (see C1, C2, and C4). We regret that there was not time to engage in further discussion on this matter, given that the response to our rebuttal was posted with less than 3 hours left in the discussion period and does not contain any specific feedback, suggestion, or followup question for us to respond to on the final day for author responses.

Although we were unable to engage with you during the discussion period, we kindly request that you please consider whether our rebuttal addressed the important potential concerns you posed in your initial review, and hope that you will update your assessment during the meta-review process accordingly.

Thank you,

Authors

Thank you for your thoughtful review. We appreciate that you found our paper to be “well-written and organized,” and our method to be “easy to follow.”

We address each of the concerns you raised as follows:

C1: Relationship between oracle probes and interventional probes

It is true that the “evaluator” (approximated oracle probe) and “intervention generator” (probes used to obtain interventions) are trained on very similar tasks: in each case, to predict the value of a property $Z_i$ (i.e., a discrete random variable denoting an input feature) from language model embeddings. This is intended: oracle probes are used to validate whether or not an intervention has had the desired effect on the representation of the target property encoded by embeddings. There are several important details concerning the relationship between oracles and interventional probes that make oracle probes appropriate for evaluating interventions:

1. As noted in lines 282-285, there is no overlap between the data used to train or validate approximated oracle probes versus interventional probes, and the test data used to evaluate interventions is also fully disjoint from the training and validation set of both kinds of probes.
2. Oracle probes used to measure completeness are indeed trained to predict the same property $Z_i$ as interventional probes. However, at test time, interventional probes are used to modify embeddings to encode a different value of $Z_i$ ($z' \neq z$ for counterfactual interventions, and $z’=0$ for nullifying interventions; see Section 3), and oracle probes are used to validate the extent to which the intervention has carried out the desired operation. Thus, it is in no sense “guaranteed” that interventions performed using a given interventional probe will produce embeddings that are recognized by an oracle probe as encoding the target value (i.e., completeness = 1); and as observed in our experiments, oracle probes usually do not show that the desired operation has been fully carried out.
3. Oracle probes used to measure selectivity are trained to predict a different property $Z_j$, while interventional probes are trained to predict property $Z_i$, and $Z_j \neq Z_i$. This means that there is no “overlap” between the tasks learned by oracle probes measuring selectivity and interventional probes. Thus, given a perfect intervention method, interventions performed using this method over a given interventional probe (trained to predict $Z_i$) should not affect the oracle probe’s prediction of $Z_j$.
4. As explained in lines 285-290, there is an important difference between the training distributions of interventional and oracle probes:
   • In order to study interventions in a realistic setting (where it is generally not possible to control for all non-targeted properties), interventional probes are trained with data that is fully i.i.d. with respect to the data distribution, meaning that they might incorrectly leverage spurious correlations with $Z_j$ to predict the target property.
   • On the other hand, oracle probes are trained on data that is subsampled in order to enforce conditional independence between a target property $Z_i$ and the other properties $Z_j \neq Z_i$ that are considered in measuring selectivity. This means that oracle probes are unable to leverage spurious correlations between these properties, which is important for ensuring that oracle probes can serve as objective evaluators of interventions.

(Continued in rebuttal post 2/N.)

(Continued from rebuttal post 1/N.)

C2: Are oracle probes biased in favor of counterfactual interventions over nullifying interventions?

First, we will describe our understanding of your specific concern as follows – please let us know whether or not our understanding is correct:

• As discussed in Section 3, both (a) oracle probe training distributions $P_o(Z | h)$ (where $h$ is an embedding), and (b) target counterfactual distributions (where probability mass is shifted from the original ground-truth value of a property to a counterfactual value), are “one-hot” distributions (where probability is 1 for a given value and 0 for all others), whereas nullifying interventions are uniform.
• Thus, because the training objective of oracle probes has the “same shape” (one-hot) as counterfactual distributions, this might be interpreted as biasing oracle probes in favor of counterfactual interventions over nullifying interventions.

However, there are two reasons that this interpretation is incorrect:

1. Nullifying interventions are obtained using interventional probes that are also trained on “one-hot” distributions, and these interventions explicitly operate by modifying embeddings such that these interventional probes predict the uniform distribution over possible values. For instance, INLP projects embeddings into the nullspace of linear interventional probes, guaranteeing that each of the interventional probes will assign equal probability to each possible value of the target property. Thus, it is entirely appropriate to evaluate such interventions on the basis of how similar an impact they yield on oracle probes, in precisely the same way that counterfactual interventions are evaluated based on whether they have the same impact on oracle probes as the interventional probes used to perform them.
2. One possible response to (1) is that, e.g., INLP is only designed to induce a uniform distribution over linear interventional probes, whereas the oracle probes used in the experiments reported in the main paper are MLPs. However, AlterRep also only considers linear interventional probes – in fact, in our implementation, they are the exact same probes – and while AlterRep yields the greatest “peak” reliability of all interventions (using an MLP oracle probe), INLP performs worse than all counterfactual methods (see Table 1). Furthermore, in Appendix D.3, we also perform the same experiments using linear oracle probes, and observe similar results: the same tradeoff exists between completeness and selectivity, and comparing intervention methods by maximum reliability yields the same ordering of most- to least-reliable (in particular, still placing both nullifying methods below all counterfactual methods). Therefore, our finding that nullifying methods are less complete and reliable than counterfactual methods is independent of whether the approximated oracle probe is linear or nonlinear.

C3: Simplicity of experimental setup (binary variables)

First, it is important to clarify that our experiments are restricted to binary random variables, but our framework is explicitly defined to accommodate variables with more than two possible variables – see lines 178-180.

However, we agree that the specific experimental instantiation of our framework, as explored in this work, is somewhat limited in that we only experiment with a single well-studied language model, a simple linguistic task, and two binary variables denoting surface-level syntactic properties. We begin with this simple setting because, despite the abundance of interpretability research on this model and task, there is still no consensus regarding what causal probing methods are appropriate for studying this setting, let alone how to compare and evaluate such methods. Thus, we argue that it is important to begin by comprehensively studying existing causal probing methods using the proposed framework in this simple and well-studied setting, rather than prematurely scaling our analysis to more complex models and tasks (see Appendix A). Note that this choice also enables us to study existing methods in contexts for which they were originally designed, rather than applying them in new scenarios for which they were never intended (as doing so might have yielded unfair results).

(Continued in rebuttal post 3/N.)

(Continued from rebuttal post 2/N.)

C4: Introducing task-irrelevant interventions

In the review, you suggested we introduce “interventions that could potentially not impact [language model] task accuracy” in order to “provide a clearer understanding of whether performance loss is due to the intervention’s effectiveness or simply the disruption caused by perturbing the representation.” However, it is not entirely clear to us what such an intervention would be, as any intervention over embedding representations requires perturbing them, and will correspondingly lead to performance loss if the perturbation magnitude is too great.

Our best interpretation is that we might consider interventions that are completely unrelated to task-relevant properties. As a random baseline to better understand an appropriate range for perturbation magnitude $\epsilon$ used to constrain GBIs, we have run an additional experiment with interventions that randomly perturb each element of an embedding vector by $+\epsilon$ or $-\epsilon$. (Note that this is similar to FGSM GBIs, which also yield perturbation vectors where each element is positive or negative $\epsilon$, but here positive and negative values are randomly determined instead of setting them to the sign that most increases probe loss.)

The results are visible at this anonymized link, where we plot the reliability (solid line) versus $\Delta$Acc of this random perturbation by $\epsilon$ value in purple, and include FGSM for comparison. (For reference, this plot corresponds to Figure 6 in the appendix.)

In these results, we observe that the random intervention has both a much lower reliability, and a lower impact on LM task accuracy, relative to FGSM for a given $\epsilon$. As such, this may be taken as evidence that LM performance loss is indeed primarily due to the specific representation targeted by FGSM, and not simply the disruption caused by perturbing the embedding.

C5: Considering scenarios where properties are independent

You recommended that we “evaluate a scenario where $Z_c$ and $Z_e$ are independent, $Z_c$ only impact label.” As explained in Section 4.2, this is precisely the scenario we consider in all experiments: by definition, $Z_c$ causally determines the ground-truth label in the subject-verb agreement task, and $Z_e$ is spurious (meaning that it may be correlated with the label, but does not determine it).

C6: Evaluate completeness of nullifying methods by adding a new class for oracle probes?

We evaluate nullifying interventions by comparing the probability distribution emitted by an oracle probe (using the nullified embedding as input) to the uniform distribution. You suggested that instead we might “add a distinct category representing ‘neither’ of the options, rather than assuming a uniform distribution”; but this would be ill-defined for two-reasons:

1. As discussed in our response to C2, nullifying interventions like INLP work by modifying embeddings such that interventional probes predict the uniform distribution over possible values, meaning that it is appropriate to evaluate such interventions on the basis of how similar an impact they have on oracle probes as they (are designed to) have on interventional probes.
2. It would also be impossible to train probes on the proposed “none of the above” category. Probing is defined as learning a mapping from an embedding $h$ to a property (discrete random variable) $Z_i$ that takes a specific value $Z_i = z$ for any given input $x$ (Belinkov, 2022). Thus, in order to train an oracle probe with respect to the “none of the above” class, it would be necessary to provide models with an input $x_0$ where $Z_i$ does not take any value; but by definition, there exists no such $x_0$ in English. (See lines 484-488 for further discussion.)

Conclusion

We hope we have resolved all of your concerns. If not, we remain available for further discussion.

Thank you for your thoughtful review. We appreciate that you find the topic of our paper to be “important and timely,” that our proposed metrics are “intuitive and reasonable,” and that our paper is “well-written and well-structured” and “easy to follow.”

We address each of the concerns you raised as follows:

C1: Can the authors discuss possible directions for improving existing approaches?

While the primary goal of our work is to define an empirical analysis framework to measure the reliability of causal probing methods, and not explicitly to improve the methods under investigation, we do see several ways our framework could be leveraged to improve causal probing methodologies:

1. Our framework allows one to calibrate the hyperparameters of existing causal probing interventions to optimize completeness, selectivity, or the compromise between them (reliability). Previously, it has not been clear how to define, measure, or balance such priorities, and prior work has resorted to a variety of differing heuristic methods for determining such hyperparameters (see, e.g., Ravfogel et al., 2020, 2021; Elazar et al., 2021; Tucker et al., 2021; Davies et al., 2023, etc.). In our experiments, we show that there is a tradeoff between completeness and selectivity; and we compare methods by setting their hyperparameters to the values that yield the maximum reliability, demonstrating that our framework can be leveraged as the first concrete basis for optimizing intervention hyperparameters.

2. How can we objectively measure and compare the “success” of causal probing methods, and how can we make progress as a community if we are unable to do so? It has previously been unclear how to quantify or demonstrate improvements offered by new methods, given the lack of any clear metric for comparing them or measuring progress. Our framework makes it possible, for the first time, to directly compare the reliability of various causal probing methods, providing future work with a clear basis to show whether or not a new approach improves over existing methods.

We will add discussion on both of these points to our conclusion..

C2: Reliance on approximated oracle probe

It is true that our framework crucially relies on oracle probe approximations, and the question of how best to obtain and evaluate such approximations is indeed an open one. In our experiments, we simply perform standard hyperparameter grid search over possible oracle probes and select the architecture and learning rate that yields the highest validation-set accuracy (see Section 4.3, Appendix C.2, and Appendix D.3). We opted for this approach in accordance with Occam’s razor: as we are the first to define and experiment with oracle probes, we selected from the range of possible oracle probes in the same way that one would typically select any other model or probe – i.e., the one with the best performance on the validation set. However, we did also carry out experiments with some of the less-performant probe architectures (see Appendix D.3), yielding similar results to those obtained using the top-performing oracles whose results are listed in the main paper.

C3: Oracle probe assumptions

We enumerate the full set of experimental assumptions for oracle probe approximations in Section 4.3 (lines 282-290) – i.e., that they should be trained on a disjoint set from interventional probes, that the target property should be conditionally independent of the other properties, and that the label distribution of the target property should be preserved while enforcing conditional independence. However, please note that these are not theoretical assumptions for defining oracle probes – rather, they are empirical measures to obtain more reliable oracle probe approximations (see lines 285-290).

We will better clarify this point in Section 4.3 – specifically, that these experimental assumptions are introduced as initial “best practices”, and that future work could experiment with additional measures to improve oracle probe approximations.

Conclusion

We hope we have resolved all of your concerns. If not, we remain available for further discussion.