Thank you for your thoughtful review!

A critical point to clarify is that Assumptions 1 and 2 in Section 4 are merely sufficient conditions. There’s no reason a priori to expect that imperfect rewrites can be used in causal estimation, so the role of Theorem 4.1 is simply to show that the approach is not entirely vacuous! That being said, we also note that all causal estimation is based on strong assumptions, and the assumptions in Section 4 have the virtue of being largely checkable! Unlike most assumptions in causality, a practitioner can actually just manually inspect a random subset of the rewrite data to see if it matches the desired counterfactual elicitation.

Those are some clever toy examples, let's apply the considerations above to them:

1. Suppose you visually inspected some samples and had a suspicion that the rewriter was cutting the length by 20% each time it rewrites, as you describe. Unlike e.g. adjustment-based causal methods, you can just test your hypothesis directly, which we consider a major advantage of our method. To illustrate this, we ran two-sample t-tests on the rewrite vs. double-rewrite and found no significant differences between the mean length when rewriting on sentiment. Specifically, p-values were 0.1014 (IMDB) and 0.7496 (hh-rlhf).
2. We agree that in general Assumption 2 (additivity) may not hold, as more complex interaction effects could occur, but note that the ATE is itself only a sensible causal estimand in the absence of interaction effects (generally, not just in our setting). When such interaction effects are present, other estimands like the conditional ATE are more appropriate. Our work here aims to cleanly spell out a simple case, which we hope lays the foundation for more complicated causal estimands.

Responses to other comments and questions:

• Good point about Figure 1, we have edited it. Edit: Here is a link to the new figure https://postimg.cc/9wwXZ1bQ
• The key distinction is that Jiang, et al. focus on altering examples to change a classification score while RATE focuses on estimating a treatment effect of a specific latent attribute on a reward score. We now clarify this earlier in the manuscript per your suggestion.
• All single-rewrite and double-rewrite comparisons for the "Real World Reward Models" Experiments in Section 5.2 are already provided in Appendix D.1; we’ll note this in the main body. We don’t show the single rewrites in Figure 5 because the semi-synthetic experiments demonstrated they don’t properly cancel out rewrite errors.
• We would expect models to become better at generating “true” counterfactuals as they get better (i.e. better at following instructions, more similar to human-generated text).
• While it’s true that the double-rewrite method ends up using entirely LLM-generated data, it's not obvious that the original data distribution has any privileged status if the goal is to understand alignment—it might even be the case that LLM-rewrites are closer to the generations of the downstream fine-tuned model. A benefit of the rewrite method (as compared to e.g. adjustment) is the ability to spotcheck rewrites to make sure they look typical of the domains we care about.
• For our third experiment (Appendix C1), we will edit this section to give quantitative details supporting our claim that the double-rewrite method is more stable. In particular, the slopes of lines interpolating the data points differ: the single-rewrite method has a slope of -0.0413 while the double-rewrite method has a slope of -0.0114. Thus, while both are fairly stable in absolute terms, the double-rewrite is relatively more stable.

Thank you for your detailed review and your support!

We will address some of your higher-level comments and questions first.

• We appreciate your kind words about relevance and novelty. We agree that practical implications are important, which is why we demonstrate the utility of our method on real-world reward models drawn from RewardBench. Following the reviewer’s suggestion, we will emphasize this application in the abstract.
• We are happy to hear that you find the experiments well-designed. On the topic of rewrite quality, quantitative metrics are fundamentally proxy measurements for linguistic similarity. As such, we believe that simply allowing the reader to see random samples of our rewrites (Appendix D) is a more direct way to demonstrate rewrite quality.
• While we agree that the ultimate application is in downstream tasks, for the purposes of debugging it’s important to have modular evaluations. For instance, our work allows practitioners to identify whether issues identified in alignment (e.g., length bias) are due to the reward model itself or the alignment procedure. On the other hand, if we were to evaluate our method just by its impact on downstream performance, we would not be able to disentangle these two components.

Responses to other comments and questions:

• The computational complexity of RATE remains O(n) since we generate two rewrites per example via API calls and score them with a reward model, where the primary cost comes from generation. Using the gpt-4o-2024-08-06 model, processing 25K IMDB samples (including rewrites and rewrites-of-rewrites) costs only $60, and in practice, a relatively small n suffices for reliable confidence bounds on treatment effects.
• Regarding using other LLMs for rewrites, our priority was to estimate the treatment effect of concepts on reward model scores, so we used the best rewriter available at the time of writing. As LLMs improve in quality and efficiency, we imagine that smaller, more efficient LLMs will be able to perform acceptable rewrites, but considering the cost is so small, and the emphasis here is on precise evaluation, there is little benefit to using less capable LLMs.

Thank you for your review and your detailed comments.

We agree that the assumptions in Theorem 4.1 are somewhat strong, albeit less onerous than they may first appear: the cancellations only need to occur in expectation, which is a weaker condition than cancellation for each unit.

However, we emphasize that these are merely sufficient conditions whose purpose is to show that there exist any situations under which RATE is consistent: after all, there’s no reason a priori to expect that imperfect rewrites can provide a causal estimation—hence, the purpose of the theorem is simply to show that the approach is not vacuous. The substantive evidence of the paper are empirical: that the causal view must be considered (e.g., the experiments in Section 5.1 showing a mismatch with the naive and single-rewrite estimates) and that the double-rewrite method provides better counterfactual pairs than the single-rewrite method (e.g., based on Figure 3 and inspection of the candidate counterfactual pairs in the appendix).

That being said, we also note that all causal estimation is based on strong assumptions, and the assumptions in Section 4 have the virtue of being largely checkable! Unlike most assumptions in causality, a practitioner can actually just manually inspect a random subset of the rewrite data to see if it matches the desired counterfactual elicitation.

Regarding the specific questions:

1. RATE is agnostic to the particulars of how the rewriter-LLM was trained (e.g. with RLHF or not): all that matters are the produced rewrites. In particular note that humans also produce imperfect rewrites, and the same estimation procedure could be applied in principle to these expensive human-generated rewrites (subject to the same validity tests above), even though humans are not trained via any explicit reward model.
2. See main response above.
3. Probabilities of causation are appropriate for binary outcome variables like legal decisions, which is why we opted for the ATE to formalize how much the reward model (continuous-valued) is responding to attributes of text—hence, the cited papers use a fundamentally different notion of explanations. It’s an exciting direction for future work to explore whether the double-rewrite trick might improve estimation for other counterfactual estimands.

Thanks for your review. We agree that the behavior of reward models is an important and understudied area of alignment research.

First, we will respond to some higher level points:

• Regarding your concern about the causal claims, we think there may be a confusion here. We are not defining our counterfactual examples as examples that result in a changed classification decision, as in Jiang, et al. 2024. Rather, we are creating counterfactual examples with our rewrite process that differ on one binary attribute (and rewrite noise). That is, our counterfactuals are of the form “what would the text of this response be if it were generated with binary latent variable W set to 1 rather than 0.” We then use these counterfactual examples to estimate an ATE on the reward score itself. That is, RATE does not remove attributes, but rather changes the value of attributes (e.g., swapping sentiment from positive to negative) in order to understand how the reward model’s output is affected by the attribute. So, our treatment effect is in “units of reward score” regardless of any classification decision. We agree that we should mention the Jiang work earlier and more clearly distinguish our notion of counterfactual from the notion of counterfactual explanations addressed in Jiang. We have updated the manuscript to clarify this in the introduction.
• We agree that the ultimate goal of RATE is model improvement. However, the current alignment pipeline involves several different components. If an aligned model is producing undesirable behavior (e.g. excessively long responses, annoying bullet pointed lists, etc.) it’s not obvious where in the pipeline this behavior is coming from. Is this from the data used or generated during alignment? The reward model? An unintended consequence of the alignment objective? We focus on a modular evaluation of reward models themselves in order to localize failure points in the alignment procedure, and we plan to examine the impact of reward model bias on aligned models in future work.

Here are responses to some other comments and questions:

• Thank you for suggesting improving figure 1. We agree that this leaves something to be desired. Edit: Here is a link to the updated figure https://postimg.cc/9wwXZ1bQ
• Thanks for catching the typo on line 76. This is fixed on the updated version of the paper.
• Can you clarify what you mean by “significant” regarding the difference in distributions in figure 2? “Significant” as in hypothesis testing or as in “meaningful”?
• It’s not possible to guarantee that the rewrite procedure produces sensible text, but examination of (many) randomly selected examples shows the procedure is robust. When applying the method, it is, as always, important for practitioners to spot-check the data.
• Regarding your question about Assumption 1, note that Assumption 1 is actually much weaker than the $\xi$ terms cancelling out for each unit: rather, it suffices for the cancellations to occur in expectation, which is very permissive. This is easy to check via visual inspection on a random subset.