Preference Learning for AI Alignment: a Causal Perspective

Thank you for your review and for engaging with the core ideas of our work. We appreciate your time and feedback, and we’d like to address your concerns in hopes of clarifying our contributions and positioning.

The semi-synthetic design We appreciate the reviewer’s attention to our experimental design and understand their concern about the use of semi-synthetic tasks. However, we’d like to clarify why semi-synthetic evaluations are a necessary and principled choice for studying causality in preference learning. Evaluation of causal inference methods demands knowledge of the true data-generating process (e.g., presence of confounders, the true set of causal variables), which is unknowable in purely observational real-world datasets. Semi-synthetic data allows us to:

• Control the degree of overlap.
• Inject controlled confounding (to simulate real-world biases).
• Precisely measure how well the competing methods recover true relationships.

Real-world datasets only show the observed human choices, not the counterfactual "what-if" responses needed to assess how a user would react if the same prompt were answered differently or if they acted according to a different objective. By augmenting these real-world public datasets with synthetic variations (e.g., simulating distribution shifts or conflicting objectives), we create a controlled testbed to isolate causal challenges—something impossible with static, observational data. Our approach aligns with established causal inference literature, where semi-synthetic data is the standard for evaluating methods [1, 2].

Useful vs. harmful correlations We agree with your observation that correlations like “academic-style queries → rigorous answers" are often valid and desirable—indeed, in many cases, these patterns reflect genuine user needs. However, our focus is on the challenges such correlations pose when training reward models from observational data where confounding is present. This issue is particularly critical in the context of steerable alignment, where we aim to gain control over which objective an LLM aligns to at inference time, rather than letting the model infer it purely from its input. Our case study in Section 4 examines the problem of learning a multi-objective, steerable reward model, where the two objectives considered are learned from different data distributions. We demonstrate that the naive approach fails when the train reward model is applied to data distributions different than at training time, which can plausibly occur in the real world. We also show how insights from causal representation learning can help build more robust reward models. ACTION: We acknowledge that not all correlations are harmful, and we will clarify this nuance in the revised version of our manuscript.

Practicality vs. coverage You also raise a valid concern about the practicality of ensuring broad coverage of latent factors. We agree that exhaustive interventions are often infeasible, and we do not advocate for unrealistic data collection burdens. However, our goal is not to advocate for rigid control but rather to bring attention to challenges in learning across heterogeneous user populations, merging datasets collected with different objectives, or collecting feedback directly from users who wrote the prompts—practices that are present in preference learning today. While training reward models in an unsupervised, uncontrolled manner is flexible and scalable, we argue that the community should consider seeking a better balance between practicality and robustness. We highlight cost-effective strategies such as active querying (Appendix C) and causal regularisation techniques (as exemplified with the adversarial model) that can improve robustness without excessive overhead.

ACTION: We will explicitly discuss the trade-offs between practicality and robustness in the final version, clarifying our position.

---

We hope this response alleviates your concerns about our paper’s positioning. Our work does not dismiss observational data but instead provides tools to diagnose its limitations—a step we find crucial to achieving scalable and robust alignment. We believe the paper’s insights are valuable for the community, especially as alignment research moves towards greater personalisation.

Thank you again for your time and feedback. We are happy to answer any further questions you may have and incorporate further revisions to ensure clarity in the final version.

[1] arxiv.org/abs/1606.03976

[2] arxiv.org/abs/1705.08821

Thank you for your detailed and insightful review. We appreciate the overall positive evaluation of our work. Below we address your comments and concerns:

The Multihead architecture We appreciate your suggestion, ACTION: we clarify how the Multihead architecture incorporates the causal structure in the camera-ready version.

Additional References Thank you for your suggestion regarding the additional references. ACTION: We suggest including [1, 2, 3, 4] as additional references strengthening our claims on page 5 and Appendix C.

Overlap/Positivity.

• Terminology: Preference learning conventionally assumes homogeneity of user preferences, so we stated identifiability assumptions in the unconditional case; this also reduces the notational clutter. However, we acknowledge that overlap is usually used conditionally. ACTION: We propose to rename Assumption 3 as Unconditional Positivity and Assumption 5 as Unconditional Latent Positivity. We will also clarify that we use positivity as shorthand for the unconditional case, while Conditional Positivity (a.k.a. Overlap) applies when user-specific covariates are considered (e.g., Example 4, Section 4).
• Sufficiency: Indeed, positivity is not necessary for identification but is required for non-parametric estimation as shown in the proofs of Proposition 1 and Proposition 2. Thank you for pointing this out. ACTION: We will adapt the presentation of these results to make this clear.
• Statistical vs. causal confusion: We agree with the reviewer that the UltraFeedback experiment primarily addresses statistical generalisation. The setup does not strictly violate the positivity condition, making the problem solvable in the limit of infinite data. Yet, a perfect correlation is not expected to occur in the real world so we focus on the practical challenge of limited latent positivity. It has been shown that near-violations of positivity inflate estimator variance and necessitate impractical amounts of data for robust inference [5, 6, 7]. Thus, while the observed performance degradation is a statistical issue, it stems from the weak satisfiability of the causal positivity assumption. ACTION: We appreciate the feedback and agree that the presentation of this experiment could be sharpened. We will explicitly frame this experiment as a statistical challenge arising from near-violations of the causal assumption and not a causal non-identifiability problem.

Other comments Thank you for catching all the typos and your suggestions regarding the introduction. ACTION: we will implement them!

Heterogeneity of user preferences We would like to highlight that while the case study of section 4 is presented as a study of confounding, the issues discussed are inherently a consequence of the heterogenous preferences of the two user groups: the helpfulness and the harmlessness preferring, where the two objectives are often conflicting. We find the investigation of more fine-grained variability in user preferences, going beyond the simplified setup of just two, an exciting direction for future work.

Distinction from prior work with user-specific covariates Siththaranjan et al. highlight the possibility of heterogeneity in human preferences, pointing out the existence of potentially unobservable, hidden contexts that can influence user preferences. They demonstrate the negative consequences of implicitly aggregating over these hidden contexts when performing preference learning under a homogenous BTL model. Using our notation, they show that applying $\mathbb{E}[L(x; y, y')]$ as a global estimate of the expected preference for all users (instead of $\mathbb{E}[L(x; y, y') \mid C = c]$ separately for each set of user-specific covariates $c$) can lead to misalignment, especially with respect to the minority groups.

However, their work does not address the issue of confounding, which arises when user-specific covariates $C$ not only influence the preference $L$ but also the distribution of the prompts $X$, making $C$ a confounder. More generally, the limited number of related works considering user-specific covariates in preference learning assumes that user-specific preference datasets are collected under a randomised distribution of treatments. Our work challenges this assumption and explicitly considers the scenario where this distribution is influenced by $C$ (which is the case for instance when the prompts $X$ are written by the users themselves), leading to confounding issues examined in the case study of Section 4.

---

We thank the reviewer for their positive evaluation of our work and are happy to answer any further questions!

[1] arxiv.org/abs/1811.12359

[2] arxiv.org/abs/2105.06422

[3] arxiv.org/abs/2209.11924

[4] arxiv.org/abs/2203.16437

[5] doi.org/10.2307/2998560

[6] doi.org/10.1111/1468-0262.00442

[7] doi.org/10.3386/t0330

Thank you for your thorough and insightful evaluation of our work. We appreciate the time and effort you put into assessing our paper. Below, we address your questions and comments point by point.

Presentation of assumptions We acknowledge your concern regarding the potential misinterpretation of our assumptions as necessary rather than sufficient. ACTION: We will make appropriate adjustments in the phrasing in the revised version of our paper. In particular, we suggest that the first paragraph of section 3.1 is instead formulated as:

Causal inference provides a framework to answer counterfactual, 'what if' questions even when only observational data is available. A key part of causal analysis involves ensuring that the causal quantity of interest can be estimated from observational data alone. The following commonly made assumptions, adapted to our preference learning setup, are sufficient to guarantee identifiability and non-parametric estimability:

The multi-environment setup We greatly appreciate your suggestion to consider multi-environment setups and acknowledge their importance in broadening identifiability guarantees. ACTION: To further emphasise the fact that the assumptions discussed in our paper are sufficient rather than necessary, we will elaborate that alternative setups, such as having access to data from multiple environments, under certain conditions may also enable identifiability. We will reference the related literature you highlighted and contextualise it within multi-dataset reward model training. While learning reward models from multiple datasets is an emerging research area, its causal implications warrant further investigation. We view extending our framework to multi-environment settings as a promising avenue for future work.

That said, our case study in Section 4 underscores a fundamental challenge in merging datasets collected under divergent objectives. Specifically, the HH-RLHF dataset comprises two independent subsets: the helpfulness subset and the harmlessness subset, which can be interpreted as distinct environments. Our experiments highlight potential pitfalls in training a multi-objective reward model when datasets exhibit distributional differences.

We acknowledge the relevance of investigating our results in larger-scale multi-environment settings, such as those considered in ArmoRM. However, the datasets involved in ArmoRM lack counterfactual preference labels—i.e., they provide preferences only under their respective training objectives but do not indicate how preferences would transfer across datasets. The lack of overlap in objectives between the datasets of ArmoRM makes it difficult to assess generalisation beyond the dataset-specific correlations present at training time. In our experiments, the assessment of the counterfactual scenarios is possible thanks to the provision of additional, counterfactual labels to the HH-RLHF by [1]. The presented results exemplify potential risks associated with training multi-objective, steerable reward models from a composition of datasets collected under different, potentially conflicting objectives. We hope this motivates the development of new datasets tailored for robust reward model training.

Related work & contributions We appreciate your pointers to related work. ACTION: we will incorporate the suggested references into our discussion of related literature. Additionally, we would like to point out that our contribution extends beyond articulating sufficiency assumptions for preference identification. A second key contribution of this work is the highlighting of the potential confounding effects due to user-specific covariates. While there indeed exist prior works that aim to infer user-specific covariates (e.g., [2, 3]), they do not examine the confounding issue. The implicit assumption of prior works is that the covariates $C$ only influence the preference label $L$ but not the prompt distribution $X$. Our work considers cases where $C$ influences both $X$ and $L$, and thus it acts as a confounder. This is the case when users write the prompts $X$ and subsequently score the LLM’s generated responses to these prompts, which may in practice be the case (e.g. the HH-RLHF dataset).

Minor corrections Thank you for catching the typo on line 20, we will correct it!

---

We are grateful for the opportunity to refine our work based on your thoughtful feedback. Your suggestions have helped us improve the presentation clarity, depth and broader impact of our research. We hope our explanations and proposed revisions address your concerns. Thank you for your time and consideration.

[1] arxiv.org/abs/2312.08358

[2] arxiv.org/abs/2402.05133

[3] arxiv.org/abs/2409.11901

Thank you for the time taken to evaluate our work. We your recognition of our work as a meaningful addition to the broader literature. Below we address your comments:

Presentation. In the camera-ready version, we will use the additional space to clarify key intuitions, especially for readers less familiar with causal representation learning, and include citations to introductory causality resources. Thank you for this suggestion.

Q1: Treatment effects. Thank you for your question regarding our causal framework. Due to space constraints, we have provided an intuitive explanation here.

Q2: Evidence. Thank you for prompting us to substantiate the claim about data collection practices. We are happy to provide supporting evidence:

1. the widely-used Anthropic HH-RLHF dataset provides concrete evidence of this. As described in [2], the dataset construction process explicitly involved users both writing prompts and providing preference judgments on LLM’s responses.

2. OpenAI's ChatGPT interface empirically demonstrates this collection approach. Users are frequently presented with pairs of responses to their own queries and asked to indicate preferences. OpenAI's privacy policy confirms these interactions are used for model improvement.

Q1: Bandits. Thank you for the suggestion to consider contextual dueling bandits. However, we clarify that the confounder in Example 1 is the unobserved user-specific variable $C$, which influences both the prompt $X$(during dataset generation) and the preference labels $L$—not $X$ itself. Secondly, contextual dueling bandits focus on optimising a selection policy for choosing responses $Y,Y'$ given the observable context $X$ (the prompt), whereas in our setup, these responses are passively sampled from an LLM, whose policy we do not control. Our work focuses on learning a reward model from static, observational data. While applying a contextual dueling bandit framework to jointly learn preferences and control the sampling policy is an interesting direction (e.g. [3]), our work focuses on biases in preference learning from observational data rather than on designing a strategy for sampling the candidate responses. Furthermore, even in a dueling bandits, preferences are typically modelled with the BTL model, which as discussed, does not account for confounding. If an unobserved $C$ influences both the prompts $X$ and preference labels $L$, the learned preference model would remain biased.

Q2: Example 2. We would like to clarify that Ex. 2 is not intended to prescribe a fixed functional form for the reward model for learning. It is an illustration showing why conditioning on prompt-specific features ($Z^X$) is essential. The provided equation exemplifies a possible structure of the ground-truth reward function to highlight the plausibility of the effects of $Z^X$ being non-additive.

In practice, to learn a reward model we would not assume knowledge of the parametric form or the set of relevant latent features. Instead, as discussed in the paragraph Discovery of Z, the latent factors are learned from data. This flexibility, however, induces challenges associated with causal discovery under minimal supervision [4, 1], including overfitting to spurious correlations, as illustrated by our case studies.

Q3: The UltraFeedback experiment. This experiment highlights the impact of the latent overlap assumption—when strictly violated, causal identification is impossible. We show that weaker overlap makes the recovery of the ground-truth model significantly more challenging. This experiment highlights the inherent limitations of learning from observational data only, rather than promoting a specific approach.

UltraFeedback was chosen for its unique structure, allowing control over correlations between latent features. Unlike other datasets like Stanford Human Preferences or HH-RLHF, which label instances based on a single objective, UltraFeedback scores each prompt-response pair across four latent factors.

Q4: DPO. Thank you for this suggestion. While DPO bypasses the explicit reward modelling of RLHF, it defines an implicit reward function by shaping the policy’s logits to match preference probabilities according to the same BTL model as used in RLHF. Consequently, we expect DPO to inherit the same generalisation failures in OOD settings. While an empirical DPO comparison could offer additional insights, we prioritised analysing reward modelling, as RHLF remains the predominant approach in alignment and DPO is based on the same underlying assumptions.

---

Thank you for your valuable feedback, which helped us improve the paper. We hope our answer resolves your concerns, and we’d be grateful if you could reconsider your initial rating to help us make a meaningful addition to the literature.

[1] arxiv.org/abs/2102.11107

[2] arxiv.org/abs/2204.05862

[3] arxiv.org/abs/2402.00396

[4] arxiv.org/abs/1811.12359

Objective: <helpfulness / harmlessness>
User: <prompt>
Assistant: <response>