Guiding LLM Decision-Making with Fairness Reward Models

We thank the reviewer for the time and effort taken to review our manuscript. We are glad that the reviewer found our proposed algorithm to:

• Be novel and timely
• Demonstrate notable generalization capabilities, a key challenge in fairness-oriented machine learning

Also, we appreciate that they found our presentation to be clear, transparent, and reproducible.

Overall, we would like to emphasize that our proposed approach (and selection of baselines) were driven by several factors:

1. Accuracy as first order concern
2. Generalization to new decisions tasks
3. Modularity

Our empirical results validate the potential for our approach to achieve both (1) and (2), whereas existing approaches primarily serve one goal or the other. By unlocking this potential, we believe that our work makes a significant and original contribution to the field of LLM decision-making (as recognized by reviewers z2Cx and jRkp).

Below we respond to particular concerns raised in the review in more detail.

[W2/Q1: Fairness prompting baselines]

The goal in designing our algorithm was to facilitate fair decisions while incurring a minimal (or no) accuracy tradeoff and without requiring any task-specific training data. This reflects:

1. The practical need for accurate predictions
2. Legal realities around discrimination law in the US, where the existence of an equally accurate yet fairer model (i.e., a less discriminatory alternative) is a requirement for legal action against an organization accused of deploying discriminatory algorithms (Black et al., 2024).
3. The fact that many LLMs will be tasked with diverse one-off decisions for which no training data will be available (nor will training for each decision be practical).

We contend (and find experimentally) that fairness prompting will fall short of delivering the required accuracy to be considered as a viable alternative to the most accurate available algorithm (noted in Tamkin et al., 2023, and that (3) precludes the widespread use of fine-tuning base approaches. To avoid experimental bias in tuning each prompt, we used GPT-4o to produce both the base CoT prompt and the fairness prompt. No prompts in our experiments received extensive tuning or were optimized for accuracy or fairness. Additionally, we did not tune the fine-tuning hyperparameters for our FRM, instead adopting those of (Snell et al., 2025; Wang et al., 2024). These results support our claim that prompting is brittle and hurts accuracy. To further address these concerns, we have performed additional experiments on a set of stronger fairness prompting baselines.

• New experiments: To further address these concerns, we have performed additional experiments on a set of stronger fairness prompting baselines. Using the COMPAS task, we systematically evaluated 10 original fairness prompts (written by GPT-4o) alongside the 7 prompts proposed by Tamkin et al. (2023) (https://arxiv.org/abs/2312.03689) to reduce discrimination in high-stakes language model applications. Prompts were tested with and without CoT and we additionally included a CoT@32 setting using the best-performing prompt from each group. To highlight the modularity of our approach, we also apply the FRM to combine the 32 CoTs produced using fairness prompts. We report equalized odds deviation (sum of absolute FPR and TPR differences) and equalized opportunity deviation (absolute TPR difference).

• New findings: Consistent with our hypothesis and original findings, some prompts yielded reductions in equalized odds and equalized opportunity. However this came at the cost of significant reductions in accuracy. This tradeoff remained consistent across settings and accuracy was reduced further when we sampled 32 times with fairness prompts. In many cases, fairness was improved not through better reasoning but by predicting the same label for all inputs. These results reinforce our core claim that reasoning-level supervision offers a more effective and robust fairness intervention than prompting when accuracy is a primary concern. Our findings echo concerns in prior work, including Tamkin et al. regarding output distortion from fairness prompting.

• Results:

Best Settings:

[W3/Q2: Other fairness-aware techniques]

We are not aware of any existing fairness method that is directly applicable to CoT reasoning with many samples, which is the focus of our work given our goal of achieving fairness under minimal disruptions to accuracy (i.e., creating a similarly accurate, yet more fair model). While classic approaches like reweighting, adversarial debiasing and postprocessing adjustments are relevant in classification settings, they are largely orthogonal to our methods.

Also, we emphasize that we offer a framework for improving fairness on decisions and tasks that were previously unseen during model training. In many applications (including agentic ones), an LLM can be faced with many different and sometimes novel decisions, and it would not be practical or even feasible to fine-tune a different model for each decision. Thus we perform our experiments in a setting where no training data is available, and benchmark against methods that do not require training data.

Also, many such fairness techniques could be used in conjunction with our approach (see experiment above). For example, if a training intervention is proposed to improve the fairness of LLM reasoning, then our FRM could be used to further monitor and improve the fairness of the model at inference time.

If there are any specific techniques the reviewer believes are applicable to CoT-based inference, we would be grateful for the reference and happy to add more baselines.

[W1/Q3: Quality and bias of GPT-4o-mini annotations]

We appreciate the reviewer’s concern about label quality. Our use of LLM-generated labels reflects a pragmatic approach common in recent work. Weak supervision is increasingly standard for tasks requiring large-scale, subjective annotation, especially when the ground truth is ill-defined. Our intent is to demonstrate a broader idea: that fairness can be modeled at the level of reasoning, not just outcomes, and that such an approach can yield fairness gains without sacrificing (or even improving) accuracy. Our FRM implementation is a proof of concept for this.

We have updated our manuscript to explore the impact of using LLM-generated fairness data. We added a section discussing LLM annotation limitations and key failure modes from our human study. To give you an idea of this content, our qualitative study revealed a couple failure modes:

1. Group names trigger biased labels: The LLM may mark steps as biased where a group was mentioned even if the text is benign.
2. Failure to recognize implicit bias: The LLM may fail to label biased reasoning that is implicit and requires a deeper contextual understanding.
3. Reasoning is incoherent, but LLM labels it as biased: LLM annotates vague/incoherent reasoning as biased.
4. LLM annotator believes hallucinated reasoning: The model used for generating reasoning data occasionally hallucinated details based on stereotypes.

Despite these annotation issues, the aggregate performance of our FRM suggests that the model is robust to some label noise. Future work could seek to improve annotation quality.

We have also added a section with qualitative examples of FRM scores and outcomes to demonstrate how the weak-labeling approach leads to both successes and failures. Finally, we note that our dataset will be released upon publication.

[W4: Fairness metrics as normalized improvement percentages]: We appreciate the suggestion to present the improvement percentages of our FRM and will add these results in the appendix.

[Q4: Fairness metrics and group fairness]: While we agree that real-world fairness requires reasoning across multiple groups and considering different (sometimes context-specific) fairness metrics, our focus on binary-sensitive attribute settings and the equalized odds and equalized opportunity metrics follows standard practice in the literature: Zemel et al., 2013, Hardt et al. 2016, Agarwal et al., 2018, Cruz & Hardt, 2024

We believe that this serves as a controlled and standard setting for evaluating reasoning-level bias mitigation. We leave it to future work to consider how our framework can be applied to address important questions such as how to optimize for different notions of algorithmic bias and fairness.

We thank the reviewer for the time and consideration taken in reviewing our submission. We are glad that they consider our approach to be promising, and that they found the paper to be clear and easy to follow.

Below we respond to the individual concerns raised throughout the review.

[Weakness 1: Quality and bias of LLM annotations]

We appreciate the reviewer’s concern about label quality and discuss some of these concerns in our limitations section. Our use of LLM-generated labels reflects a pragmatic approach common in recent work. Weak supervision is increasingly standard for tasks requiring large-scale, subjective annotation, particularly in cases where the ground truth is ill-defined. Our intent is to demonstrate a broader idea: that fairness can be modeled at the level of reasoning, not just outcomes. The FRM is a proof of concept for this, and future work could improve on this method using higher quality supervised labels. In practice, our experiments show that even though the labels may be imperfect, they can still lead to significant downstream fairness gains.

We have updated our manuscript to more clearly address this point. Specifically, we:

• Added a section discussing the limitations of LLM annotations, examining each main failure mode observed in our human alignment study (a preview shown below):

  • Group-name triggers: Labels sometimes flagged any mention of a group as biased, even when neutral.
  • Implicit bias missed: The LLM failed to catch contextually embedded biases.
  • Misattributed bias: Reasoning that was vague or incoherent was occasionally marked as biased.
  • Hallucinated details: The reasoning model occasionally added stereotypical details not in the prompt, which the labeling model failed to flag.

• Included additional qualitative examples of FRM scores and outcomes to illustrate both the strengths and limitations of our approach under weak supervision.

• Committed to releasing our dataset of LLM labels upon publication, along with a dataset card (see Appendix) to support transparency.

If the reviewer can specify what they found unconvincing about the human study we conducted to check the alignment of LLM labels with human judgments, we would be happy to speak to those concerns.

[Weakness 2: Comparison to prompt-based approaches] Prompt-based approaches rely on the model following the prompt to improve fairness while still making accurate decisions. Given that LLMs cannot always follow prompts and respond to prompt changes in unpredictable ways (Perez et al., 2021, Webson & Pavlick, 2021, Zhou et al., 2022), in some cases this works, in other cases it doesn’t. We observed that in many cases, this can lead to the model following similar types of reasoning and giving the same answer every time to “play it safe” from a bias perspective. Our method addresses these issues by scoring reasoning steps. In this way, we don’t change the natural output diversity of the model and we provide transparency into which steps are downvoted as opposed to relying on the model to follow a fairness prompt and adjust its reasoning accordingly. Output diversity is important in these tasks because taking the majority vote over many reasoning chains has been widely shown to increase task accuracy, but if output diversity is significantly reduced, this can remove the benefits of a majority vote.

• New experiments: To further address these concerns, we have performed additional experiments on a set of stronger fairness prompting baselines. Using the COMPAS task, we systematically evaluated 10 original fairness prompts (written by GPT-4o) alongside the 7 prompts proposed by Tamkin et al., 2023 to reduce discrimination in high-stakes language model applications. Each prompt was tested with and without CoT and we additionally included a CoT@32 setting using the best-performing prompt from each group. To highlight the modularity of our approach, we also apply the FRM to combine the 32 CoTs produced using fairness prompts. We report equalized odds and equalized opportunity deviations, defined as the sum and individual differences in FPR and TPR across groups, consistent with our original paper.

• New findings: Consistent with our hypothesis and original findings, some prompts yielded reductions in equalized odds and equalized opportunity. However, this came at the cost of significant reductions in accuracy. This tradeoff remained consistent across settings and accuracy was reduced further when we sampled 32 times with fairness prompts. In many cases, fairness was improved not through better reasoning but by predicting the same label for all inputs. These results reinforce our core claim that reasoning-level supervision offers a more effective and robust fairness intervention than prompting when accuracy is a primary concern. Our findings echo concerns in prior work, including Tamkin et al., regarding output distortion from fairness prompting.

• Results:

Best Settings:

[Weakness 3: Computational cost]

The goal in designing our algorithm was to facilitate fair decisions while incurring a minimal (or no) tradeoff with accuracy and without requiring any task-specific training data. This reflects:

• the practical need for accurate predictions
• legal realities around discrimination law in the US, where the existence of an equally accurate yet fairer model (i.e., a less discriminatory alternative) is a requirement for legal action against an organization accused of deploying discriminatory algorithms (Black et al., 2024)

Given the need for accuracy, our main baselines are those that sample multiple CoTs, and our FRM is further applied to this baseline. Thus, for a CoT inference example where N chains are sampled, the cost of applying our FRM only adds the compute related to batched inference to predict a binary label for each reasoning step in the N chains. This should require fewer FLOPS (and less wall clock time) than, e.g., autoregressively producing one more reasoning chain.

[Weakness 4: Baseline selection]

Our selection of baselines was driven by:

1. Accuracy as a first order concern, given practical and legal realities
2. Generalization to new decisions without further training data

We are not aware of any existing fairness method that is directly applicable to CoT reasoning with many samples, which is the focus of our work given our constraint of achieving fairness under minimal disruptions to accuracy.

[Question 1: Impact of unfaithful reasoning]

If we understand this question correctly, the reviewer is wondering about the implications of the LLM ignoring its generated reasoning process when making the final decision. We will aim to highlight this concern further in the limitations section, where we discuss how some reasoning steps may be inconsequential to the final decision. We also highlight and analyze this issue in the previously mentioned sections exploring qualitative examples of the GPT-4 labels and test time labels and decisions.

[Question 2: Fine-tuning with RL]

This is a natural follow up step that we would be curious to see explored in future work. The benefit of our method, though, is that it can be applied out-of-the-box to any LLM and works with those that may be too unwieldy to train (particularly in an academic setting). Also, applying our model at inference is in keeping with our focus on retaining prediction accuracy.

We thank the reviewer for taking the time to review and offer feedback on our submission. Below we respond to the particular concerns as summarized in the questions section:

[Q1: Framework vs. algorithm terminology]

We understand that our somewhat interchangeable use of these terms may have caused some confusion. With this work, we aim to contribute both a conceptual framework and an example algorithmic realization of that framework. Conceptually, we offer a framework for balancing accuracy and fairness in LLM decision-making by only intervening once the reasoning and decision generating processes have been completed. We argue (and show empirically) that this offers flexible control over fairness/accuracy trade-offs that can be elusive using other existing approaches.

To demonstrate the potential benefits of applying this framework, we propose one potential algorithmic instantiation of it in Section 3, and show how it can perform in practice via experiments. Other instantiations could be imagined: for example, using strong human labels to train the FRM, or applying some weighted step combination based on estimated contribution to the final decision. In our final version of the paper, we will closely examine our usage of these terms and make sure they are appropriate to the context in which they are being used.

[Q2: Generalization claims]

Across tasks: We appreciate the reviewer’s suggestion to make sure we scope our claims to what can be supported by our experimental findings. In response to this concern, we will modify the cited claim in line 270 to “across 3 real-world tasks” (removing “a wide range of”). We will also closely check the paper for any further unsubstantiated claims (although we note that we identify no further such instances in our preliminary check).

Across LLMs: We appreciate the reviewer’s feedback that more exploration of transfer across reasoning models could further strengthen experiments. Distribution shifts may come from many factors, and thus it can be hard to thoroughly test for generalization. In this work, we chose to focus more on generalization across tasks and protected attributes than models. This is because, practically, it is easier to train one FRM per reasoning model (using that model in step 1, section 3.1) than it would be to train one FRM per task and/or group, given that most organizations will serve only a handful of LLMs, but these LLMs may face a diverse set of decisions concerning people from many different groups, and often no relevant data is available. In response to this concern, we will modify the paper to clarify the limitations of our results on LLM generalization, and also highlight that we believe the other dimensions of generalization (tasks and groups) are of more concern.

[Q3: Experimental design clarifications]

Dataset usage in different sections: We used our full set of 4 (task/dataset, attribute) pairs for our main experiment in Section 6.1. For the rest of our experiments, we included as many results as possible given our goals and the constraints on our academic compute budget and space in our submission. In Section 6.1, we aimed to test the FRM across a diverse set of tasks and demographic contexts; in Section 6.2, our focus was on model generalization, specifically to reasoning chains generated by a previously unseen LLM (Mistral). While we hope our work and open source asset releases spur further empirical research in this area, we believe that our experiments stand as a solid foundation for building on this approach to fair decision-making.

Figure formatting: The rows and columns in Figures 4 and 5 were flipped to save space in our submission. In the final version, we can unify the format for the figures given the extra space.

[Q4: Temperature parameter]

The temperature parameter is meant to control the emphasis on accuracy versus fairness in combining the final decisions. As this is a key hyperparameter in our algorithm (and the only one that needs to be chosen given a baseline of CoT w/ majority voting), we ablate its effect across three tasks and protected attributes in Figure 9, which is in appendix C because of space constraints. We believe that our ablation results indicate that this is a promising direction for flexible control of these key tradeoffs.

For each LLM tested, we set the temperature via a small amount of hand tuning (comparing among 0.01, 0.2, 0.4, and 0.8) on a single task. Since for a given LLM, a single temperature works across different tasks, this level of tuning should be possible in practice. Finally, we note that our method does not “fail” under any temperature setting in the ablation, it’s just a matter of choosing the desired tradeoff level. We will clarify this in the experiments section of our main paper. Additionally, the train-test split we use is included in Section 3.1. As noted in the paper, our dataset will be released upon publication to further support reproducibility and research.

[Q5: Editorial issues]

We thank the reviewer for carefully reading our submission and highlighting several specific editing issues. We will correct these issues in the final camera ready version of our work.

We thank the reviewer for their thoughtful and positive feedback! We are encouraged that they found our Fairness Reward Model (FRM) to be novel, an important contribution, and highly generalizable. We respond to the particular concerns raised below:

[Q1: Risk of Reflective Bias]

Excellent question! We agree that any attempt to codify fairness is inherently normative. Our goal, however, is not to identify universally unfair reasoning but to make our assumptions explicit and grounded in prior empirical findings. Specifically, our heuristic is based on prior work demonstrating that CoT applied to socially sensitive contexts can systematically amplify stereotypical associations (Shaikh, 2023). We believe that it is better to specifically name one’s fairness priors, however contingent, than to use the ones embedded deeper in the model.

The FRM is additionally constructed to be flexible: it can be retrained using a different fairness heuristic for another deployment setting. Our ultimate goal is to reduce bias on downstream tasks as measured by established fairness metrics. Our positive results on these tasks demonstrate that, while the labels may carry some normative assumptions, they are universal enough that they effectively reduce unfair results on objective downstream metrics.

[Q2: Fair Reasoning vs. Fair Outcomes]

Our results show both improved final outcomes (improved fairness while maintaining accuracy) and improved reasoning (down-weighting the contribution of unfair reasoning to the final outcome). Fair reasoning does not guarantee a fair outcome in every case; however, it increases the likelihood of fair outcomes and makes those outcomes more interpretable. Even in cases where outcome disparities persist, we think fair reasoning is an intrinsically valuable objective.

We will make this process-outcome distinction more explicit and we have added qualitative examples to the paper that highlight these edge cases. For example, we have included an instance where the model reasons fairly but chooses an outcome that contradicts that fairness.

[Q3: Surface-Level Disparities vs. Deeper Structural Harms]

We recognize the concern that a model trained to reward fairness in its reasoning may be susceptible to targeting surface-level disparities. The FRM is not designed to appear fair for the sake of appearances, instead, it is trained to disincentivize reasoning that relies on stereotypes or irrelevant demographic information. For example, a CoT that justifies its decision by referencing someone’s race when it is not relevant would receive a low score regardless of whether the final answer appears fair. We do not claim to solve all types of bias but we do believe that mitigating bias in reasoning is an important objective. We see process-supervision as a complementary strategy to interventions that operate at other levels such as data augmentation and causal modeling.

[Q4: Generalization to other fairness notions]

We share the view that generalizing to different fairness definitions is critical. We would like to emphasize that our method does not optimize for equalized opportunity and/or equalized odds. We use these only as evaluation metrics, based on their prominence in the ML fairness literature. Our algorithm should be able to be adapted to improve outcomes according to most popular fairness definitions of machine learning fairness. We see this adaptability as a strength of the approach of using process-level fairness supervision. We thank the reviewer for highlighting this important issue, and we will update our final paper to emphasize this point.

[Q5: Cases where FRM leads to worse outputs]

FRM scores only reflect fairness, which we agree is not sufficient for selecting the highest quality outputs overall. To address this, we introduce a temperature parameter that balances optimization for accuracy with the FRM score. One might also imagine other ways for integrating our FRM score into final output selection, for example considering it alongside some reward model in a chatbot setting.

[Q6: Limitations regarding demographic parity]

We agree that demographic parity is not universally appropriate and we wish to clarify that the FRM does not explicitly enforce any particular fairness metric. Demographic parity is a standard evaluation metric for ML fairness that we used to demonstrate the FRM’s impact on downstream performance. The FRM is agnostic to how fairness is measured.