On Optimal Steering to Achieve Exact Fairness

Thank you for your careful reading and valuable feedback. We answer your questions below.

1. Could you provide the trade-off results on accuracy?: On page 12 of the supplementary material, Proposition 2.5 shows how to bound the change in accuracy in terms of the KL divergence between the original distribution and the ideal distribution. Interestingly, this change can be a drop in accuracy in the worst case, but could also be a gain in the best case, unlike the accuracy traded off for fairness when we do not change the underlying data distribution.

We also comment on the some points raised in the Weakness section below:

1. Most theoretical results of finding in the main paper are based on Gaussian distribution. And the conditions are only sufficient for multivariate Gaussian. Also, the discussed classifiers are Bayes optimal ones. I understand it is somehow standard in theoretical papers, but not sure of the practical applicability. Is it possible to discuss more general distributions?: You are correct in pointing that out. The fairness of the Bayes optimal classifier, and the corresponding optimization programs to search for the ideal distributions, are easier to mathematically analyze for parametric families of distributions, e.g., multivariate Gaussians. Multivariate Gaussians may not always fit real-world data but can be justifiably used to model concepts and internal representations in machine learning, as shown in prior work [3,4]. For future work, we can move towards general distributions by considering a class of distributions with bounded moments [5] or working with invertible transformations to parametric families [6].

2. Theorem 4.1 and Prop 4.3 demonstrate how we can find $D$, but the discussions on accuracy guarantee are deferred to Appendix (prop 2.5). However, this trade-off can be very important while understanding whether the preprocessing methods should be used in practice. I suggest discussing this more in the main paper.: We will definitely include it in the main manuscript in further revisions of the paper and discuss the implications of this result.

3. Similarly, in the experiment results of section 6, it would be better to show the accuracy comparison (though it is mentioned that accuracy of EF Affirmative is at the same level of other methods). Please find the accuracy numbers for all the methods in Section 6.1, where we reduced the disparity in multi-class classification -> Before steering: 0.789, Mean Matching: 0.788, MiMiC (Mean+Cov Matching): 0.793, LEACE: 0.788, EF Affirmative: 0.779. We will also include them in the main manuscript in further revisions.

4. Minor: I think the literature on long-term fairness can also be related to this work. For example, "steering" and long-term fairness in performative prediction happens naturally when agents respond to the decision-maker [1,2]. It would be interesting to consider whether your preprocessing method can also benefit long-term fairness.: This is certainly a very interesting direction, and we will definitely comment on this as a potential future work in further revisions. Thank you so much for suggesting this direction.

We greatly appreciate your positive feedback and request you to champion our paper for acceptance.

[1] Hardt M, Jagadeesan M, Mendler-Dünner C. Performative power[J]. Advances in Neural Information Processing Systems, 2022, 35: 22969-22981.

[2] Jin, Kun, et al. "Addressing Polarization and Unfairness in Performative Prediction." arXiv preprint arXiv:2406.16756 (2024).

[3] Zhao, Haiyan, et al. "Beyond single concept vector: Modeling concept subspace in llms with gaussian distribution." arXiv preprint arXiv:2410.00153 (2024)

[4] Eftekhari, Daniel, and Vardan Papyan. "On the Importance of Gaussianizing Representations." arXiv preprint arXiv:2505.00685 (2025).

[5] Murti, Chaitanya, and Chiranjib Bhattacharyya. "DisCEdit: Model Editing by Identifying Discriminative Components." Advances in Neural Information Processing Systems 37 (2024): 47261-47296.

[6] Theisen, Ryan, et al. "Evaluating state-of-the-art classification models against bayes optimality." Advances in Neural Information Processing Systems 34 (2021): 9367-9377.

Thank you for carefully reading our paper and providing your valuable feedback. We address your concerns below.

1. On the definition of the ideal distribution, I don't understand why DP can also be incorporated. As the optimal classifier and DP can be in conflict.: We clarify the definition of ideal distribution and correct the typos you pointed out in lines 127 and 148. Definition 2.1 for Demographic Parity (line 122) and lines 126-127 should read as: $\Delta_{DP, y}(h, D) = \max_{a, a' \in \mathcal{A}} |Pr(h(X, A)=y|A=a) - Pr(h(X, A)=y|A=a')|.$ We can explicitly define $l_C$ in line 136 as: Let $C \in \mathbb{R}^{|\mathcal{Y}| \times |\mathcal{Y}|}$ be a matrix that defines the cost-sensitive loss function $l_{C}$ of predicting label $i$ when the true label is $y$ as $l_C(h(x,a), y) = c_{h(x, a) y}$, the $(h(x, a), y)$-th entry of $C$. The cost-sensitive risk of a classifier $h$ is then given by $E_{D}[l_{C} (h(X,A), Y)]$. Finally, Definition 3.1 should be: Let $D$ be a distribution over $\mathcal{X} \times \mathcal{A} \times \mathcal{Y}$. Let $\mathcal{H}$ be a hypothesis class of group-aware classifiers $h: \mathcal{X} \times \mathcal{A} \rightarrow \mathcal{Y}$, and let $h^* = \underset{h \in \mathcal{H}}{\arg\min}~ E_{D}[l_{C}(h(X,A), Y)]$, where $(X,A,Y) \sim D$. We call $D$ an ideal distribution if $\Delta_{DP, y} ( h^{\*} , D)=0$, for all $y \in \mathcal{Y}$ (similarly, $\Delta_{EO, y}(h^{\*}, D)=0$, for all $y \in \mathcal{Y}~$). Propositions 3.3 and 3.2 show conditions under which many univariate and multivariate Gaussians provably give ideal distributions for Equal Opportunity (i.e., equal TPRs across groups). The exact same idea works for equality of FPRs and gives equality of positive rates across groups, or equivalently, Demographic Parity (DP).

2. How is it possible to make the prediction both fair and accurate?: This observation is not new, as several prior works demonstrate that the fairness-accuracy trade-off disappears when data bias is accounted for [3,4,5,6,7,8]. Our Theorem 4.1 essentially gives an algorithmic recipe to minimally change a given data distribution and provably achieve this.

3. Does it mean the previous Pareto frontier works [1, 2] are not meaningful?: Pareto-frontier or fairness-accuracy trade-offs are meaningful when the underlying data distribution is fixed. [1,2] maximize accuracy subject to various fairness constraints to explore this trade-off, but do not change the data distribution.

We will now address some of the points raised in the Weakness Section.

1. Presentation lacks rigor. For example, line 122, 148, 127 make it very hard to check the correctness.: Thank you for pointing out these typos. We will correct all of them and have explained our corrections above in our response to your question 1. Our proofs are correct, and I hope our explanation helps you verify the same.

2. I don't believe real-world data can fit such a condition [as in line 171].: We do not model real-world data using Gaussians. Our empirical work applies Gaussian modeling for concepts and internal representations where it is justified by previous work [9,10]. The condition in line 171 can be simplified further to f-divergence as follows. If $\Sigma_{ia} = \Sigma_a$ (same group covariance for all classes), then our condition is equivalent to having the same f-divergence between any two class-conditional distributions across different groups. In other words, $\Delta_{\Sigma_{a}}^{2}(\mu_{ia}, \mu_{ja})= \Delta_{\Sigma_ {a'}}^{2}(\mu_{ia'}, \mu_{ja'})$, $\forall i,j \in \mathcal{Y}, a,a' \in \mathcal{A}$; see f-divergence formula (P1) near the end of page 2 in arxiv: 2204.10952. f-divergence is commonly used in fair classification and generation.

3. It also assumes univariate Gaussian on condition in Proposition 3.3, Theorem 4.1. It is an oversimplification of the data distribution. Theorem 4.1 and Proposition 3.2 are stated for multivariate Gaussian distributions. Proposition 3.3 and Corollary 4.2 for univariate Gaussians are only to help the reader as special cases.

4. The theoretical part is based on a Bayesian optimal classifier. I don’t quite understand the experiment, which involves an LLM-based representation steering task.: Our theoretical results are on the fairness of the Bayes optimal classifier, which is easier to mathematically analyse for parametric families of distributions, e.g., multivariate Gaussians. Multivariate Gaussians may not always fit real-world data, but can be justifiably used to model concepts and internal representations in machine learning, as shown in prior work [9,10]. Hence, we felt that LLM-based representation-steering makes a compelling application for our technique.

5. Did the author assume that Test and Training are IID?: Yes. It is a standard assumption in Machine Learning.

We sincerely thank you for reading our paper carefully. We request you to kindly revisit our paper with our detailed explanations above and upgrade your rating.

[1] Kim, Joon Sik, Jiahao Chen, and Ameet Talwalkar. "Fact: A diagnostic for group fairness trade-offs." International Conference on Machine Learning. PMLR, 2020.

[2] Xian, Ruicheng, Lang Yin, and Han Zhao. "Fair and optimal classification via post-processing." International conference on machine learning. PMLR, 2023.

[3] Wick, Michael, and Jean-Baptiste Tristan. "Unlocking fairness: a trade-off revisited." Advances in neural information processing systems 32 (2019).

[4] Blum, Avrim, and Kevin Stangl. "Recovering from biased data: Can fairness constraints improve accuracy?." arXiv preprint arXiv:1912.01094 (2019).

[5] Dutta, Sanghamitra, et al. "Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing." International conference on machine learning. PMLR, 2020.

[6] Maity, Subha, et al. "Does enforcing fairness mitigate biases caused by subpopulation shift?." Advances in Neural Information Processing Systems 34 (2021): 25773-25784.

[7] Sharma, Mohit, and Amit Deshpande. "How far can fairness constraints help recover from biased data?." arXiv preprint arXiv:2312.10396 (2023).

[8] Leininger, Charlotte, Simon Rittel, and Ludwig Bothmann. "Overcoming Fairness Trade-offs via Pre-processing: A Causal Perspective." arXiv preprint arXiv:2501.14710 (2025).

[9] Zhao, Haiyan, et al. "Beyond single concept vector: Modeling concept subspace in llms with gaussian distribution." arXiv preprint arXiv:2410.00153 (2024)

[10] Eftekhari, Daniel, and Vardan Papyan. "On the Importance of Gaussianizing Representations." arXiv preprint arXiv:2505.00685 (2025).

Thank you for carefully reading our paper and providing valuable feedback. We address your concerns below.

1. This work is limited to common parametric families. What kind of applications still work under this assumption, and what others do not? A couple of examples would help. Although the checklist mentions that limitations are discussed in Section 7, this is not clearly stated (on the other hand, the future work is well discussed).: Our theoretical results are on the fairness of the Bayes optimal classifier and the corresponding optimization programs, which are tractable and mathematically easier to analyse for parametric families of distributions, e.g., multivariate Gaussians. Multivariate Gaussians may not always fit real-world data but can be justifiably used to model concepts and internal representations in machine learning as shown in prior work [1,2]. There can be scenarios and applications where a parametric assumption may not fit well with the input data distribution, and hence, analyzing the Bayes optimal fair classifier will become highly non-trivial. We will discuss these limitations more elaborately in future versions of the manuscript. Thank you for this feedback.

2. The authors mention the framework of ideal distributions can also be applied to audit deployed model for fairness, particularly when the evaluation data may itself be biased. Beyond the steering showed, how can this be achieved in practice? Use the KL gap as a proxy for what’s needed to debias the model? It's not trivial to understand from the text.: Prior work on fairness audit has mostly focused on auditing a given model instead of auditing training or evaluation data. There is a long line of work to demonstrate that the fairness-accuracy trade-off disappears when data bias is accounted for [4,5,6,7,8,9]. The KL gap in our formulation can be used as a metric of data bias and our Theorem 4.1 essentially gives an algorithmic recipe to minimally change a given data distribution to bridge this gap.

3. In the future work the authors mention normalising flows or mixture models as possibly integrated. Would this be a relaxation .... or would this have more profound implications?: Thisen et al. [5] showed that the Bayes error is invariant to invertible transforms, like the ones used with Normalizing flows. Furthermore, they show how to obtain an entire family of distributions using a single family of flows. The ideas presented in this paper can be used to construct invertible maps to a desired parametric family, apply our ideas of searching for ideal distributions within the available family of distributions, and then use the map back using the invertible transformation. This is our plan for future work.

4. It’s not clear if the framework .... one binary sensitive attribute.: Proposition 3.2 is presented for a multi-class, multiple attribute, and multivariate Gaussian setting. To show a stronger necessary and sufficient set of results, we make use of the binary label and sensitive attribute setup for Proposition 3.3. Furthermore, even for the Affirmative action intervention in Theorem 4.1, we can write down a multi-class and multiple sensitive attributes version. In fact, that is what we used to set up experiments for Section 6.1 (Multi-class debiasing) and the details of this is described in Section 4.1 of the supplementary material. One thing to note is that Affirmative action needs a non-trivial definition for situations beyond binary labels and binary sensitive attributes, since we don't have the notion of a favourable group and a positive outcome anymore. That is why we can either define Affirmative action class-wise, assuming the most "accurate" group to be favourable, or we can do it the other way around, where we first find the most favourable group and then find ideal distributions relative to that fixed group. This is what we did for the results in Section 6.1, where we worked with a multi-class, binary attribute setup, but the same setup can be perfectly used for multiple sensitive attributes.

Below, we comment on some of the points raised in the Weaknesses section.

1. The paper is dense in some .... at this level: We will carefully review all notations and make the exposition easier in the next version of this manuscript. More discussion about the experiments in Section 6.2 is included in Section 4.2 of the Supplementary material. We will add more details in the main text in future revisions.

2. Abstract: initial sentence is a strong .... seem repetitive: The scope of this work is exact group fairness, and hence we wanted to be very precise in the abstract. We can refine the abstract to highlight this. Regarding the repetitiveness, we wanted to emphasize in the first sentence that we want to improve the representations used for downstream classification. In the second sentence, we wanted to express that we are also able to steer the internal representations of an LLM so that we can improve generation across groups in LLMs. We will make these clarifications in future versions of this manuscript.

3. The optimization is for an ideal distribution, but some text parts give space to there being more of such distributions, although it is not clearly discussed.: Our aim in Section 4 is to propose an optimization program to obtain an ideal distribution where the Bayes optimal classifier is also fair. The objective is to find the optimal ideal distribution that minimizes the KL divergence relative to the given distribution.

4. In Definition 2.1 .... positive rate difference?: Thank you for pointing that out. We will fix that typo in future versions of the manuscript.

5. It’s not really clear in Section 6.2 how the simulation results are obtained exactly.: Due to space constraints, we have laid out the details of the experiments in Section 6.2 in Section 4.2 of the supplementary material, along with all the prompts and a flow of the pipeline. To summarize, we observe a disparity in performance between groups when we measure the effectiveness of steering generation of joyful responses, using the method a recent work on modeling concepts using gaussian distributions [1]. We therefore applied our Affirmative intervention from Theorem 4.1 to obtain the set of ideal distributions for the concepts such that both groups in the data get steered effectively during inference. We use the obtained ideal distribution to nudge the optimal steering vector towards better generation for both groups.

6. Some details are not clear on the optimization procedure in practice..... intervention?.: For Section 6.1, the downstream classifier is not retrained. For the setup in Section 6.1, we are steering the distribution of representations towards an ideal distribution, and then training a classifier without any constraints. As indicated in Figure 3, this improves the fairness across multiple professions and for some professions, even matches and beats the Affine Steering method from Singh et al. [10].

7. Also, the alpha value influence ... not discussed. As stated in the paper, for the experiments in Section 6.2, alpha is required because after our intervention, we get the ideal subgroup distributions where the tradeoff between utility and fairness is alleviated. This notion of utility is related to, but different from, accuracy, for which we have ideal distribution guarantees in Theorem 4.1. Therefore, the newly obtained steering vectors can be drastically different than the optimal steering vectors (as demonstrated in Figure 4, where a high value of alpha closer to 1 puts more weight on the affirmative action vector and therefore cannot steer towards joyful generation). But we can use our obtained ideal distribution as a direction to nudge the optimal steering vector. As indicated by Figure 4, this successfully steers the text generation towards joyful sentiment for the underperforming group. We have laid out more details of the steering experiment in Section 4.2 of the supplementary material, and if you recommend, we can also add a pseudocode of the algorithm for more clarity in future revisions of the manuscript.

We sincerely thank you for reading our paper carefully. We request that you kindly revisit our paper with our detailed explanations above and upgrade your rating.

[1] Zhao, Haiyan, et al. "Beyond single concept vector: Modeling concept subspace in llms with gaussian distribution." arXiv preprint arXiv:2410.00153 (2024)

[2] Eftekhari, Daniel, and Vardan Papyan. "On the Importance of Gaussianizing Representations." arXiv preprint arXiv:2505.00685 (2025).

[3] Theisen, Ryan, et al. "Evaluating state-of-the-art classification models against bayes optimality." Advances in Neural Information Processing Systems 34 (2021): 9367-9377.

[4] Wick, Michael, and Jean-Baptiste Tristan. "Unlocking fairness: a trade-off revisited." Advances in neural information processing systems 32 (2019).

[5] Blum, Avrim, and Kevin Stangl. "Recovering from biased data: Can fairness constraints improve accuracy?." arXiv preprint arXiv:1912.01094 (2019).

[6] Dutta, Sanghamitra, et al. "Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing." International conference on machine learning. PMLR, 2020.

[7] Maity, Subha, et al. "Does enforcing fairness mitigate biases caused by subpopulation shift?." Advances in Neural Information Processing Systems 34 (2021): 25773-25784.

[8] Sharma, Mohit, and Amit Deshpande. "How far can fairness constraints help recover from biased data?." arXiv preprint arXiv:2312.10396 (2023).

[9] Leininger, Charlotte, Simon Rittel, and Ludwig Bothmann. "Overcoming Fairness Trade-offs via Pre-processing: A Causal Perspective." arXiv preprint arXiv:2501.14710 (2025).

[10] Singh, Shashwat, et al. "Representation surgery: Theory and practice of affine steering." arXiv preprint arXiv:2402.09631 (2024).

Thank you for carefully reading our paper and providing valuable feedback. We address your questions regarding our work in the context of existing fair classification works below. We will certainly include this discussion in future versions of this manuscript.

1. Relation to fair representation learning and benefit to practitioners: We study the foundational problem of finding the nearest ideal distribution to a given distribution. This has many potential applications, such as correcting training data bias, learning fair representations, steering intermediate representations for fair generation, etc. Fair pre-processing or reweighing is closer to our mathematical setup, and steering intermediate representations gives a compelling application for LLMs. Our KL optimization program can be used to measure training data bias and guide data collection policies in practice, especially when the models are trained for fairness, but the inherent bias in data is unresolved. Prior to our work, the popular strategies to correct data bias were reweighing, fair pre-processing, and collecting more data from underrepresented groups.

2. Going beyond Distribution-level analysis and working with finite samples: This is certainly an important future direction for us. A distributional-level analysis allows us to express the Bayes error and true positive rates (and analogously false positive rates) as functions of the distribution parameters. These are generally intractable without any distributional assumptions. In a finite-sample regime, we have high probability estimates of the moments and other distribution-dependent quantities, so our approach leads to approximate fairness guarantees. Since the scope of our paper is exact fairness, we did not include this further analysis. To go beyond distributional assumptions, we can leverage previous work that maps given data distribution to tractable, parametric families using invertible transforms that preserve Bayes error, so our results become applicable [1]. We can also assume distributions with bounded moments instead of parametric families, and still bound the Bayes error and other relevant quantities [2]. This requires a careful analysis, and is certainly a very important future direction for us.

We greatly appreciate your positive feedback and request that you champion our paper for acceptance.

[1] Theisen, Ryan, et al. "Evaluating state-of-the-art classification models against bayes optimality." Advances in Neural Information Processing Systems 34 (2021): 9367-9377.

[2] Murti, Chaitanya, and Chiranjib Bhattacharyya. "DisCEdit: Model Editing by Identifying Discriminative Components." Advances in Neural Information Processing Systems 37 (2024): 47261-47296.