Understanding Fairness and Prediction Error through Subspace Decomposition and Influence Analysis

We thank the reviewer for recognizing that our work is novel, well-motivated and grounded in solid theoretical foundations, and we appreciate the constructive feedback. We will address the remaining clarifications below point by point.

Weakness

Since the reviewer raised the same concerns in both the weaknesses and questions sections, we provide a consolidated response under the questions section only.

Questions

Q1: Our framework does not assume any specific structure for $Z$ in the model assumptions (Equation (3)). In particular, $Z$ is not restricted to being binary; the SDR framework naturally allows $Z$ to be scalar or vector-valued, and either categorical or continuous. However, since most existing fairness evaluation metrics (e.g., EO or DP) are defined for binary sensitive attributes, and most benchmark datasets contain only binary sensitive attributes, our experiments focus on that setting.

Q2: The computational cost of obtaining the fair projection matrix primarily arises from estimating the candidate matrices $M_Y$, $M_Z$ and $M_{Y,Z}$, each typically constructed as a weighted covariance matrix. These computations scale linearly with the sample size $n$. In addition, the procedure includes an eigen-decomposition step for $p\times p$ matrices, and the rank estimation step scales linearly with dimension $p$.

Q3: The finite sample analysis would be extremely challenging due to the chain of mappings described in Section 5.2, line 233. Existing theoretical results show that each individual step in the process achieves a $1/\sqrt{n}$ convergence rate to the population level, allowing us to drop higher-order terms in asymptotic analysis.

However, for a finite sample analysis, it would require us to carefully characterize the error propagation across multiple stages of the estimation pipeline. This includes quantifying the cumulative effect of the estimation errors in the SDR subspaces, the projection matrices, and the final predictive model. Such an analysis would involve bounding the composition of errors and developing non-asymptotic guarantees that depend explicitly on the sample size, dimensionality, and possibly the eigenvalue gaps in the estimated subspaces. We recognize this as an important and nontrivial direction for future work.

Q4: Our setting already accommodates intersectional fairness. When considering multiple sensitive attributes, we can simply concatenate them to form a multidimensional $Z$ without altering the model assumptions in Equation (3). This requires no changes to our framework or estimation procedure, and all asymptotic results remain valid. We will clarify and illustrate this point in the revised manuscript.

We appreciate your feedback, which has helped strengthen our work, and we hope our responses have addressed all of your concerns. Please let us know if there is any additional information or clarification we can provide to further support your understanding and confidence in our work.

First, we thank the reviewer for recognizing that our work is well-motivated and grounded in solid theoretical foundations, and we appreciate the constructive feedback.

Moreover, we would like to respectfully clarify that the primary concern regarding our use of linear SDR methods has already been acknowledged in the Limitations section. As discussed below, our work represents the first principled attempt to address the fairness–utility trade-off through a theoretically grounded representation learning framework, directly targeting key gaps in existing methods, namely, the lack of explicit control over fairness–accuracy trade-offs and the absence of theoretical guarantees. As is often the case with methodological contributions, it is expected that subsequent work will address the limitations of the initial framework. Indeed, we fully agree that extending the framework to nonlinear and large-scale domains (e.g., image or text data) would be a valuable direction for future work; we view these as natural and promising evolutions of our approach rather than shortcomings of the current contribution.

We will address the remaining clarifications below point by point.

Weakness

W1: We thank the reviewer for pointing us to these additional references, which are highly relevant. However, we would like to clarify that our method differs significantly from these prior works, as outlined below:

First, prior methods such as Creager et al. (2019) and Louizos et al. (2015) lack explicit control over the fairness–accuracy trade-off. These approaches impose fairness constraints through disentangled variational autoencoders and rely on tuning hyperparameters to achieve different levels of fairness and utility. However, they do not offer explicit guarantees or clear interpretability regarding how these trade-offs are managed. In contrast, our method is grounded in a statistical decomposition of the covariates $X$ into components that are predictive of the target label $Y$ and either orthogonal to or shared with the sensitive attribute $Z$. This results in a representation that is both interpretable and transparent on the information discarded and retained, allowing for explicit and principled control over the fairness–accuracy trade-off.

Second, these prior methods rely on optimization-based frameworks involving variational autoencoders with adversarial or mutual information-based regularization. Such approaches are often sensitive to hyperparameter tuning and prone to convergence issues. In contrast, our approach requires no model training, and is supported by theoretical guarantees, thereby avoiding the instability, lack of transparency, and limited interpretability associated with adversarial learning.

CausalVAE (Yang et al., 2021), while conceptually related in its goal of learning disentangled and causally interpretable representations, is not directly focused on fairness. Its framework is aimed at generative modeling through causal factorization and intervention, rather than fair prediction. When applied to fairness settings, CausalVAE would likely encounter the same two limitations noted above: lack of explicit fairness–utility control and reliance on complex optimization procedures.

We will incorporate this discussion into the related work section of the revised paper.

W2: We appreciate the reviewer’s insightful comment. It is true that our approach is built on a linear sufficient dimension reduction (SDR) framework, and as discussed in the Limitation, this assumption may limit applicability in highly nonlinear settings. However, our primary goal in this work is to develop a principled and interpretable framework for balancing fairness and utility through explicit statistical information decomposition.

In line with our theoretical contribution, we evaluated the method on two widely used tabular datasets that allow for clear, interpretable assessments of fairness–utility trade-offs and are commonly used in fairness research. While our current implementation focuses on structured tabular data, we agree that evaluating the method on higher-dimensional and unstructured domains (such as CelebA or BiasBios) would be valuable. We view this as an important next step, and note that the subspace decomposition ideas we introduce could be extended using nonlinear SDR techniques (e.g., kernel or deep SDR), which we plan to explore in future work.

Non-linear SDR techniques are indeed promising and are expected to outperform linear SDR methods, particularly on complex datasets such as those involving images and text. A straightforward extension would be to perform fair representation learning using neural network architectures that explicitly decorrelate the learned representations from sensitive variables. However, analyzing such models theoretically poses greater challenges. We believe this is a highly promising direction and plan to explore it in future work.

W3: We are unclear about which sense of efficiency the reviewer is referring to, as statistical and computational efficiency are conceptually distinct.

If the comment concerns statistical efficiency, this typically refers to the convergence rate or asymptotic variance of an estimator. Our method achieves a convergence rate of $1/\sqrt{n}$ for all subspace estimators and post-SDR parameters. In contrast, most deep learning-based approaches lack theoretical guarantees on convergence or asymptotic behavior, making direct statistical efficiency comparisons infeasible.

If the concern is about computational efficiency, we note that our method avoids any optimization during subspace estimation. It relies only on matrix multiplications and closed-form calculations, enabling efficient execution on CPUs without the need for GPU acceleration. For instance, on the Adult and Bank datasets, our method takes 80.47 and 50.22 seconds, respectively, using only CPU. In comparison, deep learning-based methods such as AdvDebias, FairMixup, DRAlign, and DiffMCDP require iterative training and take approximately 110 and 90 seconds, while RLACE takes 155 and 127 seconds, respectively. This highlights that our method is more lightweight and accessible in terms of computational resources.

Questions

Q1: See our reply to W3.

Q2: Yes, our method can be applied to high-dimensional settings, as SDR techniques are designed to extract low-dimensional structure from high-dimensional data. The performance of our approach depends on the structure of the original representation, and since our framework is limited to linear transformations, it may not fully capture non-linear dependencies. To address this, one could extend our method using non-linear SDR techniques such as kernel-based methods (as discussed in the appendix), or adopt information-theoretic approaches (e.g., mutual information) to better model non-linear relationships.

We acknowledge that in complex settings, such as image and text embeddings, the assumptions of linear SDR may not hold, where our method may not be the most effective one. Nonetheless, the core idea of subspace disentanglement remains applicable. Developing a theoretical and interpretable non-linear extension is a promising yet challenging direction, which we will highlight in the revised manuscript.

We appreciate your feedback, which has helped strengthen our work, and we hope our responses have addressed all of your concerns. Please let us know if there is any additional information or clarification we can provide to further support your understanding and confidence in our work.

First, we thank the reviewer for recognizing that our work is well-motivated and grounded in solid theoretical foundations, and we appreciate the constructive feedback.

Moreover, we would like to respectfully clarify that the primary concern regarding our use of linear SDR methods has already been discussed in the Limitations section. As discussed below, our work represents the first attempt to address the fairness–utility trade-off through a theoretically grounded representation learning framework, closing the gaps in existing methods: lack of explicit control over fairness–accuracy trade-offs and the absence of theoretical guarantees. We fully agree that extending the framework to nonlinear setting would be a valuable direction for future work; we view these as natural and promising evolutions of our approach rather than shortcomings of the current contribution.

We will address the remaining clarifications below point by point.

Weakness

W1: We appreciate the reviewer’s comments that our work shares a common tool, i.e., Sufficient Dimension Reduction, with Shi et al. (2024), and that related add-back methods also aim to achieve an intermediate level of fairness by balancing fairness and predictive accuracy. However, our contribution differs from these approaches in two important ways:

1. First, unlike Shi et al. (2024), which completely removes the influence of the sensitive attribute $Z$ from the representation of covariates $Z$ without leveraging label information $Y$, our method addresses a fundamentally different problem setup. We adopt a supervised approach that explicitly incorporates $Y$ to decompose the predictive information in $X$ into two components: (i) the part that is orthogonal to $Z$, and (ii) the part that is shared with $Z$. This label-informed decomposition enables more effective identification of useful predictive signal.

2. Second, while existing ``add-back’’ heuristics attempt to recover lost utility after fairness constraints, they are typically ad hoc and lack theoretical guarantees. In contrast, our approach formalizes the decomposition of shared and orthogonal information among $X$, $Y$ and $Z$, resulting in a method that is both interpretable and supported by statistically theory. And importantly, our principled formulation enables one to explicitly tune the fairness–accuracy trade-off while providing transparency on retained versus discarded information into the learned.

W2: You are right that our proposed method is built on linear sufficient dimension reduction (SDR) projections, and as discussed in the Limitation section, violations of this assumption may affect its effectiveness. However, quantifying the degree of departure from this assumption is a non-trivial problem and depends on several factors, including the data distribution, signal-to-noise ratio, and the nature of the downstream task, as widely acknowledged in the SDR literature. We will clarify this point further and acknowledge it as an important direction for future research.

W3: We appreciate the reviewer’s comment. Given the theoretical and methodological focus of our work, our goal in this paper is to present a principled framework rather than to conduct extensive, large-scale benchmarking. As is common in theory-oriented papers, we validate our approach on representative tabular datasets to illustrate the key properties and practical performance of the method. While we agree that evaluating on large-scale or pre-trained models would further support the broader narrative, we view this as a natural next step and plan to pursue such empirical validation in future work to complement the current theoretical contribution.

W4: We thank the reviewer for this thoughtful observation. However, we would like to clarify that our method does not require the use of any accuracy threshold. The 95% threshold is chosen to ensure a fair comparison across different methods. All methods and results reported in Table 1 follow this same criterion for hyperparameter tuning. Specifically, this threshold allows us to assess how fair the trained models are when maintaining a reasonable level of predictive performance. Users can easily adjust this threshold based on their specific tolerance for accuracy sacrifice in exchange for fairness.

Regarding the feature-by-feature loop, it is designed to systematically identify and incorporate informative components by allowing a controlled level of unfairness, thereby achieving an intermediate point in the fairness-utility trade-off. This loop functions as a tuning parameter with improved interpretability. Unlike optimization-based methods that also require tuning a penalty coefficient $\lambda$, our approach offers explicit control and provides two main advantages: (1) the shared dimension is selected from a finite set of values, avoiding the need for grid search over a continuous $\lambda$ range, and (2) the interpretability of each added component allows practitioners to better understand the trade-off dynamics.

Importantly, the number of shared dimensions is determined by the degree of entanglement between $Y$ and $Z$, rather than the dimensionality of the representation space. Thus, even in high-dimensional settings, the number of shared components may remain small.

W5: We thank the reviewer for pointing out the repetitions, typos, and the placement of key observables in the appendix. We will carefully revise the paper to eliminate redundant text and correct all typos. We will move key summaries and metrics into the main text and clarify their relevance to the evaluation of our method.

Questions

Q1: Conditional independence tests can be employed to assess whether the estimated central subspaces are sufficient to capture all relevant information. For example, one can statistically test whether $Y \perp X \mid M_Y X$ and $Z \perp X \mid M_Z X$ hold. A number of existing works focus on this problem, such as the paper ``Kernel-based Conditional Independence Test and Application in Causal Discovery'' (UAI 2011).

If the linear SDR assumptions are strongly violated, then the estimated central subspaces may not adequately represent the conditional structure, and the matrices $M_Y$ and $M_Z$ may become full rank (i.e., rank $p$), indicating the need to include all dimensions. In such cases, we can still manually choose how many directions to retain in the post-SDR training based on the ordering of the eigenvalues, and form a fair projection accordingly.

Q2: Our theoretical analysis and fairness utility trade-off does not rely on this 95% threshold. It is intended solely to ensure a fair comparison between different methods. We will clarify this in our revised version.

Q3: The computational cost of obtaining the fair projection matrix primarily arises from estimating the candidate matrices $M_Y$, $M_Z$ and $M_{Y,Z}$. These computations scale linearly with the sample size $n$. In addition, the procedure includes an eigen-decomposition step for $p\times p$ matrices, and the rank estimation step scales linearly with the ambient dimension $p$. For reference, the computation times for obtaining the all SDR subspaces $M_Y$, $M_Z$ and $M_{Y,Z}$ and their ranks on the Adult and Bank datasets are 80.47 seconds and 50.22 seconds only use CPU, respectively. We did not conduct additional experiments on large scaled image and contextual datasets as our work focuses on providing theoretical analysis and explanation, but it is an interesting task worth exploring for nonlinear SDR setting in the future work.

Q4: Theorems 5.2--5.4 provide practitioners with statistical uncertainty quantification results for parameters estimated using the fair representation as well as their impact on predictive performance. These results enhance the interpretability of our framework by elucidating how the choice of dimension and the specific SDR technique employed influence the resulting parameter estimates. The practitioners can also benefit from constructing confidence intervals for the estimated parameters.

Q5: The purpose of our simulation and real data experiments is to demonstrate that our proposed framework is effective in practice and that the underlying idea is sound. Our main contribution lies in the theoretical analysis and the insights it provides into the fairness-utility trade-off through the lens of subspace interactions.

In more complex scenarios, such as those embeddings with intricate non-linear relationships, our linear SDR methods may be less effective in producing fair representations. Nonetheless, the core idea behind our approach remains applicable. By extending our framework to incorporate non-linear structures, e.g., through kernel-based SDR or information-theoretic formulation, we aim to close this gap in future work.

Q6: Although the value of TPR gap is low for both the Adult and Bank datasets when using logistic regression, the DP and MCDP gaps remain high, indicating substantial group-level unfairness in terms of both expectations and distributions. Therefore, simply using post-hoc methods, such as threshold adjustment, can not address the underlying fairness issues.

Q7: The number of slices is chosen as $p+1$, as stated in Section B.1. The parameters for rank estimation follow the same settings as presented in the original paper. For post-SDR training, we apply the baseline logistic regression model directly on the fair representations for both Adult and Bank datasets. We will clarify these implementation details in the revised version of the manuscript.

We appreciate your feedback, which has helped strengthen our work, and we hope our responses have addressed all of your concerns. Please let us know if there is any additional information or clarification we can provide to further support your understanding and confidence in our work.

We thank the reviewer for recognizing that our work is well-motivated and grounded in solid theoretical foundations, and we appreciate the constructive feedback. We will address the remaining clarifications below point by point.

Weakness

W1: We do not claim that predictive approaches are inadequate or must be entirely replaced. Instead, our contribution lies in offering additional insights into achieving intermediate levels of fairness and providing interpretability for the fair representations we construct. While the predictive setting is the most commonly used approach for achieving fairness, typically by incorporating a fairness penalty term during training, it often suffers from optimization instability. This is because the prediction loss and the fairness loss can be inherently conflicting, leading to convergence issues and fluctuating training dynamics. Moreover, predictive fairness methods are usually task-specific: if one wishes to apply the learned model to a different downstream task or to enforce a different type of fairness constraint, it often requires retraining the model from scratch.

In contrast, our approach focuses on representational fairness, which seeks to remove information about the sensitive attribute directly from the learned representation. A key advantage of this approach is that it offers a clear and interpretable trade-off between fairness and utility: by controlling the number of dimensions, we can balance fairness constraints against predictive performance. Moreover, once such a fair representation is obtained, it can be reused across multiple downstream tasks or fairness definitions without requiring retraining. This flexibility makes representational fairness particularly appealing in multi-task or evolving fairness scenarios, as it decouples fairness enforcement from task-specific prediction objectives.

We will incorporate this discussion into the related work section in the revised version of the manuscript.

W2: We thank the reviewer for suggesting this reference. Indeed, it shares a similar motivation, which aims to ensure privacy by minimizing the information content related to sensitive variables in the learned representation. This aligns well with our fairness perspective, and it opens an interesting direction for extending our framework. Specifically, we can consider non-linear transformations of representations and formalize fairness objectives using dependence measures such as mutual information or distance covariance. For example, by minimizing the dependence between the representation and the sensitive attribute (e.g., minimizing $I(X,Z)$ while preserving utility (e.g., maximizing $I(X,Y)$, we can construct fair representations in a more general, potentially non-linear setting. We will incorporate this perspective into the conclusion and future work section of our revised manuscript.

Questions

Q1: The computational cost of obtaining the fair projection matrix primarily arises from estimating the candidate matrices $M_Y$, $M_Z$ and $M_{Y,Z}$, each typically constructed as a weighted covariance matrix. These computations scale linearly with the sample size $n$. In addition, the procedure includes an eigen-decomposition step for $p\times p$ matrices, and the rank estimation step scales linearly with dimension $p$. Notably, our method remains applicable in high-dimensional settings, as SDR techniques are specifically designed to extract low-dimensional structure from high-dimensional data.

As for performance, it largely depends on the structure of the original representation. Since our current framework focuses on linear transformations, it may fail to capture non-linear dependencies. One potential solution is to extend our approach using non-linear SDR techniques, such as kernel-based methods, as discussed in the limitations section of the appendix. Alternatively, as mentioned in our response to your comment on weaknesses, we can explore information-theoretic approaches (e.g., mutual information) to handle non-linear relationships in the representation learning process.

Paper Formatting Concerns

We thank the reviewer for pointing out the typos and potential confusion notations, we will revise them in the final manuscript. In line 147, $B$ is the standardized eigenvectors of $M_Y$, which makes $BB^{\top}$ a projection matrix without losing any information relates to $Y$.

We appreciate your feedback, which has helped strengthen our work, and we hope our responses have addressed all of your concerns. Please let us know if there is any additional information or clarification we can provide to further support your understanding and confidence in our work.