Wasserstein Distributionally Robust Optimization through the Lens of Structural Causal Models and Individual Fairness

First of all, thank you for your detailed comments. We will address each of them in detail.

W1. In the global response, we explained that introducing a new DRO framework requires considering key theorems such as strong duality, closed-form worst-case loss, regularizer estimation, and finite sample guarantees to ensure practical applicability. We aimed to present a minimal and complete framework, making our theoretical results the main part of our work.

W2. The paper you mentioned shares only keywords with my work and differs significantly in scope. They aim to provide a new method for computing counterfactual instances by bypassing the conventional three steps (Abduction, Action, and Prediction). They propose using an optimal transport map, suggesting that if $(X, S=s) \sim \mathbb{P}\_s$ and $(X, S=s' \mid X=x, S=s) \sim \mathbb{P}\_{s'}$ are probability distributions, then under strict conditions, the optimal transport theory can find a map $T: \mathcal{X} \to \mathcal{X}$ such that T\_{\\#}\mathbb{P}\_s = \mathbb{P\}\_{s'}. This means they use $T(x)$ instead of causally computing counterfactuals. However, since they rely on Euclidean distance to find the optimal map, this approach only works in very special cases, as the authors mentioned in the paper [1]. Therefore, this work does not align with the scope of our work.

As mentioned in the global response, our work is not limited to algorithmic fairness and might omit some works in that field, similar to how we excluded papers in reinforcement learning and other fields. Our focus is on introducing new distributionally ambiguity set, and we have cited relevant references as comprehensively as possible.

W3. In the global response, we discussed the philosophy and advantages of our work compared to existing methods. In addition, Chapter 4.1 outlines the benefits of our approach through comparisons with other works.

W4. As mentioned in the background section, when we refer to counterfactuals, we mean computing them through the three steps: Abduction, Action, and Prediction. In line 100, we provide the counterfactual formulation. To clarify further, we detailed Example 1, demonstrating that the counterfactual of the instance $(M,1,1)$ is $(F,0,-2)$.

W5. As mentioned in the global response, our framework is a general approach to applying DRO with causality and protected variables, applicable in fairness, adverse learning, robust learning, reinforcement learning, transfer learning, GAN, NLP, and more. We used fair learning as an example to demonstrate one application and compared our method only with those having similar assumptions, not all algorithmic fairness methods.

W6. In lines 236-240, we discuss the computational aspect of the proposed DRO and reference papers that guarantee fast algorithms for computing our methods. We provide Theorems 2, 3, and 4 to demonstrate the computational efficiency of our approach by solving or estimating the worst-case quantity and incorporating it into the loss function.

W7-9. We thank you for your suggestions; we have incorporated the changes and addressed these minor comments in the new version, and the related works may form a distinct chapter.

Q1. We provided a detailed response to this in our global response.

Q2. The uncertainty set in our method is defined using the Wasserstein distance, incorporating the causally fair dissimilarity function, effectively creating a Wasserstein ball equipped with CFDF. Proposition 2 shows that we can estimate the worst-case loss quantity using the robust optimization method defined in Equation 16, establishing their equivalence.

Q3. The first part of Assumption 2 is natural in real applications, as it assumes that the feature space is bounded and the parameter space is bounded and closed.

The third part is reasonable, as it relates to the statistical properties of the estimator for the cost function or causal model structure. For example, in a linear SCM, we can estimate its functional structure with a convergence rate of $O(N^{-\frac{1}{2}})$, allowing us to design a metric that satisfies the third assumption.

The second part of Assumption 2 might seem new but is essentially a Lipschitz condition on perturbing non-sensitive features. Since perturbations in the causal structure are derived from counterfactuals, this is quite natural. For example, consider a linear SCM with reduced-form mapping $M$, where $X = MU$ and all sensitive attributes are parents. Here, $CF_0(v, \Delta) = v + M\Delta$. Given a loss function $\ell(v, y, \theta) = h(\theta^T v - y)$ where $h$ is Lipschitz, assuming a norm $\ell_p$ in the exogenous space leads to $d(v, CF_0(v, \Delta)) = \|\Delta\|_q$. This makes the Lipschitz condition:

$$\vert \ell(v, y, \theta) - \ell(CF_0(v, \Delta), y, \theta) \vert = \vert h(\theta^T v - y) - h(\theta^T v - y + \theta^T M \Delta) \vert \leq L \|\theta^T M\|_p \|\Delta\|_q$$

This naturally satisfies condition 2.

Q4. Our method can be applied in fields where the Wasserstein distance or optimal transport is used, allowing for the incorporation of causality and protection of feature variables. In neural networks, especially GAN models, it enhances sensivity of model respct to causal structure. In signal processing, reinforcement learning, and NLP, it introduces causality and fairness into the Wasserstein distance.

Q5. This statement refers to Theorem 4, which provides a first-order estimation of the worst-case loss quantity, described as a regularizer in our DRO framework. Proposition 2 further establishes the relationship between our proposed DRO method and conventional robust optimization techniques that typically involve adversarial perturbations.

In conclusion, we hope our responses are comprehensive enough to capture your interest in this work.

[1] De Lara et al. (2024). Transport-based counterfactual models.

W1. Thank you for pointing out this issue. We would like to address your question from two different perspectives:

1. Regularization and Scalability: In our work, we demonstrate the strong duality theorem (Theorem 1), which shows that the DRO learning problem can be transformed into an empirical risk minimization (ERM) problem with an added regularizer. This transformation removes the need to compute the worst-case loss quantity directly, making the computation more scalable. Previous works, such as [1] and [2], support the use of algorithms that incorporate regularizers to improve learning efficiency.

2. Curse of Dimensionality: The curse of dimensionality is a significant concern in traditional DRO, where the convergence rate of the distance between the empirical measure and the underlying distribution is affected by the feature space dimension:
   $W(\mathbf{P}\_{N}, \mathbf{P}_*) = O(N^{-\frac{1}{d}})$

However, if we assume that the distribution $\mathbf{P}_\ast$ is derived from a SCM, this rate improves to $O(N^{-\frac{1}{2}})$, effectively breaking the curse of dimensionality. We plan to explore and demonstrate this point in detail in our following work.

Q1. In our experiment, we assume that the data is generated by an additive noise model. We first learn the functional structure, and then design our causally fair dissimilarity function. Using the theorems, we can subsequently calculate the worst-case loss quantity.

If we do not make assumptions about causality (e.g. i.i.d. case) and do not have a protected variable, our results are equivalent to traditional Wasserstein DRO, ensuring that everything functions as expected.

[1] Hong TM Chu, Kim-Chuan Toh, and Yangjing Zhang. On regularized square-root regression 391 problems: distributionally robust interpretation and fast computations. Journal of Machine 392 Learning Research, 23(308):1–39, 2022.

[2] Yangjing Zhang, Ning Zhang, Defeng Sun, and Kim-Chuan Toh. An efficient hessian based 530 algorithm for solving large-scale sparse group lasso problems. Mathematical Programming, 531 179:223–263, 2020.

Thank you for your helpful comments. We will address each point of weakness and questions, labeled Wi and Qi respectively, in order.

W1. We will add the Structural Causal Model (SCM) in line 49.

W2. We appreciate your comment and agree that the numerical section could be improved. As mentioned in our global response, our method is not specific to fair learning. In the numerical section, we aim to showcase just one of the many applications of this framework. However, we faced challenges in finding datasets and methods compatible with our problem.

The algorithms in our work are similar to those used in regular optimal transport, such as the Sinkhorn algorithm [1]. In these algorithms, we simply replace the matrix of the cost function with the one calculated based on the CFDF function.

W3. As mentioned in line 54, we adopt the definition of a fair metric from Ehyaei et al. to define a causally fair dissimilarity function (CFDF). However, their notion of a fair metric is not compatible with our assumptions. We require a more general definition, namely a dissimilarity function because our cost function does not satisfy some of the fundamental properties of a metric, such as:

• Identity of Indiscernibles: $d(x, y) = 0$ if and only if $x = y$.
• Triangle Inequality: $d(x, z) \leq d(x, y) + d(y, z)$.

Therefore, to avoid misleading use of the term "metric" and to have broader applicability, we define a general notion of a dissimilarity function instead of a metric, which has weaker assumptions.

To obtain a representation theorem of the CFDF, we need Proposition 1. Unfortunately, Proposition 1 in the work of Ehyaei et al. does not work in the general case and is limited to metrics.

W4. As mentioned in the global response, since our work uses counterfactuals from structural causal models, our model must be counterfactually identifiable. Therefore, we should base our method on counterfactually identifiable SCMs, such as the bijective generative mechanism (BGM). To avoid additional complexity, we focus on the additive noise model, a specific BGM instance. However, our results are valid for general BGM.

In our framework, the primary concern is the counterfactual identifiability of the SCM model. Once the SCM model or corresponding cost function is estimated, Theorem 5 guarantees that the learning problem has a convergent solution for finite samples.

Q1. In fair learning, it is well known that there is a trade-off between individual fairness and accuracy. Achieving individual fairness necessitates adjusting the model to account for variations across individuals, which increases complexity and the potential for overfitting. This adjustment may reduce accuracy, as the model must balance fitting the overall data distribution with adhering to fairness constraints, potentially limiting its predictive precision. Therefore, increasing fairness often requires sacrificing some degree of accuracy.

This trade-off is justified by the method's primary objectives: reducing disparate impact and ensuring equitable treatment across individuals. While accuracy is important, the approach prioritizes mitigating biases to promote fairness and inclusion in algorithmic decision-making. Thus, the intentional trade-off of reduced accuracy aims to achieve a more equitable and socially responsible model.

[1] Cuturi, M. (2013). Sinkhorn Distances: Lightspeed Computation of Optimal Transport. Advances in Neural Information Processing Systems, 26, 2292–2300.

First of all, I would like to thank you for your insightful comments. We appreciate the attention given to our work. In the following, I respond to the mentioned weaknesses and questions in detail.

W1. In the global response, we explained the intuition behind our assumptions. Assumption 2 seems new, but it is common in finite sample guarantee theorems, such as Theorem 6 of [1]. The first part of Assumption 2 is natural, assuming the feature space is bounded and the parameter space is bounded and closed.

The third part is also reasonable, relating to the statistical properties of the estimator for the cost function or causal model. For example, in a linear SCM, we can estimate its functional structure with a convergence rate of $O(N^{-\frac{1}{2}})$, allowing us to design a metric that satisfies the third assumption.

The second part of Assumption 2 is the Lipschitz condition on perturbing non-sensitive features, derived from counterfactuals. For example, consider a linear SCM with reduced-form mapping $M$, where $X = MU$ and all sensitive attributes are parents. Here, $CF_0(v, \Delta) = v + M\Delta$. Given a Lipschitz loss function $\ell(v, y, \theta) = h(\theta^T v - y)$ and an $\ell_p$ norm in the exogenous space, we get $d(v, CF_0(v, \Delta)) = \\|\Delta\\|_q$. This makes the Lipschitz condition:

$$\vert \ell(v, y, \theta) - \ell(CF_0(v, \Delta), y, \theta) \vert = \vert h(\theta^T v - y) - h(\theta^T v - y + \theta^T M \Delta) \vert \leq L \|\theta^T M\|_p \\|\Delta\\|_q$$

If you need further clarification, we would be happy to engage during the discussion period.

W2. Thank you for highlighting this point. Assumption 2(iii), related to the estimation of ANM, depends on the functional structure and smoothness of the model, resulting in different convergence rate orders. For example, for a linear ANM, the convergence rate is $O(N^{-1/2})$. We omitted a detailed discussion on this rate as it pertains to the estimation of SCMs more broadly. Each estimator with convergent rate $O(N^{-\alpha})$ where $\alpha>0$ works in Theorem 5. In the revised version, we will add explanations to clarify this topic further.

W3. To present a minimal and complete version of our proposed method, we focused heavily on the theoretical section. To address its applications and implications, we detailed its relation to previous research and included numerous references.

Q1. We appreciate your attention. The words for each and exists were used incorrectly, and we have fixed them in the revised version. The correct assumption is:

• For each $t_0 \in \mathbb{R}$, there exists a sequence $\\{t_k\\}$ that goes to $\infty$ such that we have

$$\lim_{k \rightarrow \infty} \dfrac{|h(t_0 + t_k) - h(t_0)|}{|t_k|} = L_h$$

To provide intuition behind this assumption, for each $t_0$ and each $\epsilon >0$, we need to find $t_\ast$ such that $|h(t_\ast)-h(t_0)|>(L_h -\epsilon) |t_\ast-t_0|$ with $|t_\ast-t_0| > \delta$. Assumption (ii) addresses this without considering the magnitude of $\delta$.

As you mentioned, the quantile loss is $1$-Lipschitz. Let $t_0$ be given. If we choose $\\{t_k\\}$ that goes to $-\infty$, then

$$\lim_{k \rightarrow \infty} \dfrac{|h(t_0 + t_k) - h(t_0)|}{|t_k|} = 1$$

Therefore, our assumption is satisfied.

Q2. We apologize for the oversight; in fact, this should be finite. We corrected it.

[1] Blanchet, Jose, Karthyek Murthy, and Viet Anh Nguyen. "Statistical analysis of Wasserstein distributionally robust estimators."