Reply to weakness

1. Experimental results. We compared the regression and classification performance among six benchmark methods across six datasets. Additionally, we discussed the results of using F1-score as utility metric, randomly splitting groups, methods using demographic prior during the training phase, and different $lambda$ settings. By thoroughly examining various scenarios and conditions, we have ensured that our analysis and conclusions are well-founded and credible.

2. Novelty and effectiveness. We studied Rawlsian harmless fairness without demographic information in both classification and regression tasks, which has not been studied before. Moreover, as supported by reviewer kaxC in the 'Strengths' section, our idea is incredibly novel and impressive. Besides one weakly related reference, they were unable to find occurrences of this idea in prior work.

Reply to questions

1. Explanation for Dirac delta distribution. Please refer to Figure 1 in the newly uploaded supplementary PDF, where we provide a more detailed description of the Dirac delta distribution. We speculate that your misunderstanding may stem from confusion regarding the meaning of the x-axis of Figure 1 in our paper. The x-axis represents the index of sorted losses, rather than the actual loss values, which is different from Figure 1 in the supplementary PDF.

2. From MUD to loss distribution. (1) MUD is a group-level metric. When demographic information is inaccessible, we strive to minimize MUD across all possible group divisions. (2) From the perspective of individual instances, when the loss for each example is as similar as possible, MUD will approximate zero regardless of the group division. (3) If the loss for each example approximates zero, the loss distribution will resemble a Dirac delta distribution. If the loss for each example approximates $\mu$ (e.g. $\mu$=0.25 in a 0-1 regression task), the model will behave like a uniform regressor or classifier. Note that this does not imply that the loss distribution itself is uniform; rather, the model's performance is uniformly poor across instances.

3. Confusion caused by the term 'overfit'. Sorry for any confusion on this point. This section was intended to introduce the derivation of the instance-level loss, which bypasses the unobserved sensitive attributes, rather than implying that our model would overfit. The logic is as follows: (1) We aim for MUD=0 for fairness. (2) Oracle model achieves zero loss for each sample, and thus MUD=0. (3) A loss of 0 on the training set indicates overfitting risks in practice.

   And aiming for MUD=0 does not necessarily lead to overfitting in our method.

4. Difference with [1]. We study different problems. They focus on achieving fairness with access to demographic information. However, our task is more challenging as we aim to achieve fairness without demographic information.

5. Consistency of TUD and VAR in experimental results. In our experimental results, TUD and VAR consistently outperform other methods in most cases, except for some classification cases. As discussed in lines 298-301, under discrete utility metrics, even if our method has smaller loss disparities at the instance level, fairness metrics may still not improve.

6. Comparison with SOTA. Our method uniquely addresses Rawlsian harmless fairness without requiring demographic information in both classification and regression tasks, a topic that has not been extensively studied before. Existing works not included in this comparison generally struggle to adapt to this specific setting. Besides the employed baselines (which were carefully modified for harmless fairness), to the best of our knowledge, no other recent work targets this same research problem.

[1] B. H. Zhang, B. Lemoine, and M. Mitchell, “Mitigating Unwanted Biases with Adversarial Learning,” AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society, 2018.

Reply to weakness

1. Rigor of expression. Thank you for your careful reading and rigorous derivation. We will update Proposition 1 and the assumptions regarding u as you suggested.

2. Confusion caused by lines 169-180. Sorry for any confusion on this point. Lines 169-180 offer a deeper reflection on our method and do not serve as extra motivation. After deriving Equation (2), we further considered the commonalities and differences between our method and existing works. Equation (3) illustrates the fundamental difference between our method and bias-variance research, while Equation (4) demonstrates the essential similarity between our method and DRO.

3. About demographic prior. We believe that demographic prior is meaningful, for demographic prior is still used during the evaluation stage. Table 7 shows a comparison between VFair and methods using demographic prior during the training stage. Results indicate that demographic prior does indeed provide some improvement.

Reply to questions

1. Explanation for Dirac delta distribution. Please refer to Figure 1 in the newly uploaded supplementary PDF, where we provide a more detailed description of the Dirac delta distribution.

2. More concise explanation. Thank you for your suggestion. We will improve the clarity and logic of Remark 2 following your comments.

Reply to limitations

We will add a Limitations section to discuss the additional computational costs of the proposed method. Briefly, our method, VFair, requires two backward propagations, resulting in approximately double the computation time compared to ERM. Please also refer to our detailed responses to reviewer cPfu.

Reply to weakness

1. Reproducibility issues. We will provide more training details in the appendix. (1) Throughout experiments, all methods are set with a batch size of 32 and a learning rate of 0.01. As for the training epoch, as mentioned in lines 555-559, to ensure all baselines comply with the harmless fairness setting, we terminate each training process at the epoch with the nearest loss value to a well-converged ERM. (2) The datasets are randomly split in a 7:3 ratio following ARL's experimental implementation.

2. Comparison with BPF. We have discussed the relationship between our method and BPF, referenced as [12] in our paper. BPF, as an improved version of DRO, also utilizes group ratio prior. Considering your comments, we added BPF as another baseline. However, as shown in Table 1 in the newly uploaded supplementary PDF, VFair still outperforms BPF on all metrics.

3. Connection with Individual Fairness. Although VFair operates at the instance level, it differs from Individual Fairness. (1) Individual Fairness requires that if two individuals are close on the similarity metric, they should be close on the treatment metric, resulting in a continuous treatment metric outcome. (2) VFair falls in (Rawlsian) group fairness, which does not consider the similarity between individuals but focuses on performance disparities between different groups, pursuing close treatment (utility in this paper) metric outcomes. (3) As mentioned in line 145, group fairness pursues $\ell\perp z$, while Individual Fairness pursues $\ell\sim f(z)$, where $f()$ represents a similarity metric. (4) The similarity metric $f()$ also serve as a form of prior. In contrast, VFair does not call for any form of prior.

4. Analysis of results. We have strived to present our findings accurately and objectively and we believe there is no exaggeration of effects. Note that we study harmless fairness; therefore, only methods that do not compromise much on overall utility will be finally ranked. (1) Regarding Table 2, DRO turns out a uniform regressor with low utility. In this context, our method outperforms other methods on all metrics except for DRO. (2) Note that lines 307-310 mention that Table 2 aims to show the limited improvement on UCI but significant improvement on CelebA (except for DRO). We acknowledged the limited improvement on UCI and analyzed the reason.

5. Computational costs. Thank you for this suggestion. (1) Our proposed method requires two rounds of back-propagation, which thus leads to more computation cost compared to ERM or DRO. (2) Since the backward pass is the bottleneck of the total computation, we found that VFair requires approximately twice the computation time compared to the ERM method, as shown in Table 1 with the Law School dataset as an example. (3) Note that ARL, an adversarial method, requires a comparable wall-clock time to VFair due to its inner and outer optimization nature.

   Table 1: Comparison of four methods' wall-clock time on Law School with the same experimental setup.

   Method	Time
   ERM	349.4s
   DRO	243.5s
   ARL	640.1s
   VFair	677.6s

6. Eta in DRO. When applying DRO, using a defined eta can be interpreted as using some group ratio. As evidenced in the official code of DRO in dual_robust_opt.py, the calculation of eta is achieved by bi-search with group ratio eps as input of the get_rho() function.

7. Minor weakness. Thank you for this suggestion. We will edit the section title and the symbol expressions according to your advice.

Reply to questions

1. Presentation of DRO results. Similar to W4, we study harmless fairness. As mentioned in lines 270-273 and lines 65-68, DRO turns out a uniform regressor with low utility.

2. Explanation of Z-Score. We use Z-score as it is a fundamental statistical measure. It is calculated by subtracting the mean from the individual value and then dividing it by the standard deviation. In our context, the Z-score of each example's loss is calculated as $\frac{\ell - \hat{\mu}}{\hat{\sigma}}$.

3. Adaptation on larger datasets. As shown in Appendix D, our method adopts a stochastic optimization strategy, from which one can see the complexity will be linear to the number of training samples. Note that baseline methods MPFR and FKL are not designed with stochastic updates and suffer from out-of-memory issues, as shown in Table 6.

4. Confusion caused by the term 'overfit'. Sorry for any confusion on this point. This section was intended to introduce the derivation of the instance-level loss, which bypasses the unobserved sensitive attributes, rather than implying that our model would overfit. The logic is as follows: (1) We aim for MUD=0 for fairness. (2) Oracle model achieves zero loss for each sample, and thus MUD=0. (3) A loss of 0 on the training set indicates overfitting in practice.

   Aiming for MUD=0 does not necessarily lead to overfitting.

5. Discussion about outliers. (1) We agree that the outlier problem has been studied in Fairness problem. We have also investigated some works in this direction, such as DORO[1], ARL, and GRASP[2]. However, these methods have different strategies to solve the outlier problem, and there is no general, representative method. (2) According to our analysis in the Loss view on page 5, we can simply cap the value of Z-score, preventing the model from excessively concentrating on outliers that produce large losses. As shown in Table 5, on the COMPAS dataset whose labels are found to be noisy [3], VFair achieves the best performance (except for FairRF which leverages feature correlations prior). We will leave a thorough comparison with other existing strategies in future work.

Reply to limitations

Refer to the above answers.

[1] Zhai R, et al. Doro: Distributional and outlier robust optimization.

[2] Y. Zeng, et al. Outlier-robust group inference via gradient space clustering.

[3] Preethi L, et al. Fairness without demographics through adversarially reweighted learning.

Reply to weaknesses

1. Computational costs. Your comment on two backward passes is correct. We will include this extra computation cost as one of the limitations of our work. Since the backward pass is the bottleneck of the total computation, we found that VFair requires approximately twice the computation time compared to the ERM method, as shown in Table 1 with the Law School dataset as an example. Note that ARL, an adversarial method, requires a comparable wall-clock time to VFair due to its inner and outer optimization nature.

   Table 1: Comparison of four methods' wall-clock time on Law School with the same experimental setup.

   Method	Time
   ERM	349.4s
   DRO	243.5s
   ARL	640.1s
   VFair	677.6s

2. Delta value. Sorry for any confusion on this point. (1) When presenting the optimization steps in the paper, we considered VFair to be executed independently, so we might not know the appropriate delta value beforehand. In this sense, the final delta value a trained model can achieve will be mainly determined by the maximal number of epochs, assuming it is well-converged. (2) In the experiments, to ensure all baselines comply with the harmless fairness setting, we terminate each training process at the epoch with the nearest loss to the delta derived from a converged ERM (See lines 555-559).