We thank the reviewer for their valuable comments. Below we provide a careful point-by-point response to each comment. We would be happy to provide additional discussions/information in the author-reviewer discussion period.

Proof replacement

We agree that relocating the detailed proofs can improve readability and we will revise the camera ready version of the paper accordingly. Specifically, we will move the proofs of Theorems 1, 2, and 3 to the Appendix.

While we will make this change, we want to take this opportunity to clarify a key distinction and the significance of our theoretical approach. We wish to clarify that Theorems 2 and 3 are not based on the analytically tractable setup of Theorem 1. Theorems 2 and 3 establish more general results under the much broader assumption of well-behaved isotropic noisy gradient oracle with scale parameter (NGOs). Additionally, we now relax the isotropic assumption and establish a new theorem based on non-isotropic (anisotropic) noise. Please see our new theorem and its detailed proof in our response to the reviewer ``qYpZ''.

Furthermore, the analytically tractable quadratic setup in Theorem 1 was deliberately chosen. A quadratic function serves as a powerful local approximation for any sufficiently smooth loss function near a minimum (via a second-order Taylor expansion). Therefore, our analysis in this setting offers valuable insight into the local convergence behavior of optimizers in the complex settings.

Real-world significance and extended comparisons over fairness-aware algorithms

Our goal is not to replace these established fairness-aware methods, but rather to demonstrate that the choice of optimizer is a fundamental, orthogonal, and complementary mechanism for improving fairness. However, based on your comment, to elaborate more on the real world significance of our findings, we have conducted a new experiment. We test the impact of optimizers in addition to the fairness-enhancing method [1] on two new datasets (ProPublica COMPAS [2] and AdultCensus [3]) with gender as the sensitive attribute. The results are presented in the table below for SGD, Adam, and RMSProp. We repeat the process 10 times and following [3], we report the average gap in equal opportunity, equalized odds, and demographic parity below (lower is better).

As the results show, adaptive optimizers outperform SGD both with and without the explicit fairness-enhancing method. This directly leads to the contribution and practical significance of our work: choosing an adaptive optimizer is not an alternative to existing fairness algorithms, but rather a complementary strategy that amplifies their effectiveness. A practitioner can gain almost ``free'' and significant boost in fairness by simply selecting Adam or RMSProp as the optimizer for their chosen fairness-aware training algorithm.

[1] Roh, Yuji, et al. "FairBatch: Batch Selection for Model Fairness." ICML (2021).

[2] Julia Angwin, et al. "Machine bias: There’s software used across the country to predict future criminals. And its biased against blacks." ProPublica (2016).

[3] Ron Kohavi. "Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid." KDD (1996).

Missing related work

We will add the mentioned papers to the related work section:

The work by Liu et al. [1] provides a key theoretical analysis showing that standard and unconstrained machine learning implicitly favors one fairness criterion over others. The authors prove that a model's deviation from calibration is bounded by its excess risk. This is related to our work as it also investigates the fairness properties that emerge from a standard training process.

The paper by Yao et al. [2] focuses on in-processing fairness methods that use surrogate functions to ensure fairness. Their theory also highlights that balanced datasets are beneficial for achieving tighter fairness and stability guarantees. This is relevant to our findings, as we also identify data imbalance as a critical factor through SDE analysis.

[1] Liu et al., "The Implicit Fairness Criterion of Unconstrained Learning." ICML (2019).

[2] Yao et al. "Understanding Fairness Surrogate Functions in Algorithmic Fairness." TMLR (2024).

Elaboration on how findings complement or offer distinct advantages over existing fairness-aware approaches

Our core theoretical results (Theorems 2 and 3) do not make strong assumptions about the specific form of the loss function. This generality is crucial because many in-processing fairness methods operate by adding a fairness regularizer to the standard loss. Our theory suggests that even with such a modified objective, the inherent mechanics of an adaptive optimizer like Adam should still offer benefits by normalizing subgroup-specific gradient disparities more effectively than SGD.

To validate this complementary effect, and motivated by your comment, we conducted the new experiment presented in the table in our previous response. Those results show that adaptive optimizers achieve superior fairness compared to SGD, both in a standard setup and when used as the optimization backbone for an existing fairness-enhancing algorithm. This clarifies the distinct and practical contribution of our work: the choice of an adaptive optimizer is not a substitute for dedicated fairness algorithms but is a complementary decision that can amplify their impact.

Provide a stronger justification for inclusion of proofs in the main body or consider relocating them to the Appendix for improved readability

We agree that relocating the detailed proofs can improve the readability. For a more detailed discussion on this, please see our response to the Proof replacement title.

We thank the reviewer for their valuable comments. Below we provide a careful point-by-point response to each comment. We would be happy to provide additional discussions/information in the author-reviewer discussion period.

Adaptive methods like Adam offer advantages like speed

We agree Adam/RMSProp are popular for their speed. Our contribution is orthogonal: we explain and predict their fairness behavior. Our theory shows that per‑coordinate second‑moment scaling contracts subgroup‑specific update gaps under imbalance. This yields actionable intuition: the theory predicts that if one wants a higher fairness, they would increase the effective second‑moment strength (i.e., by tuning $\beta_2$ hyperparameter). This intuition is novel and it is not related to the well-known advantages of the adaptive methods.

Assumption of isotropic noise in Theorems 2 and 3

Please kindly note that the assumption of isotropic noise is not uncommon in the literature. For instance, recent work such as [1] uses an isotropic noise assumption to illustrate why decaying learning rate during training is an effective mechanism for getting to a better minima.

However, inspired by the reviewer's comment, we now relax the conditions in Theorems 2 and 3 to non-isotropic (anisotropic) noise. We show that the results are correct even in the anisotropic noise case.

Theorem 2 for anisotropic noise: Consider a population that consists of two subgroups with subgroup-specific loss functions $\mathcal{L}_0(w)$ and $\mathcal{L}_1(w)$, sampled with probabilities $p_0$ and $p_1$, respectively. Suppose a stochastic (online) training regime, in which each parameter update is computed from a sample drawn from one of the two subgroups. Suppose the gradients $\nabla \mathcal{L}_0(w_k)$ and $\nabla \mathcal{L}_1(w_k)$ are well-behaved anisotropic NGOs, namely $\mathcal{N}(\mu_0, \Sigma_0)$ and $\mathcal{N}(\mu_1, \Sigma_1)$, respectively.
Then, the difference in parameter updates between subgroups 0 and 1 under RMSProp has an upper bound given by the corresponding difference under SGD. Consequently, RMSProp offers fairer updates across subgroups.

Proof: For SGD, the difference between the parameter updates across subgroups is $||\nabla \mathcal{L}_0 - \nabla \mathcal{L}_1||$. Regarding RMSProp, let's compute the expected value and the variance of $v_k$:

$$\mathbb{E}[v_k] = \gamma^k v_0 + (1-\gamma^k) [ p_0 \left( \mu_0 ^2 + diag(\Sigma_0) \right)+ p_1 \left( \mu_1^2 + diag(\Sigma_1) \right)].$$

Where $diag(\Sigma_0)$ and $diag(\Sigma_1)$ are the vectors with the diagonal elements of $\Sigma_0$ and $\Sigma_1$, respectively. Similar to the the isotropic case, we can show that the variance of $v_k$ is negligible and thus $v_k \rightarrow E[v_k]$. Hence, the difference between parameter updates are $|| D (\nabla \mathcal{L}_0 - \nabla \mathcal{L}_1) ||$, where $D$ is a diagonal matrix, whose $jj$th element can be computed as:

$$D_{jj} = \frac{1}{\sqrt{p_0 \left( \mu_{0_j} ^2 + \sigma_{0_j}^2 \right)+ p_1 \left( \mu_{1_j}^2 + \sigma_{1_j}^2 \right) + \epsilon}}.$$

Prior work [2] has experimentally shown that $\sigma^2 > \mu^2$. Additionally, in stochastic (online) training setup: $\Theta^2 \rightarrow1$. Thus, $D_{jj}<1$. This completes the proof.

The extension of Theorem 3 follows directly as the proof of Theorem 3 relies on the conclusion of Theorem 2.

We will add this to the appendix of the final version of our paper.

[1] Shi, Bin, et al. "On learning rates and Schrödinger operators." Journal of Machine Learning Research (2023).

[2] Malladi, Sadhika, et al. "On the SDEs and scaling rules for adaptive gradient algorithms." Advances in Neural Information Processing Systems (2022).

Advantage over SGD on DPA is less obvious than EOP and EOD from the experiments.

Fig. 2 shows that the advantage of adaptive optimizers over SGD on DPA is less obvious. This empirical result actually provides the confirmation of our Theorem 1. This theorem predicts that the DPA advantage of RMSProp is most significant under severe data imbalance. In datasets with only moderate imbalance (e.g., CelebA-Gender, with a 42% male-to-all ratio), the DPA gap is smaller. However, the DPA advantage is more obvious in the case of FairFace with race sensitive attribute, as it exhibits extreme imbalance, with a minority-to-all ratio of approximately 0.9%.

How can RMSProp be modified to perform even better in terms of demographic parity?

Our SDE analysis provides a path for some modifications in RMSProp to further enhance its fairness. Our analysis (culminating in Eq. 15 and 20) pinpoints that the final model's bias stems from the SDE's drift term, which is driven by the imbalanced sampling probabilities ($p_0$ and $p_1$) of the data. Additionally, RMSProp's relative advantage comes from how it scales the updates: The denominator of the update involves the term $u_t$, which helps to avoid the domination by the majority sub-group. Based on these insights, a direct path to improving RMSProp's fairness is to correct the biased drift term while retaining the benefits of its adaptive normalization. The SDE model suggests that if we could modify the algorithm to optimize a debiased loss function, the stationary distribution of our SDE would shift its center to the truly fair minimum. This modification would use a debiased gradient in each step, computed by re-weighting subgroups within a mini-batch to simulate a balanced population. Our SDE framework predicts this change would directly correct the drift term, shifting the stationary distribution to be centered at the fair optimum.

In what extent that Theorem 2 and Theorem 3 can be extended to the fairness metrics EOP and EOD?

Please kindly note that Theorem 2 is agnostic to the fairness metrics. Theorem 2 is broadly applicable and provides the core intuition for the improvements in EOD and EOP. Its proof focuses on the fundamental mechanism of the optimizer that RMSProp shrinks the disparities between subgroup gradients ($||\nabla\mathcal{L}_0 - \nabla\mathcal{L}_1||$). By preventing the model from being disproportionately pulled toward one subgroup's minimum, it promotes more balanced error rates (e.g., true positive and false positive rates) across all groups. Since EOD and EOP are measures of error rate equality, the general mechanism established in Theorem 2 directly explains the strong empirical improvements we observe for these metrics.

Theorem 3 relies on the DPA gap. Direct formal extension to EOD or EOP is highly challenging, as it requires analyzing a fairness gap conditioned on the true label. This conditioning complicates the gradient analysis significantly and is beyond the current scope of the paper.

For the proof of Lemma 1, could the relationship between Eq 43 and zero-one loss difference be elaborated more?

Eq. 43 shows the definition of the demographic parity. Directly optimizing the non-differentiable demographic parity gap is intractable, thus it is a standard practice to replace it with a tractable setup [1]. Following prior works, we replace this with the gap in the loss functions for the sub-groups. The principle is that if a model yields a similar loss distribution for two sub groups, it is being penalized similarly for its mistakes on both, which encourages its output behavior to align.

[1] Li et al. "Achieving fairness at no utility cost via data reweighing with influence." ICML (2022).

The definition of parameter updates in Theorem. 2

In Theorem 2, the "difference in parameter updates" refers to a difference in two updates: one that is occurred by drawing the samples from sub group 0 and the one by drawing samples from sub group 1. We analyze the difference that would occur if the sample drawn at iteration $k$ came from subgroup 0 versus if it came from subgroup 1. As an example, For SGD, the update rule is $w_{k+1} = w_{k} - \eta \nabla\mathcal{L}$; thus if a sample is drawn from subgroup say 0, the update relative to $w_k$ is $\nabla\mathcal{L}_0$. If there is imbalance in the dataset (e.g., the chance of drawing one sub group is substantially higher), the gradients (updates) from the majority would dominate the optimization process.

We thank the reviewer for their valuable comments. Below we provide a careful point-by-point response to each comment. We would be happy to provide additional discussions/information in the author-reviewer discussion period.

SDE analysis (Theorem 1) based on a one-dimensional quadratic loss

We deliberately designed a one-dimensional quadratic case and studied it as a warm‑up setup for 2 main purposes: 1. The local dynamics around any sufficiently smooth minimum can be approximated by a quadratic function via Taylor expansion. Hence, our SDE analysis of the fair minima provides crucial insights into the local convergence behavior of the optimizers on more general cases. 2. To provide a clear and unambiguous intuition about the fairness behavior of adaptive and SGD optimizers that would be obscured in complex settings.

Also, please kindly note that we build upon the intuition from this tractable setup with more general, high-dimensional theorems. Our main theoretical results in Section 3.3 (Theorems 2 and 3) are not limited to the 1D quadratic case.

Other variants such as SGD with momentum, AdamW, and AdaBound

Our theoretical analysis (Theorems 1-3) identifies gradient normalization via second moment as the key mechanism for improving group fairness. This provides a clear framework for predicting the behavior of other optimizers. To validate these predictions, we have conducted a new set of experiments with SGD w/momentum, AdamW, and AdaBound. Following the protocol from our main experiments, we use the CelebA dataset with a ViT backbone, treating gender and age as sensitive attributes. We tuned the learning rate for each optimizer using Bayesian optimization with Hyperband and report the average fairness scores below (higher is better).

As shown in the table, AdamW consistently achieves the best fairness scores. This is expected, as AdamW retains the normalization mechanism of Adam and RMSProp via the second moment. In contrast, SGD w/momentum, lacking the normalization via the second moment, performs more biased. The performance of AdaBound is comparable to SGD w/momentum. The reason is that the AdaBound algorithm deliberately clips the second moments to make it act like SGD, which in turn limits the fairness advantage.

Assumption of isotropic noisy gradient oracle with scale parameter (NGOS) in Theorems 2 and 3

Please kindly note that the isotropic assumption is common in the literature for analyzing optimizer dynamics as it yields clean and interpretable results. For instance, recent work such as [1] also employs an isotropic noise assumption to illustrate why decaying learning rate during the training is an effective mechanism.

However, inspired by the reviewer's comment, we now relax the conditions in Theorems 2 and 3 to non-isotropic (anisotropic) noise. We show that the results are correct even in the anisotropic case.

Theorem 2 for anisotropic noise: Consider a population that consists of two subgroups with subgroup-specific loss functions $\mathcal{L}_0(w)$ and $\mathcal{L}_1(w)$, sampled with probabilities $p_0$ and $p_1$, respectively. Suppose a stochastic (online) training regime, in which each parameter update is computed from a sample drawn from one of the two subgroups. Suppose the gradients $\nabla \mathcal{L}_0(w_k)$ and $\nabla \mathcal{L}_1(w_k)$ are well-behaved anisotropic NGOs, namely $\mathcal{N}(\mu_0, \Sigma_0)$ and $\mathcal{N}(\mu_1, \Sigma_1)$, respectively.
Then, the difference in parameter updates between subgroups 0 and 1 under RMSProp has an upper bound given by the corresponding difference under SGD. Consequently, RMSProp offers fairer updates across subgroups.

Proof: For SGD, the difference between the parameter updates across subgroups is $||\nabla \mathcal{L}_0 - \nabla \mathcal{L}_1||$. Regarding RMSProp, let's compute the expected value and the variance of $v_k$:

$$\mathbb{E}[v_k] = \gamma^k v_0 + (1-\gamma^k) [ p_0 \left( \mu_0 ^2 + diag(\Sigma_0) \right)+ p_1 \left( \mu_1^2 + diag(\Sigma_1) \right)].$$

Where $diag(\Sigma_0)$ and $diag(\Sigma_1)$ are the vectors with the diagonal elements of $\Sigma_0$ and $\Sigma_1$, respectively. Similar to the the isotropic case, we can show that the variance of $v_k$ is negligible and thus $v_k \rightarrow E[v_k]$. Hence, the difference between parameter updates are $|| D (\nabla \mathcal{L}_0 - \nabla \mathcal{L}_1) ||$, where $D$ is a diagonal matrix, whose $jj$th element can be computed as:

$$D_{jj} = \frac{1}{\sqrt{p_0 \left( \mu_{0_j} ^2 + \sigma_{0_j}^2 \right)+ p_1 \left( \mu_{1_j}^2 + \sigma_{1_j}^2 \right) + \epsilon}}.$$

Prior work [2] has experimentally shown that $\sigma^2 > \mu^2$. Additionally, in stochastic (online) training setup: $\Theta^2 \rightarrow1$. Thus, $D_{jj}<1$. This completes the proof.

The extension of Theorem 3 follows directly as the proof of Theorem 3 relies on the conclusion of Theorem 2.

We will add this to the appendix of the final version of our paper.

[1] Shi, Bin, et al. "On learning rates and Schrödinger operators." Journal of Machine Learning Research (2023).

[2] Malladi, Sadhika, et al. "On the SDEs and scaling rules for adaptive gradient algorithms." Advances in Neural Information Processing Systems (2022).

What is the chance that the conclusions can be applied to other modalities beyond vision?

Please kindly note that our claim is not modality-specific. Theorems 1, 2, and 3 reason at the level of optimizer dynamics under subgroup imbalance. They conclude that adaptive optimizer's help enhance the group fairness regardless of the input modality. These arguments depend on the geometry of gradient updates, not on the input.

However, as per the reviewer's comment, we now perform a quick experiment using a new modality: tabular data. We use ProPublica COMPAS [1] and AdultCensus [2] datasets with gender as the sensitive attribute. We do classification using logistic regression, while optimizing the learning rate using Bayesian optimization. We repeat the process 10 times and following [3], we report the average gap in equal opportunity, equalized odds, and demographic parity below (lower is better).

As shown, adaptive optimizers offer better fairness over SGD in all cases. This confirms that our results are not limited to the vision application.

[1] Julia Angwin, et al. "Machine bias: There’s software used across the country to predict future criminals. And its biased against blacks." ProPublica (2016).

[2] Ron Kohavi. "Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid." KDD (1996).

[3] Roh, Yuji, et al. "FairBatch: Batch Selection for Model Fairness." ICML (2021).

Fairness conclusion depends on the choices of learning rates and other hyperparameters

Indeed the fairness outcomes can be influenced by hyperparameter choices, which is why we adopted a highly rigorous approach to mitigate this potential influence. Our primary method for ensuring a fair comparison was a robust and automated search for the most critical hyperparameter: the learning rate. As detailed in our Appendix J (l. 679) and F (l. 600), we did not manually select these values. Instead, we employed Bayesian Optimization with Hyperband algorithm to systematically find the best learning rate for each optimizer independently. We then run each optimizer 2 times with their tuned learning rate and report the average results. For other hyperparameters, we use the standard PyTorch default values. However, to further address the reviewer's comment, we conduct an additional hyperparameter search on the CelebA dataset with gender and age as the sensitive attributes using ViT backbone. This new set of experiments goes beyond only tuning the learning rate and uses Bayesian Optimization with Hyperband to jointly optimize learning rate, decay rate, and batch size.

The pattern in the above table is consistent with our initial findings. In all the scenarios, adaptive optimizers offer better fairness.

We thank the reviewer for their valuable comments. Below we provide a careful point-by-point response to each comment. We would be happy to provide additional discussions/information in the author-reviewer discussion period.

Individual fairness feasibility in facial recognition

Individual fairness implies that the model should treat similar individuals, similarly. Given that each individual is different in FR datasets, this would not be very meaningful. However, one could define some metric or parameter, according to which different individuals are similar. This metric could then be utilized to ensure fairness for two individuals who are close in terms of that metric. Having said this, this approach is not quite common in FR tasks.

Intuition about noisy gradient oracle with scale parameter (NGOS)

NGOS is simply a formal way to model the gradients used in mini-batch training. Because gradients are computed on a small batch of data rather than the entire dataset, the gradient at each step is not the ``true'' gradient, rather it is the true gradient plus some random noise from the sampling process. NGOS captures this by representing the stochastic gradient as the sum of the true gradient and a noise term. We will add this description to the text of the paper.

Why demographic parity for gender classification?

Please kindly note that in the gender classification experiment, while the prediction target (output class) is gender, the sensitive attributes we evaluate for fairness are race and age, not gender itself (please see l. 665). Thus, demographic parity in our work measures whether the model predicts a specific gender with the same probability across different racial or age groups. As an example, with regards to demographic parity, we are interested in the fact that the probability of being "female" is the same across "young" and "old" subgroups.

Typos

We thank the reviewer for catching these typos. We will correct all the noted typos and perform a thorough proofreading to prevent recurrences in the final version.

Clarifying ``computational overhead'' of fairness methods vs. optimizer switch

We agree that some fairness interventions, such as simple post-processing rules, can be computationally inexpensive. However, as the reviewer correctly noted, our intention with the statement on l.88 was to highlight a contrast. Specifically, we aimed to compare the introduction of a separate fairness-enhancing module with the inherent simplicity of selecting a different optimizer. This distinction is important for practitioners who are looking for a straightforward way to improve fairness in their pipeline. We will revise l.88 to reflect this.

Usage of line charts to represent the results

We will replace the line chart in Figure 2 with a grouped bar chart to more accurately represent the data. We will also apply this correction to Figures 11, 12, and 13 in the appendix to ensure consistency.

Placement and clarification of fairness metrics

As the reviewer correctly points out, the definitions of fairness metrics are currently in Appendix G, and the clarification that higher values are better is in l.629. We agree that moving them to the main text would help the readers to more easily understand the figures. We will do so in the final version, using the extra page allowed for the final version.

Change in optimizer to be used with other fairness-enhancing methods

Our Theorems 2 and 3 are general with respect to the loss function, which suggests their fairness benefits should be complementary to existing fairness-enhancing methods that add regularization terms to the primary loss. To validate this hypothesis, and inspired by feedback from this reviewer and reviewer ``fC4c'', we have now conducted a new experiment where we test the impact of optimizers in addition to the fairness-enhancing method [1] on two new datasets (ProPublica COMPAS [2] and AdultCensus [3]) with gender as the sensitive attribute. The results are presented in the table below for SGD, Adam, and RMSProp. Following [1], we report the average gap in equal opportunity, equalized odds, and demographic parity below (lower is better)vover 10 runs.

Regarding the fairness enhancing method, across all metrics on both datasets, Adam and RMSProp achieve substantially lower (better) fairness gaps than SGD. This experiment demonstrates that the fairness benefits of adaptive optimizers are not limited to standard training but are complementary to and can amplify the effectiveness of existing fairness interventions. This finding has considerable impact, and we will add this new experiment and analysis to the final version of the paper. We also report the case of not using the fairness enhancing method, in which, as expected, Adam and RMSProp achieve better fairness.

[1] Roh, Yuji, et al. "FairBatch: Batch Selection for Model Fairness." ICML (2021).

[2] Julia Angwin, et al. "Machine bias: There’s software used across the country to predict future criminals. And its biased against blacks." ProPublica (2016).

[3] Ron Kohavi. "Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid." KDD (1996).

Have you considered examining other group fairness criteria, such as sufficiency (predictive parity)?

Across recent work, the most frequently used group fairness criteria are equalized odds, equal opportunity, and demographic parity, whereas sufficiency (predictive parity/calibration) is reported less often [1]. We therefore adopted these prevalent criteria as our primary objectives.

[1] Liu, Mingxuan, et al. "A scoping review and evidence gap analysis of clinical AI fairness." npj Digital Medicine 8.1 (2025): 360.

In Fig. 1, why is SGD performing so much better early on at about 20 epochs?

Fig. 1 shows the fraction of 1000 independent runs whose iterate lies inside a fair neighborhood (near the fair minima of $w$=0) in the presence of both mild and severe imbalance. The apparent early lead of SGD at first 20 epochs is a transient crossing effect, not stable convergence. Because SGD has larger effective drift and diffusion terms (please see the stochastic differential equation of SGD in Eq. 11) many trajectories pass through the fair neighborhood early on as they move toward the minima that happens to be less fair. This inflates the early within-fair-neighborhood counts, but those trajectories do not remain there. In contrast, RMSProp’s diagonal normalization slows entry but retains more mass near the fair neighborhood. We will add this discussion to the final version of the paper.