We thank the reviewer for their constructive feedback. We address each point below.

$\newline$

Question 1:

We have shown a controlled experiment on COCO using OPS[1] based shortcuts in Appendix F. But based on your recommendation, we have also added another experiment on the CMNIST dataset to get more empirical proof.

Experiment Setup:

The experiment is controlled by two control variables $\alpha$ and $\beta$ to control the dataset and model bias, respectively.

In the CMNIST dataset, we modify the traditional MNIST dataset by replacing the black and white images with their colored versions. The images are the features ($X$), digit labels are the task ($T$), and color is the protected attribute ($A$). Each label (i.e., digit from 0-9) is correlated with a particular color (i.e., 0- red, 1- blue, 2- green, …). The magnitude of this correlation is controlled using alpha. I.e., if $\alpha =0.7$, 70% of examples of 0 are red, and the remaining 30% are randomly assigned any of the remaining 9 colors. Thus, $\alpha$ allows us to control the bias in the dataset.

A simple CNN model (with depth = 2) is trained on this dataset. The trained model is used to get predictions. The predictions go through post-processing based on $\beta$. $\beta$ defines the percentage of predicted labels that are overwritten based on the color in the input image. I.e., if $\beta=0.1$, 10% of the predicted labels will be randomly selected. These labels will then be replaced based on the color in the image. This allows us to increase the $A\rightarrow T$ bias in the model predictions. Thus, for a fixed value of $\alpha$, $\beta$ allows us to control the model bias. For $Q$ function, accuracy is used.

Results:

$\newline$

$\beta = 0.0$

As we see from the table above, $\Psi_{A}^{D}$ shows a clear increasing trend as the controlled variable $\alpha$ increases. I.e., the dataset bias detected by $DPA$ increases directly as the actual dataset bias (controlled by $\alpha$) increases. This controlled experiment allows us to verify that $DPA$ is capable of measuring the dataset bias.

$\newline$

$\alpha = 0.75$

As we see from the table above, $\text{DPA}_{A\rightarrow T}$ shows a clear increasing trend as the controlled variable $\beta$ increases. I.e., the bias amplification detected by $DPA$ increases directly as the actual model bias (controlled by $\beta$) increases. This controlled experiment allows us to verify that $DPA$ is capable of accurately measuring bias amplification.

We shall be happy to add this experiment in a more detailed fashion in the camera-ready version.

$\newline$

Question 2:

Directional bias amplification metrics (like $BA_{\rightarrow}$ and DPA) can be used to verify if bias mitigation strategies work correctly. Let us revisit an example shown by Wang et al. [2] where the Equalizer model, a model that mitigates gender bias amplification in image captions, accidentally increases $A\rightarrow T$ bias amplification.

Consider an image of a woman holding a book, and the following gender-biased caption: “A man holding a book”. The Equalizer model corrects the gender of this caption: “A woman holding a kite”. However, it incorrectly changes the object in the caption. While the Equalizer model mitigates $T\rightarrow A$, it accidentally increases $A\rightarrow T$ bias amplification.

For this caption, $DPA$ becomes useful if $A$ and $T$ are balanced in the training dataset (ML practitioners often balance datasets, assuming that balancing gives them an unbiased dataset). If $A$ and $T$ are balanced, even though the Equalizer model predicted the object incorrectly (kite instead of book) while correcting gender, the $A\rightarrow T$ bias amplification reported by $BA_{\rightarrow}$ would be 0. This is because $BA_{\rightarrow}$ incorrectly assumes that a model cannot amplify biases if it is trained on a balanced dataset. However, DPA would report the intended positive bias amplification in $A\rightarrow T$.

$\newline$

References

[1] Shutong Wu, , Sizhe Chen, Cihang Xie, Xiaolin Huang. "One-Pixel Shortcut: On the Learning Preference of Deep Neural Networks." The Eleventh International Conference on Learning Representations . 2023.

[2] Angelina Wang and Olga Russakovsky. Directional bias amplification. In International Conference on Machine Learning, pages 10882–10893. PMLR, 2021.

Thank you for your recommendation! It is definitely possible to show the superiority of our metric using the above experiment.

To show the efficacy of our metric, we started with a balanced CMNIST dataset (i.e., $\alpha$ = 0.1). We then varied the $\beta$ parameter from 0.1 to 0.4 to gradually increase the model bias. We showed the bias amplification captured by different metrics in the table below.

Since the dataset is balanced, $BA_{\rightarrow}$ reported zero bias amplification for all $\beta$ values. LA was unable to capture the increasing trend as it is not a directional metric. Only DPA correctly captured the increasing trend. This shows that only DPA can capture directionality in a balanced dataset.

We will add a more detailed version of this experiment in our supplementary for the camera-ready version.

We hope this resolves your queries. If there is anything else we can do to encourage you to increase your score, please let us know!

We thank the reviewer for their insightful feedback. We address their questions below.

$\newline$

Question 1:

Given the time constraints, we were not able to provide any further analysis on the attacker function apart from the one currently present in Appendix D. If you recommend, we could add an experiment showing the reduced sensitivity by varying other model parameters (e.g., architecture, depth, and activation functions) by the camera-ready deadline.

High attacker sensitivity can be especially problematic as it can cause misleading results. High attacker sensitivity can lead to different bias amplification scores for the same model or change the relative magnitude of biases across different models. I.e., for bias amplification in image captioning, Bakr et. al[1] show that the LIC[2] metric is inconsistent in its ranking of bias amplification in different models due to attacker model sensitivity.

$\newline$

Question 2:

To stay consistent with previous works on bias amplification, we measured bias amplification in visual and synthetic datasets on attributes like gender and race. Our experiment on racial biases using the COMPAS dataset (COMPAS is a synthetic dataset) is available in the Appendix – Section E.

Text or NLP-focused datasets (e.g., image captioning datasets) have semantic nuances. For such datasets, there are specialized metrics (like LIC) that incorporate text-based semantics when computing bias amplification. The metrics discussed in this work (and related previous works) are largely focused on classification problems in the visual and synthetic domains. We did not investigate audio datasets as we were not aware of publicly available datasets with protected attribute information. It would be helpful if you could suggest audio datasets that could be studied for bias amplification.

$\newline$

Question 3:

In the table below, we compared the time taken by different metrics to compute bias amplification on the COCO dataset. This computation was performed over an Intel Core Ultra 7 165H processor without any GPU acceleration.

Note that bias amplification is often measured in offline analysis. Hence, execution time is not the utmost priority. Moreover, DPA takes ~ 2 minutes on the COCO dataset, which is sufficient for an offline analysis in most applications.

With regards to training time scaling with dataset size, we expect a linear trend. The training process itself is similar to conventional model training and hence, would follow the same linear trend with respect to increasing dataset size.

$\newline$

Question 4:

Poor choice of attacker models can lead to misleading or unreliable results. If the attacker model grossly underfits or overfits the training data, it can lead to bias amplification being over- or under-reported. Monitoring the training and test loss of the attacker models can help identify such misleading values. We have briefly talked about this in our Limitations section. If you suggest that these details would be beneficial for readers, we can add these details in the Limitations section by the camera-ready version.

$\newline$

Question 5:

There are several potential approaches to analyze individual {A, T} pairs. One approach can be to use feature attribution. Assume we want to measure dataset and model bias in $\ A\rightarrow T$. If we one-hot encode both $A$ and $T$, we can use feature attribution to find out the influence of a specific attribute $A=A_{i}$, in predicting a specific task $T=T_k$. This attribution value can be used as a weak proxy to measure dataset or model bias for this ${A_i, T_k}$ pair. Causal attacker models or interpretable attackers are other avenues for a similar ${A, T}$ pair analysis. We can add a section discussing such ideas for future work.

$\newline$

References

[1] Eslam Mohamed Bakr, , Pengzhan Sun, Li Erran Li, Mohamed Elhoseiny. "ImageCaptioner$^2$: Image Captioner for Image Captioning Bias Amplification Assessment." (2023).

[2] Hirota, Yusuke, Yuta, Nakashima, Noa, Garcia. "Quantifying Societal Bias Amplification in Image Captioning." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022.

Thank you for your insightful and detailed review.

We have answered to your concerns in the responses below:

$\newline$

Minor Comments:

Thank you for pointing this out. We will be removing the term “causality” from our manuscript to avoid any confusion.

$\newline$

DPA clarification:

You are indeed correct, we had simplified the expression to $P(T|A)$. We will modify the manuscript to reflect the expression you provided. Note that this new expression still ensures that the metric is directional. We prove this here:

As a generalization of your expression, $\Psi_{A}^{D}$ can also be written as: $\Psi_{A}^{D} = \sum_{A_{i} \in A} P(A_i) P(T_{mi}|A_{i})$.

Where, $T_{mi}$ is such that $P(T_{mi}|A{i}) \geq P(T_{k}|A{i}) \forall k$.

Similarly, $\Psi_{A}^{M} = \sum_{A_{i} \in A} P(A_i) P(\hat{T_{mi}}|A_{i})$.

Hence, your final expression can be written as: $\sum_{A_{i} \in A} P(A_i) P(\hat{T_{mi}}|A_{i}) -\sum_{A_{i} \in A} P(A_i) P(T_{mi}|A_{i})$. We see that bias amplification is still measured with respect to a fixed a prior variable (in this case $A_{i}$), which is how a directional metric should work.

$\newline$

Section 5.1:

Discussion Section Part 1: You have raised very relevant points. We will add the following clarifications to paragraph 1 of Section 5.1: $DPA_{ A \rightarrow T}$ can be used for model selection, and $DPA_{ T \rightarrow A}$ can be used for identifying spurious correlations in datasets.

Discussion Section Part 2: While addressing your comments for the second part of Section 5.1, we found an error in how we wrote this portion. We mentioned in line 340 — “Compared to $BA_{\rightarrow}$, DPA may under-report bias amplification...” This is incorrect as $BA_{\rightarrow}$ would give a “0” bias amplification if the dataset bias is 0. To convey our point correctly, we updated the second part of Section 5.1. Please read updates here:

Table 1

Table 2

"Based on our experiments, we found $DPA$ to be the most reliable metric to measure bias amplification. However, there are cases where other directional metrics like $BA_{\rightarrow}$ could be more appropriate. Let us take two scenarios where we measure $A\rightarrow T$ bias amplification in $BA_{\rightarrow}$ and $DPA$.

Consider a hiring dataset where $200$ men ($A = 0$) and $100$ women ($A = 1$) apply for a job. Out of these, $50$ men and $22$ women are hired ($T = 0$), while the rest are rejected ($T = 1$) — refer to Table 1. Now, assume we train a model (M) on this dataset, which gives predictions as shown in Table 2.

According to DPA, the $A\rightarrow T$ bias amplification is large. This is because the counts of $T$ and $\hat{T}$ have grown farther apart for both men {initial difference = 100 (150 - 50), new difference = 170 (185 - 15)} and women {initial difference = 56 (78 - 22), new difference = 82 (91 - 9)}. According to $BA_{\rightarrow}$, $A\rightarrow T$ bias amplification is small. This is because the ratio of hired men and hired women is similar in $T$ (25% vs 22%) and $\hat{T}$ (7.5% vs 9%). Here, our goal is to ensure that men and women have a similar acceptance ratio. Since the acceptance ratio between men and women is similar in the dataset and in the model’s predictions, the small bias amplification reported by $BA_{A\rightarrow T}$ is appropriate. In scenarios where we desire equal opportunity (e.g., hiring datasets where both men and women should have equal acceptance ratios), using $BA_{\rightarrow}$ is better as it measures bias amplification by comparing task ratios between men and women.

In another scenario, consider a dataset of men ($A = 0$) and women ($A = 1$), where each person is either indoors ($T = 0$) or outdoors ($T = 1$). Let us consider the same tables for the dataset (Table 1) and the model’s predictions (Table 2). Here, our goal is to ensure that we have a similar count of men and women, indoors and outdoors. Keeping our goal in mind, we should report a high $A\rightarrow T$ bias amplification as the count of $\hat{T}$ has changed significantly with respect to $T$ (for both men and women). Here, the large bias amplification reported by $DPA_{A \rightarrow T}$ is appropriate. In scenarios where we desire equal counts across $A–T$ pairs, comparing task ratios (as done by $BA_{\rightarrow}$) is not meaningful. Instead, the count-based comparison used by DPA better reflects bias amplification. $BA_{\rightarrow}$ and $DPA$ capture different notions of bias, and the right choice of metric depends on the type of bias we seek to address."

$\newline$

Experiment Details:

You are right. We used annotations from the MS-COCO multi-label benchmark. The label $y = (y_1, y_2, …. y_{|T|})$ is a binary variable. $y_{i} = 1$ indicates that at least one instance of the $i^{th}$ object is present in the image. To adapt the ImageNet pretrained models to COCO, we removed the last MLP layer from these models and replaced it with an MLP layer of size |T|. The models were then fine-tuned on the COCO dataset. We will make sure to add these points along with more details on the model training. For the attacker models, the labels are one-hot encoded. This was not shown in the figure for the sake of simplicity. We will clarify this in the text to avoid any confusion.

$\newline$

MLP on COMPAS:

Yes, the 2x2 contingency table should be sufficient in the COMPAS example. For the sake of uniformity, we chose to use the MLP as the attacker function. We can compute DPA using the 2x2 contingency matrix, if you suggest that it adds value to the study.

We thank you for your inquisitive and candid review. Please find our responses below.

$\newline$

Key Weaknesses - Flaw in your results:

The result from our masking experiment is correct. The non-monotonic behavior shown in LA is a result of its conflation of biases from the $A\rightarrow T$ and $T\rightarrow A$ directions.

In the aforementioned experiment, we progressively mask the person to monotonically decrease the $A\rightarrow T$ bias. But, this masking also leads to a monotonic increase in the $T\rightarrow A$ biases because, as the model has less information on the gender ($A$), it relies more on task ($T$) to make its predictions.

As $DPA_{T\rightarrow A}$ measures biases only in the $T\rightarrow A$ direction, it shows a monotonic increasing behavior. On the other hand, LA confounds the biases from both directions. I.e., An increase in $T\rightarrow A$ direction may be offset by decreases in $A\rightarrow T$ direction, resulting in non-monotonic behavior. Please read the next section - [Key Weaknesses - Insufficient novelty over LA] to understand why LA conflates biases.

Thus, the experiment shows us the limitations of LA and how it can give misleading results. Meanwhile, DPA, being a directional metric, can give users a clearer understanding of the bias amplification.

$\newline$

Key Weaknesses - Insufficient novelty over LA:

A major improvement over LA (our novelty) is this — LA measures bias amplification (or change in bias) w.r.t varying priors. DPA (our metric) measures bias amplification with respect to a fixed prior.

What does LA do? To measure dataset bias, LA uses an attacker model to predict the ground-truth attribute $A$ using the ground-truth task $T$ as input. In probabilistic terms, this is proportional to $P(A|T)$. To measure model bias, LA uses an attacker model to predict the ground-truth attribute $A$ using the task predictions $\hat{T}$ as input. This is proportional to $P(A/\hat{T})$. Bias amplification in LA (model bias - dataset bias) is proportional to $P(A|\hat{T}) - P(A|T)$. Here, the prior variable changes from $\hat{T}$ to $T$. The change in bias (or bias amplification) is not grounded w.r.t a fixed variable.

What does $DPA_{T\rightarrow A}$ do? To measure dataset bias, DPA is the same as LA: dataset bias in DPA is proportional to $P(A|T)$. To measure model bias, DPA uses an attacker model to predict the ground-truth predictions $\hat{A}$ using the ground-truth task $T$ as input. In probabilistic terms, this is proportional to $P(\hat{A}|T)$. Bias amplification in DPA (model bias - dataset bias) is proportional to $P(\hat{A}|T) - P(A|T)$. In DPA, the change in bias (or bias amplification) is grounded with respect to a fixed variable (in this case, $T$).

“Fundamentally, a change in bias should be measured with respect to a fixed quantity” — this idea was conveyed by Wang et al. [1] in their paper, and we use the same idea for our work. This is our main improvement over LA (LA does not follow this idea --- thus, it conflates biases in $A \rightarrow T$ and $T \rightarrow A$). We attempted to explain this aspect of our novelty in the Appendix, Section B. If you feel this aspect of our novelty should be highlighted better in the main paper, we would be happy to do that.

Note: $P(\hat{A}|T) - P(A|T)$ is a simplified expression for DPA. The precise expression for DPA is $\sum P(T_{i})P(\hat{A_{mi}}|T_{i}) - \sum P(T_{i}) P(A_{mi}|T_{i})$. This expression also ensures a fixed prior variable. For more details, please read the section: DPA clarification of our response to Reviewer bRtk)

$\newline$

Minor Weakness 1

Thank you for the suggestion. We will add correlation-based scores in our tables to summarize the results in a single row. This will allow readers to quickly parse and compare different metrics.

Minor Weakness 2

Presently, we do not have a "theoretical" justification for why a relative metric is superior. We have shown the practical advantages of using relative metrics over absolute ones, like robustness to different attacker models.

Minor Weakness 3:

We have a controlled experiment on the COCO dataset as described in Appendix F. Herein, we use shortcuts created using the One Pixel Shortcut[2] technique to control the hidden bias. In addition to this experiment, we have added another experiment on the CMNIST dataset (see Controlled experiment in response to Reviewer dTPG).

$\newline$

Question 1: We flipped $\hat{T}$ to ensure $\hat{T}$ and $T$ have the same accuracy value. If $\hat{T}$ and $T$ do not have the same accuracy, bias amplification would be $P(\hat{T}|A) - P(T|A) + f(\text{difference in accuracy})$. Here, $f$ is some arbitrary function. The bias amplification score should only reflect the gap between the model and the dataset bias. This score should not incorporate the gap in accuracy between the model and the dataset. If the accuracy of $\hat{T}$ is $70$%, we flipped $30$% of the labels in $T$ so that $T$ comes down to the same accuracy as $\hat{T}$. This ensures that bias amplification is $P(\hat{T}|A) - P(T|A)$.

$\newline$

Question 2: While we are not fully familiar with multicalibration, we understand that it is a bias mitigation technique. We feel that metrics like DPA can be used to identify models that require or could benefit from multi-calibration.

$\newline$

Formatting Concern: Thank you for pointing this out. For practical use, a small value epsilon ~1e-10 is added to the denominator to ensure numerical stability. We will update the manuscript to reflect this.

$\newline$

References

[1] Angelina Wang and Olga Russakovsky. Directional bias amplification. In International Conference on Machine Learning, pages 10882–10893. PMLR, 2021.

[2] Shutong Wu, Sizhe Chen, Cihang Xie, Xiaolin Huang. "One-Pixel Shortcut: On the Learning Preference of Deep Neural Networks." The Eleventh International Conference on Learning Representations. 2023.