Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias

We thank the reviewer for their comments and address their questions:

“How do the authors substantiate that this accurately reflects actual distribution shift behaviors in the real world?”

All of our settings revolve around a sequence of models trained with datasets that get updated over time (whether by adding synthetic data, synthetic labels, or some mixture of synthetic and ‘real’ data). Concerns of use of synthetic data, whether intentionally or otherwise have already appeared in research and media. Our work seeks to better understand MIDS and opportunities for algorithmic reparation. We do not attempt to reflect distribution shift in general, just those induced by models. We also do not study what the impact of MIDS alongside other types of distribution shift might entail, though we do explore interactions between MIDS.

“Could the authors delve into a discussion regarding how current methodologies address the challenges of model-induced distribution shifts?” 

“Could the authors also present comparisons against established baseline methods?”

Points 2 and 3 ask for a discussion and experimental comparison against related methods. We show one comparison in Appendix E.2, where we provide an ablation study for the amount of synthetic data present at each generation. We also anticipate that training with fairness at each generation may be insufficient to stop unfairness accumulating due to MIDS (see our response to reviewer QqzD, point 4.b). Otherwise, please refer us to works that can address MIDS.

We thank the reviewers for their comments and address their questions below, followed by addressing the weaknesses.

In Figure 4, as the number of generations increases, the accuracy drops from 92% to 82%, and the fairness metrics like DP and EO are still large. Why does a reduction of accuracy and unfairness occur at the same time?

Figure 4 depicts sequential classifiers undergoing performative prediction. In the non-reparative results, there are no fairness interventions and we do not expect the classifiers to achieve lower unfairness. However, each generation’s mistakes continue to compound, lowering the accuracy. The algorithmic reparation is successful in eventually yielding batches with even\ideal class and group representation (Figure 4, bottom row), benefiting the fairness metrics. However, as these even batches no longer match the original distribution’s imbalances, there must be some incorrect classification when compared to the original distribution, which is why the accuracy still decreases. Please also see our response to reviewer qqNH’s first question for more information.

“Why do we need the settings of sequences of generators?”

The settings with sequential generators are motivated as generated content is published online and re-scraped to form new datasets. For example, in the second paragraph of the introduction we discuss the proliferation of AI-generated baby peacock photos which (as of writing) frequently show up in the first page of Google image search results (as found by Shah and Bender (2022)). A generative model trained on these images would misrepresent the baby peacocks. Another example is from Venugopal et al. (2011), when Google researchers realized that training on previous translations published online might harm future versions of their model. Similarly, Veselovsky et al (2023) showed that AI is now being used by MTurkers, suggesting that we already may have a significant proportion of AI labels in our crowdsourced datasets.

Please see our next comment which addresses the weaknesses.

We thank the reviewer for their comments, and address their questions here:

“Q1: How do you select the datasets and what are the motivations for them and the experimental setup exactly?”

Dataset choice: We follow a common experimental setting used in computer vision fairness research, adapting datasets into binary classification and binary sensitive grouping (Arjovsky et al. 2022). We use MNIST and SVHN as they both contain the same number of classes. Their difference is in difficulty (SVHN is harder, usually) and class imbalance when split at 5. These two datasets, despite their similarity, showed very different points of model collapse and opposite behavior when observing the accuracy difference between groups (Figures 14 and 15) as ColoredMNSIT reparation resulted in less accuracy difference unlike the other datasets. We chose CelebA for a much more complex/real-world dataset with many sensitive features and well-documented disparities present in the dataset’s base rates (see Table 2). All of these datasets can either be manipulated (as in Arjovsky et al. 2022) into binary group and binary class classification problems (MNIST, SVHN) or already have these features (CelebA). This allows for ease of evaluation of our metrics and allows us to ablate both group and class imbalance to study the mechanics of MIDS. Experimental setup details may be found in Appendix C, we also added further discussion for the reasons we chose these datasets in Appendix C.1.

“Q2: What are the models that you employ for the classifiers and the generator?”

Models for classifiers are CNNs or ResNets, and generators are beta-VAEs. Full details are in Appendix C.3, or in our code hosted anonymously https://anonymous.4open.science/r/ FairFeedbackLoops-1053/README.md

“Q3: Why does the accuracy drop over time in Fig. 5 and 6?”

The accuracy decreases in Figures 5 and 6 (SeqGenSeqClass) due to model collapse. The accuracy is measured at each generation by inputting a held-out set into the classifier and recording performance. However, each classifier was trained from the distribution represented by a generative model undergoing model collapse. As the generators lose the ability to accurately represent the initial distribution (see Figure 19 to see how MNIST and SVHN digits deform over time), we expect the classifiers to lose their ability to discriminate between classes, and their accuracy should decrease.

Please see our next comment where we address the weaknesses.

We thank the reviewer for their comments, and address their questions here.

What is the reason for the decrease in the equalized odds difference with reparation in Figure 4 not being monotonic?

Equalized odds is the maximum of the difference between the true positive rate (TPR) difference or false positive rate (FPR) difference between groups, and is therefore dependent on the ground truth. In Figure 4, we show our metrics as measured with respect to the original distribution (see Table 2), where there is correlation between the sensitive attribute value and class. The first classifiers in early generations start off unfair wrt the original distribution. As AR starts to balance the training set, the classifiers achieve a minimal unfairness wrt the original distribution (generation 13). However, as AR continues to balance the dataset and reach its ideal (Figure 4 bottom right), in the view of the original distribution the minoritized group is being “unfairly” benefited over the majoritized group, which is why the unfairness increases. When plotted, AR causes the majoritized TPR to decrease until it is lower (by about 0.035 also the final EOdds at generation 40) than the minoritized TPR, showing the “unfair” benefit AR gives to the minoritized group when seen from the view of fairness in the original distribution. There is also the presence of label shift due to the sequential classifiers, but these effects taken together is why we mention that we might not want to achieve fairness in Figures 4-6.

The paper claims that "Generator-side AR improves fairness and minoritized representation", but plots on fairness metrics in Figure 5 do not corroborate this claim.

We find that using the generator-side AR technique does improve fairness and minoritized representation. In Figure 5, this is shown by the purple dotted line where the demographic parity and equalized odds values are lower (fairer) than (a) classifier-side reparation and (b) results without reparation. The generator-side reparation maintains the original minoritized group population at 50% of the total, as in the ColoredMNIST dataset. Because classifier-side AR and baseline results diminish this population, they result in less representation.

I do not understand the claim, "Performative prediction on model collapse leads to higher utility". Please explain what you mean by this.

This claim means that the combined setting SeqGenSeqClass allows model collapse (MIDS on the generative models) and performative prediction (MIDS on the classifiers) to co-exist. Compared to the SeqGenNoSeqClass (i.e., just model collapse), this setting has higher utility (by this we mean accuracy). This result highlights cooperation between the model collapse and performative prediction MIDS as the SeqGenSeqClass classifiers adapt to changes due to model collapse. Please see Appendix E.3 for further discussion.

In retrospect, the wording of this claim is confusing so we amended it to “Performative prediction combined with model collapse yields higher accuracy.”

We also address the weaknesses:

• The paper does not propose a mathematical model for the proposed MIDS scheme. If interested in theoretical backgrounds behind MIDS, please see Shumailov et al. (2023) for a formulation of the SeqGenNoSeqClass setting in terms of a Markov chain and Taori and Hashimoto (2023) for a formulation of SeqClass in terms of calibration.
• the decrease in the equalized odds difference with reparation in Figure 4 is not monotonic. The gap closes for a few generations and then seems to rise up again. The plots on fairness metrics in Figure 5 are even less convincing. Question about Figure 4 is answered above in point 1. The AR scheme as presented in Figure 5 shows that generator-side reparation, while performing with less accuracy, maintains class and group balance and achieves far lower unfairness than the results without any reparation.

[1] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget, 2023.

[2] Taori, Hashimoto. Data feedback loops: Model-driven amplification of dataset biases. InInternational Conference on Machine Learning 2023.