From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Bias

The approach is rather simplistic, which may affect the significance of its contribution. We agree with the reviewer that the approach is simple. However, it addresses an issue not solved in prior work and is effective, which is a strength, not a weakness. Moreover, our contribution also involves uncovering a critical issue in using synthetic data for bias mitigation and providing a theoretical analysis in addition to the proposed method that fixes the issue. While there is some theoretical support for the notion that the model could learn to distinguish 'real or fake,' a quantitative analysis is necessary. This could involve assessing the model's performance on a discriminator, for instance.

We provide further insight into this phenomenon by providing the worst group accuracy over the synthetic data for our benchmarked synthetic augmentation methods (USB and ASB) and our method (FFR). Indeed, if the model leveraged such artifact-based correlations, then the model will be biased not only on the real data but also on the synthetic data too. Observe the results in the table below:

Note how both USB and ASB perform poorly on both the real and synthetic data. This indicates that rather than using synthetic data to mitigate the bias and generalize to real data, both methods learned the bias on both real and synthetic data. However, our method (FFR), doesn't suffer from this issue. This is clear from achieving the best worst-group accuracy on both the synthetic and real data.

The notations employed in the script do not adhere to a professional standard, leading to potential confusion in interpretation.

We thank the reviewer for catching the mistake with $P_{D}(Y|B) \neq P(B)$. We believe that this is a typo as all the rest of our notation, in fact, correctly references the dataset on which the probability is calculated. We will fix this in the final version.

How is the balancing of synthetic data achieved?

As discussed in our work, we ensure that every group in the synthetic pretraining dataset has the same number of samples. And yes, we require that the practitioner is aware of which biases they seek to address.

It seems that the phenomenon of 'catastrophic forgetting' [1] in continual learning could potentially occur when transitioning from step 1 to step 2 if all model parameters are fine-tuned.

We train the entire model and we note significant improvements from steps 1 to 2 as we can see from the table below, which indicates that our model doesn’t suffer from catastrophic forgetting.

The paper fails to add comparison methods to address the problem of the newly introduced synthetic artifact bias.

We note that the reviewer is mistaken. GroupDRO, DFR, and resampling are applied on $(B, G)$ to address the synthetic-real bias. In fact, as we discuss in our work, this significantly improves the performance of synthetic augmentation methods (ASB and USB) due to precisely these methods addressing this issue. However, as we note in our results, they don’t effectively mitigate synthetic-real bias in high bias setting due to as the reviewer cited in our work “increase in the optimization difficulty”.

The paper also fails to compare with methods that infer group (or environment) labels, including LfF [3], George [4], EIIL [5], JTT [6], PGI [7], DebiAN [8], and CnC [9]. By inferring group labels instead of using the predefined one, these methods have the opportunity to mitigate the newly introduced artifact bias (or even other unknown biases).

Note that the methods the reviewer suggests are developed to account for cases when bias labels are not available at training time, but we can always expect to have those labels in our setting since we can assume we know what images we generated. However, one could argue that the methods suggested by the reviewer are unsupervised methods, which tend to underperform those with supervision (i.e., those who assume they know what the bias groups are and aim to effectively take advantage of this information). Indeed, one of our benchmarked methods is DFR which is state of the art in the unsupervised setting. While it doesn’t infer group labels on the training set, it uses a balanced labeled validation set to fine-tune a linear classifier.

These methods have the opportunity to mitigate the newly introduced artifact bias

The real-synthetic group labels are trivially inferred because we know which samples are generated by a generator. While methods that infer group labels could be deployed to infer such labels, it would simply attempt to estimate labels that are already known. Nevertheless, even if we did not have these labels, identifying those with generator bias can be accomplished by simply deploying models that predict which images are generated by stable diffusion such as [1] which achieves a high accuracy.

High bias settings are unrealistic.

We disagree with the reviewer. In fact, SpuCO animals naturally have a 95% bias and CelebA have many attributes with over 99% bias like “5-o-clock-shadow” and “Wearing_Lipstick”. However, we don’t test on these attributes as there are not enough samples in the test set for minority subgroups to estimate the worst group accuracy. Therefore, we pick Smiling, where there are enough samples in the validation and test set (at least 15 per group) and artificially vary the bias ratio.

First, there is no guarantee that SD will faithfully follow the text prompts for image generation.

First, we note that our approach is generative model agnostic and that it will still work with future developments in generative models that address such issues. Either way, it is clear that the synthetic data from Stable Diffusion on average generates correct images in our benchmarks since synthetic data augmentation, as we show in our results, significantly improves performance. Moreover, SD has also shown great improvements in the quality of synthetic data generation. However, as we show in our work, it still leaves artifacts in images that could be exploited by recognition models, which is what our work primarily addresses.

New biases that are not controlled (e.g., explicitly specified) by the text prompts may also be introduced.

We agree with the reviewer and discuss this issue in the limitations section. Indeed, future work could benefit from building fairer and more robust generative models. Either way, we note that our approach is generative model agnostic and that it will still work with future developments in generative models that address such issues.

The paper fails to discuss many works on “uncovering spurious correlations”.

We thank the reviewer for pointing this out. We will expand our related section and discuss the relevance of the reviewer-cited work. Note, however, that none of these papers explore the same setting as our work, so are only loosely related.

The main issue of the paper is that it ignores the (large-scale) datasources used to train the generative model, and how they can be leveraged as simple baselines for the proposed method, by either finetuning a pretrained classification model (like OpenCLIP [1]) or retrieving images from these datasets to reduce the spurious correlations. Alternatively, a fair comparison would be to use a generative model trained only on the datasets being evaluated.

We thank the reviewer for the comment. First, we note that many large scale generative models are trained on non-public datasets, which means that there are already many problems for which we may be able to take advantage of the pretrained model, but not the data. As these types of models continue to be explored, we expect that this will only increase during time.

Second, we note that retrieving images from these datasets to reduce the spurious correlations is not a trivial task. It is computationally expensive to sort and categorize through datasets like LAION given their sheer size, and rare categories may be challenging to even detect as inaccurate models used to filter them may result in many false positives. However, using generative models like Stable Diffusion is much cheaper, and as our work demonstrates, can be done in an automated way and still boost performance. Therefore, we deem methods that rely on filtering these datasets as out of the scope of our work.

or by finetuning a pretrained classification model (like OpenCLIP [1])

We agree with the reviewer that this is an interesting question. In fact, recent work [1] has noted that fine-tuning models like CLIP has to be done with care so the bias is not learned. Thus, we would expect that our work would also benefit CLIP, as it could simply be used as a drop-in replacement. However, we explore ResNet-50 models for a fair comparison to prior work [2,3,4].

Alternatively, a fair comparison would be to use a generative model trained only on the datasets being evaluated

One of our main paper observations is that synthetic data is most useful in high bias settings (99.9% bias) where minority subgroups have as few samples as few as 10 samples. Thus, obtaining useful synthetic data for the minority subgroups by training generative models only on the evaluated dataset is likely impossible. Thus, this motivates our use of Large Scale generative models, where these samples may have been seen and learned due to the significant training datasets. Additionally, we note that all the methods we compare to use the synthetic data during training, so our comparisons are fair.

Overall, we note that our main contribution is noting a critical issue using generative models for bias mitigation. Our analysis and method are agnostic to the generative model being used as artifacts are a consistent problem across all current generative models [5]

The experiment does not show results for FFR without rebalancing, which is a simpler setting, as one doesn't need to know the {bias} attributes that cause the spurious correlations to generate images. It has been shown that just randomization can tackle domain missmatch and alleviate spurious correlations [2], making models perform well during evaluation, and this effect is not considered. Given that the generative model is powerful, it is possible that it is introducing extra variation (e.g. lots of different poses, background...), that biases the model to ignore the spurious correlation. To test that the hypothesis that rebalancing is the main driver of improved performance and not extra random diversity gained by the usage of large scale generative models, it would be good to compare the proposed rebalanced sampling method for the prompts to random sampling (i.e. instead of A photo of {bias} {class}, A photo of {class}, or A photo of {random word} {class}).

We thank the reviewer for this comment. Indeed, this is an interesting question. We will include such questions in the revised version of our work.

[1] Debiased Fine-Tuning for Vision-Language Models by Prompt Regularization, Beier Zhu1, Yulei Niu2*, Saeil Lee3, Minhoe Hur4, Hanwang Zhang, in AAAI 2023

[2] Kirichenko, Polina, Pavel Izmailov, and Andrew Gordon Wilson. "Last layer re-training is sufficient for robustness to spurious correlations." ICLR 2023

[3] Sagawa, Shiori, et al. "Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization." arXiv preprint arXiv:1911.08731 (2019).

[4] Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz, “Simple data balancing achieves competitive worst-group-accuracy,” in CLeaR, 2022.

[5] Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Giovanni Poggi, Koki Nagano, and Luisa Verdoliva. On the detection of synthetic images generated by diffusion models, ICASSP 2023

FFR, the central contribution of this work, is based on the assumption that a model can differentiate between real and synthetic images and is likely to leverage correlations that are not stable between these two sources of data to make predictions, which could lead to biased predictions in the real dataset. However, there is no discussion or result (theoretical or empirical) in the manuscript to support this rather strong claim. (See the next section for questions related to this point)

We conduct a theoretical analysis where we prove that assuming these artifacts exist, then models should pick up on these correlations as the dataset distribution that combines real and synthetic data would be biased. Refer to Theorem 1 for more details. In addition, we cite prior work [1] that shows how artifacts are present in even modern generative models like Stable Diffusion. Therefore, our method, by separating the two data sources, achieves significant improvement in performance. We provide further insight into this phenomenon by providing the worst group accuracy over the synthetic data in addition to real data for our benchmarked synthetic augmentation methods (USB and ASB) and our method (FFR). Indeed, if the model leveraged such artifact-based correlations, then the model will be biased not only on the real data but also on the synthetic data. Observe the results in the table below:

Note that our method not only achieves the best worst group accuracy on real data but also on synthetic data. This means that our method is not relearning the bias on both real and synthetic data but is effectively using the synthetic data to mitigate the bias and generalize to real data.

How would the performance of the approach be impacted by low-quality generated data? More specifically, what would happen in case the generative is not able to generate some classes in the original dataset with good quality?

We perform the pixelate effect from [2]. We provide severity 5 which is the highest level. Observe the results of the synthetic augmentation methods below:

Note that while the performance drops due to the deterioration of quality, our method remains the most effective among the other synthetic augmentation methods.

The authors consider a two-step training approach but only report and discuss in the manuscript the performance of the second step.

Observe the results of step 1 and 2 broken down on SpuCO animals:

Note that Stage 2 (second line in table) by itself yields poor performance. This is expected as Stage 2 amounts to training a model with ERM on the biased data without pertaining on synthetic data. Thus, the model, as expected, learns the bias. Training with Stage 1 by itself (first line in the table) while improving performance over Stage 2 by itself, doesn't match the performance of FFR (Stage 1 → Stage 2). This is likely due to the real and synthetic data distribution gap. Therefore, following up Stage 2 with Stage 1 is important to bridge the performance gap.

[1] Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Giovanni Poggi, Koki Nagano, and Luisa Verdoliva. On the detection of synthetic images generated by diffusion models, ICASSP 2023

[2] Michaelis, Claudio and Mitzkus, Benjamin and Geirhos, Robert and Rusak, Evgenia and Bringmann, Oliver and Ecker, Alexander S. and Bethge, Matthias and Brendel, Wieland, Benchmarking robustness in object detection: Autonomous driving when winter is coming