Can our method be useful for general-content image restoration?

This is a very interesting point that definitely deserves an explanation. While the proposed GPI is indeed suitable for evaluating fairness in natural images with complex structures, fairness issues are particularly critical when dealing with human images due to their societal implications (which is why we only evaluated facial images). For example, if a general-content image restoration algorithm performs better on images with complex structures than on images of clear skies, this discrepancy is unlikely to be problematic for practitioners, as long as the algorithm attains good performance overall. Moreover, previous works (e.g., [1]) evaluated fairness with respect to non-human subjects (e.g., dogs and cats), but these studies provide limited insights into human-related fairness issues, which often arise due to subtle differences between images. Expanding our method to other datasets remains an avenue for future work. We will include a discussion about this point in the paper.

How can we train models to achieve perceptual fairness?

Please let us propose a method to train algorithms to achieve the best possible perceptual fairness when the sensitive attributes of each ground truth image in the data is known. In particular, one can train a conditional discriminator that takes the sensitive attribute as an additional input, namely, the discriminator is trained to distinguish between the ground truth images of a group (e.g., women images) and their reconstructed images (e.g., the reconstructions produced for degraded images of women). While training such a discriminator, one can add a regularization to the objective of the restoration algorithm (in an "adversarial" fashion) to minimize the deviation in the discriminator scores for different groups. For example, at each training step of the restoration algorithm, one can average the discriminator scores produced for each group and then minimize the standard deviation of the results. This will ensure that the average discriminator scores produced for the reconstructions of each group would be similar for all groups. While this proposed approach was not explored in our paper, it represents an interesting direction for training fair image restoration algorithms. We will thoroughly describe this proposed approach in our manuscript, as a suggestion for future work.

Can our method be extended to other fields such as image generation?

Yes! Thank you for raising this interesting question. The concept of GPI can indeed be extended to other fields, and in particular to image generation. For example, one can regularize a diffusion model (a "time"-dependent image denoiser) during training to attain similar GPIs for all groups. This can be achieved, e.g., by regularizing the denoiser at each time-step using the "adversarial" technique we described above ("How can we train models to achieve perceptual fairness?"), while making the discriminator time-dependent as well. Incorporating such a regularization would ensure that the output distribution at each time-step of the diffusion would be balanced with respect to the specified sensitive attributes. This implies that the output distribution at the end of the generation process would also be balanced. We will discuss this interesting potential avenue in our paper.

References

[1] Ajil Jalal et al. "Fairness for Image Generation with Uncertain Sensitive Attributes." Proceedings of the International Conference on Machine Learning (ICML), 2021.

Demonstration only on image super-resolution tasks

We opted to illustrate our approach on 12 different super-resolution tasks (4 scale factors and 3 noise levels) simply because of the availability of many methods to compare against on these tasks. Note that this choice aligns with common practice in the field, as related works also focus on image super-resolution [1,2,3], which serves as a standard benchmark due to the availability of numerous algorithms for comparison. Nonetheless, we agree that validating our method on other types of image restoration tasks, such as image denoising or deblurring, or on different modalities like audio, video, or text restoration, would further substantiate its robustness and broad applicability. To address this, we will add experiments with image denoising and deblurring to the appendices and discuss the potential for future work on additional types of degradations and data modalities.

What kind of changes to the existing super-resolution methods might result in better fairness handling?

Please let us propose a method to train algorithms to achieve the best possible perceptual fairness when the sensitive attributes of each ground truth image in the data is known. In particular, one can train a conditional discriminator that takes the sensitive attribute as an additional input, namely, the discriminator is trained to distinguish between the ground truth images of a group (e.g., women images) and their reconstructed images (e.g., the reconstructions produced for degraded images of women). While training such a discriminator, one can add a regularization to the objective of the restoration algorithm (in an "adversarial" fashion) to minimize the deviation in the discriminator scores for different groups. For example, at each training step of the restoration algorithm, one can average the discriminator scores produced for each group and then minimize the standard deviation of the results. This will ensure that the average discriminator scores produced for the reconstructions of each group would be similar for all groups. While this proposed approach was not explored in our paper, it represents an interesting direction for training fair image restoration algorithms. We will thoroughly describe this proposed approach in our manuscript, as a suggestion for future work.

Insights into why certain methods are not good at fairness handling compared to others

Thank you for raising this important point. Please note that our theoretical section (Section 3) attempts to provide such insights. For example, we show that common image restoration algorithms, such as the MMSE point estimate or the posterior sampler, may often lead to poor perceptual fairness. This means that perceptual fairness is not "trivially" acquired by common algorithms. Thus, it is interesting to ask:

1. Under which circumstances can some algorithm achieve perfect GPI for all groups simultaneously (Theorem 2)?
2. Otherwise, when perfect GPI cannot be achieved for all groups simultaneously, under which circumstances can some algorithm achieve perfect perceptual fairness? (Theorems 3 and 4). For example, from Theorem 4 we learn that, among all the algorithms that achieve perfect Perceptual Index (PI), the better ones in terms of perceptual fairness are those that excel on the "toughest" groups, and this is not directly achieved by methods that simply attain perfect PI (e.g., posterior sampling).

References

[1] Ajil Jalal et al., "Fairness for Image Generation with Uncertain Sensitive Attributes." Proceedings of the International Conference on Machine Learning (ICML), 2021.

[2] Yochai Blau and Tomer Michaeli, "The Perception-Distortion Tradeoff." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

[3] Guy Ohayon et al., "The Perception-Robustness Tradeoff in Deterministic Image Restoration." Proceedings of the International Conference on Machine Learning (ICML), 2024.

Demonstration only on image super-resolution tasks

We opted to illustrate our approach on 12 different super-resolution tasks (4 scale factors and 3 noise levels) simply because of the availability of many methods to compare against on these tasks. Note that this choice aligns with common practice in the field, as related works also focus on image super-resolution [1,2,3], which serves as a standard benchmark due to the availability of numerous algorithms for comparison. Nonetheless, we agree that validating our method on other types of image restoration tasks, such as image denoising or deblurring, or on different modalities like audio, video, or text restoration, would further substantiate its robustness and broad applicability. To address this, we will add experiments with image denoising and deblurring to the appendices and discuss the potential for future work on additional types of degradations and data modalities.

How can sensitive attributes be detected and acquired?

Thank you for raising this interesting question. Please note that our work assumes that the sensitive attributes are prespecified, which is the case handled by most works that tackle fairness concerns. It should be noted that the term "sensitive" here is application dependent. For example, in one application gender may be considered the only sensitive attribute, whereas in another application age may be considered sensitive as well. This is despite the fact that both applications use the precise same data for training. Thus, sensitive attributes cannot be detected automatically from the data. Rather, they should be specified by the user according to the particular societal concerns in question. We will discuss this interesting point in the manuscript.

The difference between GPI and standard metrics like PSNR

We appreciate this question and address it in Appendix G.5 (L794 onwards), where we show that such metrics are not good indicators of perceptual bias. Indeed, our experiments (Figures 8-10 in the appendix) show that, while the different groups attain roughly the same distortion (e.g., PSNR, LPIPS), their GPIs differ significantly, a discrepancy that is also visually evident.

How can our method contribute to the design of fair and unbiased image restoration algorithms?

Please let us propose a method to train algorithms to achieve the best possible perceptual fairness when the sensitive attributes of each ground truth image in the data is known. In particular, one can train a conditional discriminator that takes the sensitive attribute as an additional input, namely, the discriminator is trained to distinguish between the ground truth images of a group (e.g., women images) and their reconstructed images (e.g., the reconstructions produced for degraded images of women). While training such a discriminator, one can add a regularization to the objective of the restoration algorithm (in an "adversarial" fashion) to minimize the deviation in the discriminator scores for different groups. For example, at each training step of the restoration algorithm, one can average the discriminator scores produced for each group and then minimize the standard deviation of the results. This will ensure that the average discriminator scores produced for the reconstructions of each group would be similar for all groups. While this proposed approach was not explored in our paper, it represents an interesting direction for training fair image restoration algorithms. We will thoroughly describe this proposed approach in our manuscript, as a suggestion for future work.

References

[1] Ajil Jalal et al., "Fairness for Image Generation with Uncertain Sensitive Attributes." Proceedings of the International Conference on Machine Learning (ICML), 2021.

[2] Yochai Blau and Tomer Michaeli, "The Perception-Distortion Tradeoff." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

[3] Guy Ohayon et al., "The Perception-Robustness Tradeoff in Deterministic Image Restoration." Proceedings of the International Conference on Machine Learning (ICML), 2024.

Complexity of evaluating perceptual fairness

Thank you for raising this important point. We acknowledge that computing the GPI of each group increases the complexity of evaluating fairness compared to previous methods, which typically compute the classification hit rates (e.g., counting the reconstructed images classified as having a specified sensitive attribute). However, it is important to note that both our method and previous methods require each sample to be processed through a classifier. In our method, the classifier is used to extract deep image features to compute metrics such as FID and KID, whereas in previous methods the classifier evaluates the predicted sensitive attributes of each image. Thus, the additional computational overhead of our method arises from two factors:

1. Extracting deep features from the source images in addition to the reconstructed images, effectively doubling the computation required for feature extraction compared to previous methods.
2. Approximating a statistical divergence (e.g., FID) between the extracted features of the ground truth images and the extracted features of the reconstructed images. This additional step introduces some computational overhead (e.g., computing the empirical mean and covariance matrix of the samples, as in FID), but it is relatively minor in practice compared to the benefits of our more nuanced fairness evaluation. We will point this limitation in our paper.

Why do we use synthetic datasets?

We appreciate the reviewer's concern regarding our use of synthetic data sets. Please note that, as discussed in Section 4.1 of our paper, we opted to use synthetic, high-quality datasets because existing face datasets often lack ground truth labels for sensitive attributes such as ethnicity, age, and gender. Moreover, the datasets that do include such labels are typically imbalanced w.r.t. these attributes. To adequately approximate and compare the GPI of different groups, it is essential to have equal amounts of data from each group. Otherwise, approximating the GPI with metrics like FID would lead to completely different scoring scales for varying sample sizes (see, e.g., Figure 1 in [1], which shows that FID suffers from bias when using small sample sizes). Generating synthetic datasets allows us to control the number of images from each group, and therefore allows to ensure balanced and fair evaluations.

Applicability to other types of data besides facial images

This is a very interesting point that definitely deserves an explanation. While the proposed GPI is indeed suitable for evaluating fairness in natural images with complex structures, fairness issues are particularly critical when dealing with human images due to their societal implications (which is why we only evaluated facial images). For example, if a general-content image restoration algorithm performs better on images with complex structures than on images of clear skies, this discrepancy is unlikely to be problematic for practitioners, as long as the algorithm attains good performance overall. Moreover, previous works (e.g., [2]) evaluated fairness with respect to non-human subjects (e.g., dogs and cats), but these studies provide limited insights into human-related fairness issues, which often arise due to subtle differences between images. Expanding our method to other datasets remains an avenue for future work. We will include a discussion about this point in the paper.

How to balance fairness among different groups in practice?

Please let us propose a method to train algorithms to achieve the best possible perceptual fairness when the sensitive attributes of each ground truth image in the data is known. In particular, one can train a conditional discriminator that takes the sensitive attribute as an additional input, namely, the discriminator is trained to distinguish between the ground truth images of a group (e.g., women images) and their reconstructed images (e.g., the reconstructions produced for degraded images of women). While training such a discriminator, one can add a regularization to the objective of the restoration algorithm (in an "adversarial" fashion) to minimize the deviation in the discriminator scores for different groups. For example, at each training step of the restoration algorithm, one can average the discriminator scores produced for each group and then minimize the standard deviation of the results. This will ensure that the average discriminator scores produced for the reconstructions of each group would be similar for all groups. While this proposed approach was not explored in our paper, it represents an interesting direction for training fair image restoration algorithms. We will thoroughly describe this proposed approach in our manuscript, as a suggestion for future work.

How do we apply the proposed GPI to real-world applications?

Many practical imaging systems should ensure that their algorithms do not introduce or amplify biases against any particular group. Mobile phones, for example, are used by everyone, and all of them incorporate image restoration algorithms within the Image Signal Processing (ISP) pipeline to produce a high-quality image from the given sensor measurements. When the degradation is sufficiently severe (e.g., in low-light conditions), even a well-performing image restoration algorithm (e.g., posterior sampler) may treat some groups better than others. Thus, it is important to evaluate the fairness of such algorithms. For example, the GPI can help identify subtle biases that traditional methods might overlook, thereby alerting for fairness issues in these systems.

References

[1] Mikołaj Bińkowski et al. "Demystifying MMD GANs." Proceedings of the International Conference on Learning Representations (ICLR), 2018.

[2] Ajil Jalal et al., "Fairness for Image Generation with Uncertain Sensitive Attributes." Proceedings of the International Conference on Machine Learning (ICML), 2021.