UIFace: Unleashing Inherent Model Capabilities to Enhance Intra-Class Diversity in Synthetic Face Recognition

We are glad that Reviewer zDSW appreciates our work in terms of novelty and effectiveness. Here are responses to the questions you raised, hoping to address your concerns.

W1: Thanks for your suggestion. Here we include additional experiment involving 30K identities * 50 images/identity, total 1.5M images. It is worth noting that the most recent works utilize maximum ~1M synthetic images to train FR models and our method achieves comparable performance to their ~1M datasets with just 0.5 million images. As the number of identities increases from 0.5M to 1.5M, the performance of UIFace exhibits continuous improvement, demonstrating the scalability of our method.

W2: Thank you for noticing this point. Your view about the factors that can have an impact on synthetic-based FR is very accurate and comprehensive, i.e, both intra-class diversity and inter-class discrepancy are indispensable.

Since our paper mainly aims to address intra-class diversity, we directly adopt the strategy from DCFace [1] to ensure inter-class discrepancy. Specifically, we filter out similar synthetic identities by calculating cosine distances using a pretrained FR model and then use these refined non-existent identities to generate synthetic datasets in section 3.5. Such strategy from DCFace guarantees the discrepancy among synthetic identities in our paper.

We will include this explanation in revised version.

[1] Kim M, Liu F, Jain A, et al. Dcface: Synthetic face generation with dual condition diffusion model[C]//Proceedings of the ieee/cvf conference on computer vision and pattern recognition. 2023: 12715-12725.

W3: First and foremost, we would like to clarify a few points about our method for better understanding.

Our training process closely adheres to previous DDPM-based methods such as IDiff-Face [1]. The key improvement in training phase is the introduction of an additional empty context $c_e$. And the loss is consistent (equation 3) for both conditioned and unconditioned denoise.

To sum up, the training is one-stage, and the loss is unified for both conditioned and uncondtioned denoise, same as vanilla DDPM. The difference lies in the conditions, which can be identity contexts or the empty context. And the two-stage strategy in our paper refer to sampling stage, not training stage.

Perhaps some of our expressions such as "denosing stage", which can be used for both training and sampling, lead to misunderstandings. We will revise the phrasing and enhance readability in later version.

[1] Boutros F, Grebe J H, Kuijper A, et al. Idiff-face: Synthetic-based face recognition through fizzy identity-conditioned diffusion model[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 19650-19661.

We are glad that Reviewer TRrB appreciates our work in terms of the improvements in synthetic dataset diversity and face recognition performance. We provide more details to help address your concerns.

W1: First, it is important to note that Figure 2 illustrates our key observation: the model initially recovers identity-irrelevant contents and later restores identity-relevant details in sampling process. This observation underpins our adoption of the two-stage denoising strategy in diffusion model's sampling process.

And here we substantiate our observation through analysis in the frequency domain. Firstly, we posit that identity-relevant information predominantly resides in higher frequency components; a notion supported by prior research. Specifically, research [1, 2] shows that despite low-frequency components containing most of the image's energy, removing them does not hinder the performance of FR models, which continue to perform well with only high-frequency components present. These results indicate that the low-frequency components mainly consist of identity-irrelevant contents, thus not affecting the performance of FR models. On the other hand, the high-frequency components contain id-relevant details, facilitating accurate recognition performance with only high-frequency features available.

Secondly, we demonstrate that high-frequency information is restored in the later stage of reverse process, whereas the low-frequency components are restored in relatively early stage. Consider a continuous diffusion process $\mathbf{x} _t = \mathbf{x} _0 + \int^{t} _{0} g(s) d\mathbf{w} _t .$ where $g(s)$ and $d\mathbf{w}_t$ stand for noise schedule and white noise. In frequency domain, we have $\hat{\mathbf{x}} _t(\omega) = \hat{\mathbf{x}} _0(\omega) + \hat{\mathbf{\epsilon}} _t(\omega)$ where $\hat{\mathbf{\epsilon}}_t$ is the Fourier transform of noise term that the expectation $\mathbb{E}[\hat{\mathbf{\epsilon}}_t(\omega)] = 0$ and variance $\mathbb{E}[|\hat{\mathbf{\epsilon}} _t(\omega)|^{2}] = \int^{t} _{0} |g(s)|^2 d\mathbf{w} _t$.

The power spectral density (PSD) of signal $\mathbf{x}_t$ is as follows, $S _{\mathbf{x}_t}(\omega) = \mathbb{E}[|\hat{\mathbf{x}} _t(\omega)|^{2}] = \mathbb{E}[|\hat{\mathbf{x}} _0(\omega) + \hat{\mathbf{\epsilon}} _t(\omega)|^{2}]$ $= \mathbb{E}[|\hat{\mathbf{x}} _0(\omega)|^2] + \mathbb{E}[|\hat{\mathbf{\epsilon}} _t(\omega)|^2] + 2*\mathbb{E}[\hat{\mathbf{x}} _0(\omega)]*\mathbb{E}[\hat{\mathbf{\epsilon}} _t(\omega)]$ $= \mathbb{E}[|\hat{\mathbf{x}} _0(\omega)|^2] + \int^{t} _{0} |g(s)|^2 d\mathbf{w} _t.$ The Signal-to-Noise Ratio (SNR) can be written as $SNR _{\mathbf{x} _t}(\omega) = \frac{|\hat{\mathbf{x}} _0(\omega)|^{2}}{\mathbb{E}[|\hat{\mathbf{\epsilon}} _t(\omega)|^2]} = \frac{|\hat{\mathbf{x}} _0(\omega)|^2}{\int^{t} _{0} |g(s)|^2 d\mathbf{w} _t}.$ Equation above indicates that SNR is only dependent on the PSD of the original image $\hat{\mathbf{x}}_0(\omega)$ since the PSD of white noise is constant at different frequency. As a natural signal, PSD of the image is predominantly distributed in the low frequencies. Therefore, we can assume that $|\hat{\mathbf{x}}_0(\omega)|^{2} \propto |\omega|^{-\alpha}$ So we have $SNR _{\mathbf{x} _t}(\omega) \propto \frac{|\omega|^{-\alpha}}{\int^{t} _{0} |g(s)|^2 d\mathbf{w} _t}$ The equation above demonstrates that

• when timestep $t$ is fixed, higher frequency components have a smaller $SNR$
• as $t$ increases, $\int^{t}_{0} |g(s)|^2 d\mathbf{w}_t$ grows, leading to a decrease in $SNR$ for all frequencies

The process of adding noise to disrupt image can be considered as a process of reducing $SNR$. Therefore, the higher frequency components are obscured for smaller $t$ in the diffusion forward process, hence restored in the later stage of diffusion reverse process, while the low-frequency components are restored in relatively early stage. It is worth noting that, in addition to the analysis above, [3] also provide experimental evidence to support the aforementioned points. Please refer to it for further details.

Combining the above two statements, we can draw a conclusion that the recovery of identity-relevant properties is performed in the later stage of sampling, while identity-irrelevant contents are restored in relatively early stage.

[1] Mi Y, Huang Y, Ji J, et al. Duetface: Collaborative privacy-preserving face recognition via channel splitting in the frequency domain[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 6755-6764.
[2] Mi Y, Huang Y, Ji J, et al. Privacy-preserving face recognition using random frequency components[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 19673-19684.
[3] Qian Y, Cai Q, Pan Y, et al. Boosting Diffusion Models with Moving Average Sampling in Frequency Domain[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 8911-8920.

W6: about privacy and legal issues

As stated in section 3.5, we utilize an additional unconditional face generation model to generate non-existent identities. Based on these identities, we use our method to generate synthetic datasets to train FR models. In other words, all images used to train the final FR model are derived from non-existent identities.

As mentioned in [1, 2], using such synthetic face datasets to train FR models can address privacy and legal issues caused by real datasets, which are directly collected from the internet without the consent of the individuals. And such argument has been widely acknowledged and adopted in rencent works such as [3, 4, 5].

[1] Regulation P. Regulation (EU) 2016/679 of the European Parliament and of the Council[J]. Regulation (eu), 2016, 679: 2016.
[2] DeAndres-Tame I, Tolosana R, Melzi P, et al. Frcsyn challenge at cvpr 2024: Face recognition challenge in the era of synthetic data[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 3173-3183.
[3] Kim M, Liu F, Jain A, et al. Dcface: Synthetic face generation with dual condition diffusion model[C]//Proceedings of the ieee/cvf conference on computer vision and pattern recognition. 2023: 12715-12725.
[4] Qiu H, Yu B, Gong D, et al. Synface: Face recognition with synthetic data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 10880-10890.
[5] Boutros F, Grebe J H, Kuijper A, et al. Idiff-face: Synthetic-based face recognition through fizzy identity-conditioned diffusion model[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 19650-19661.

W7: Here, we conduct more experiments about ablation and different settings of $t_{0}$ as you suggested.

It can be observed that for the fixed strategy, as $t_0$ decreases, the accuracy of the final FR model continues to increase. However, when $t_0$ is less than 500, the training of the recognition model collapses (random guess：50%).

This is because a smaller $t_0$ implies a longer first stage in the sampling process (unconditional generation), which enhances diversity but decreases the intra-class consistency (discussed in Line 438-448). The diversity helps improve the recognition performance. However, when $t_0$ is too small, the intra-class consistency of the generated dataset is insufficient, resulting in training collapse.

In contrast, our adaptive strategy outperforms all settings of fixed strategy, without the need to manually select the optimal $t_0$.

Q4: Sorry for the unclear presentation of Figure 2. Figure 2 illustrates the intra-class similarity of images generated with different intervals where identity context $c$ is used. For example, [0, 500] on the x-axis represents that, during the entire sampling process where $t$ decreases from 1000 to 0, in the interval of [0, 500] the model is conditioned on identity context $c$, while in the remaining interval [500, 1000] is conditioned on the empty context $c_e$. The trend in the red box indicates that increasing the length of identity context-conditioned interval in the early sampling stages (larger t) has little effect on intra-class similarity, which demonstrates our motivation that in later stage the model recovers the identity-relevant details while restores identity-irrelevant contents in early stage.

We appreciate Reviewer JbLs's acknowledgment of the solidity, originality, clarity and significance of our work. We are glad to address your concerns and provide further details.

W1: The motivation behind our approach lies in the diffusion model's different focus of identity-relevant and irrelevant attributes during sampling, which, to the best of our knowledge, has not been revealed or discussed in previous works.

Moreover, our method effectively addresses the diversity gap observed in synthetic versus real face datasets, a gap that has previously led to diminished FR performance. By generates more diverse synthetic datasets, our method achieves state-of-the-art performance with even half the number of identities. Furthermore, when further increasing the number of identities, our method can even match the performance of real datasets. These results demonstrate the scalability and effectiveness of our approach.

W2: Since our motivation lies in different denosing preference in different stages of diffusion reverse process, our method can be applied to various diffusion-based generation methods.

Here we choose FPNDM [1] as a representative and conduct more experiments below (the results related to DDIM have already been included in the paper). As demonstrated by the results, UIFace achieves consistent improvements on both DDIM and FPNDM, showing the effectiveness of our method.

[1] Liu L, Ren Y, Lin Z, et al. Pseudo Numerical Methods for Diffusion Models on Manifolds[C]//International Conference on Learning Representations.

W3: We will carefully review and correct grammar and typo errors in the later version.

W4: We have discussed synthetic-based face recognition in the related work section 2.3 and line 53-91 in the introduction. As you suggested, we will add more discussion of synthetic-based face recognition methods in the revised version.

W5: We have compared the latest ECCV24 work Arc2Face in the paper at the time of our paper submission. Here, we add more comparisons of our method with:

• methods from SDFR: Synthetic Data for Face Recognition Competition [1]
• latest works from NeurIPS24 including [2] and [3], which are not released at the time of our paper submission.

Our method still outperforms them with fewer images used, demonstrating the effectiveness of our method.

[1] Shahreza H O, Ecabert C, George A, et al. Sdfr: Synthetic data for face recognition competition[C]//2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 2024: 1-9.
[2] Xu J, Li S, Wu J, et al. ID^3: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[3] Sun Z, Song S, Patras I, et al. CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems.