$\text{ID}^3$: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition

We are glad that Reviewer enp6 finds our formulas and algorithm flow clear and that ID$^3$ has advantages over existing models over the past two years. Here we respond to your questions as follows. Hopefully it will address your concerns.

Response to Weakness 1

Thanks for pointing out these interesting works on LDMs. However, we would like to clarify that the primary goal of LDMs differs fundamentally from SFR. These LDMs aim to change styles or expressions given a base face image whereas SFR is to generate a fake face dataset all from scratch in place of the real one to achieve face recognition. Yet, we were curious about LDMs' performance on SFR as well before starting this research, although they were not designed for SFR. Among LDMs, we chose the best performing model InstantID and ran multiple tests on it. We empirically found that the best test result is no better than: 81.5 on LFW, 68.9 on CFP-FP, 62.9 on CP-LFW, 62.9 on AgeDB, 63.8 on CA-LFW, with an average of 68.01. These results are far behind the results of the SFR generative model family: SynFace, SFace, ID-Net, DigiFace, iDiffFace, DCFace and ID$^3$. (Please compare these results with those in Table 1.)

We investigated the reasons: we found that while these LDMs claimed to be ID-preserving in the pixel space, their feature embeddings are not discriminative enough for face recognition, since there was no inductive bias (neither loss functions nor architectures) to achieve face discriminativeness. Hence, there is no reason for them to work reasonably well in SFR. In our submitted work, we followed the prevailing practice in SFR by benchmarking ID$^3$ against SFR generative models other than these LDMs. Thanks for the suggestion, though. We will add these comparisons and discussions to the manuscript.

Response to Weakness 2

The pose angle range and the age range in Line 134 are for theoretical purpose and are the domains of definition only. The extreme values (e.g. -90 in degree, 90 in degree, 100 in age) have low occurrences even in the training dataset and thus has low prior probability to be captured from ID$^3$. We demonstrate the distribution of age and pose in the training set shown in Fig. 2 in the one-page PDF.

From the distribution, we observe that the highest age in our training set is 70; the largest pose is 60 in degree and the smallest -60 in degree. Our model can interpolate within these ranges but is less likely to extrapolate outside of these ranges. To generate face images of large pose and high age, one can collect more such data and add them to the training dataset, which increases their occurrences during training.

Response to Weakness 3

Thanks for the suggestion. For intuitive understanding of our work, we will add more visual results to the manuscript which demonstrate how ID$^3$ works while generating different identities of different poses and ages. Now that the ID embeddings and face attributes affect UNet through the cross-attention mechanism, we visualize the attention map in the UNet given different identities and face attributes, as shown in the one-page PDF.

From the visualization result, we observe that ID$^3$ first determines the general pose and the outline of a face, and then fill in the facial details and background noise. This provides an interesting insight for understanding how the diffusion model works in ID$^3$.

Besides, we would like to divert your attention to Fig. 2 in the main paper, which gives insights for how our proposed ID-preserving sampling algorithm (Alg. 2) works. We observe that in the middle of Fig. 2, the original score function $\nabla \log p(\mathbf{x}_t | \mathbf{y}, s)$ generates low-quality face images with cluttered facial details whereas the adjusted score function $\nabla \log \tilde{p}(\mathbf{x}_t | \mathbf{y}, s)$ produces high-quality results of various attributes. One can find the correspondence between the adjustment of the vector field and the resulting images.

Response to Q1: The resolution of the training and generated images is 64x64.

Response to Q2: It takes roughly one week to train ID$^3$ when using 8 NVIDIA Tesla V100 GPUs.

Response to Q3: The image generation speed is 0.5 images/second.

Response to Q4: For inference, it takes 21G GPU memory.

Response to Q5: It is through cross-attention. The ID information and attribute information both acts as keys and values in the cross-attention mechanism while $\mathbf{z}_t$ acts as queries.

Response to Limitation:

Yes, there indeed exist many diffusion-based models (LDMs) that use ID attributes to assist in generating images. But again they are designed for image generation but not for SFR. We argue that image generation and SFR are two different tasks with two different goals. Face image generation is to change styles or expressions given a base face image whereas SFR is to generate a fake face dataset all from scratch in place of the real one to achieve face recognition. Moreover, these LDMs do not perform reasonably well even when applied to SFR. It is worth noting that our technical contribution goes beyond image generation and identify three important characteristics for successful SFR: intra-class diversity, inter-class diversity and ID preservation. Various technical contributions are proposed in this paper to achieve these three characteristics, respectively. For example: ID-preserving loss, Theorem 3.1 and Algorithm 2 are proposed to achieve ID preservation; anchor generation (solving the Tammes problem) (see Line 191-193) is to achieve inter-class diversity; solving Eq. (9) is to achieve intra-class diversity.

Therefore, our technical contribution consists of not only deriving a conditional diffusion model but also a new ID-preserving sampling algorithm as well as a face dataset generation algorithm that respects face distribution on the face manifold to achieve intra-class diversity, inter-class diversity and ID preservation.

We are glad that Reviewer 37yP appreciates our work for its insight, generality and effectiveness. Here we respond to your questions as follows. Hopefully it will address your concerns.

Response to Weakness 1:

Thanks for pointing it out. Factors contributing to solid FR training are complicated, and it is impossible to disentangle them all from identity and to quantify them all explicitly and accurately. But it does not suggest that this problem is totally unsolvable. Some attributes can be easily quantified and disentangled from identity such as age and pose, while others intertwine with identity such as gender and micro-features. In this paper, for now we explicitly explore those quantifiable attributes (e.g. age and pose) by injecting them into ID$^3$ as conditioning signals and leave the unquantifiable attributes automatically and flexibly captured by the U-Net of ID$^3$ via model training. We find this practice leads to sufficiently good performance on our SFR task. Our proposed ID$^3$ serves as a framework into which more quantifiable attributes can be introduced, which we leave for future exploration. We will add more discussions to the manuscript.

To examine whether the attributes we used in our paper are fit for SFR tasks, we perform the following ablation study: as we increase intra-class pose variation, the SFR performances on cross-pose test sets (including CFPFP and CPLFW) are boosted whereas the performances on cross-age test sets (including AgeDB and CALFW) remain almost unchanged. Results and distribution plots are shown in Tab. 1 and Fig. 1 in the one-page PDF. From these results, we observe that the distribution of pose angle on FFHQ, Dataset by IDiffFace, Dataset by ID$^3$ PoseOnly 1 and 2 are all unimodal. And their performances are inferior to Dataset by ID$^3$ PoseOnly 3 which exhibits multimodal distribution of pose angle. When comparing the results of ID$^3$ (Tab. 1 in the main text) and the results of ID$^3$ PoseOnly 3, we conclude that age is another essential attribute that should be injected into ID$^3$ as conditioning signals. We also would like to divert your attention to Table 2 in the main paper, which shows that without using attributes as conditioning signals, the performance is inferior to our original ID$^3$. These empirical findings suggests that the attributes as conditioning signals are fit for SFR tasks.

Regarding the generalization of the attribute network, for simplicity and efficiency, the attribute network we used in our paper is a lightweight model, which we find performs efficiently and reasonably well in our task. Using a more powerful attribute predictor would further benefit our SFR task.

Response to Weakness 2:

The use of FFHQ as training data is for the sake of fairness, as SFR community all use FFHQ for model training. To have a fair comparison, we make sure in our paper that all competing models are trained on FFHQ and evaluated on standard benchmarks. Besides, we further use a larger dataset, CASIA, for training, as in Tab. 1.

Response to Weakness 3:

According to the original paper of DCFace (see the beginning of Section 5), the authors of DCFace claim that

"For $G_{id}$ which generates ID images, we adopt the publicly released unconditional DDPM [25] trained on FFHQ [36]. For $G_{mix}$, we train it on CASIA-WebFace [29] after initializing weights from $G_{id}$."

This suggests the final performance of DCFace depends on FFHQ and CASIA. Furthermore, we successfully replicated its performance reported in their paper using their official implementation: https://github.com/mk-minchul/dcface.git, which suggests DCFace is indeed trained on FFHQ+CASIA.

Back to the argument that "the improvement over DCFace is marginal", note that ID$^3$ trained on CASIA only and DCFace is trained on FFHQ+CASIA, and yet ID$^3$ outperforms DCFace.

Response to Weakness 4:

Thanks for the suggestion. Here we show more results on the larger data volume with the base scale of 1.2M (>=):

Response to Question 1, 2:

Yes, we did run the relevant tests, as shown in Fig. A.2 in the supplement. Given different face attributes and an $\mathbf{Y}_i$, the intra-class cosine similarity ranges from 0.25 to 0.9 whereas the inter-class cosine similarity ranges from -0.25 to 0.25. This suggests that, given $\mathbf{Y}_i$, an identity do not change as we vary attributes only. For visualization, in Fig. 3 in the main paper, we also demonstrate the invariance of identity when attributes vary.

The alignment between the generated identity and the input embedding stems from two factors: the manifold manifestation of the pretrained FR model $f_{\phi}$ and the generalizability of ID-preserving loss. First, the pretrained FR model $f_{\phi}$ is trained using ArcFace loss, which forces similar face embeddings to have large cosine values on manifold ($S^{d-1}$) and dissimilar face embeddings to have small values. Our proposed Alg. 3 (which solves Eq. (9) using $lb=0.7$ and $ub=0.9$) is based upon this manifold manifestation and ensures that $\mathbf{y}_i$ is similar to $\mathbf{w}_i$ with some inherent variations (intra-class diversity). Second, the ID-preserving loss maximizes the inner product between an input embedding and the embedding of the generated face image; we empirically find this loss function generalizes well to the embeddings that are close to (but not equal to) an anchor embedding on the manifold ($S^{d-1}$). We also experiment with $ub=1.0$, but it does not yield better SFR performance. We believe the choice of $lb$ and $ub$ is critical to this generalization of recognizing $\mathbf{y}_i$. We investigated this effect in Ablation (ii) (see Appendix E).

Question: One area that could benefit from further clarification is the explanation of notations and symbols used in the mathematical formulas. Additionally, the formatting and typesetting of some equations, such as Equation 3, could be enhanced to improve readability and aesthetic appeal.

Response: We are glad that Reviewer ThNX finds our proposed model innovative. And thanks for pointing out these issues regarding the presentation and writing. The quantity $d$ in $S^{d-1}$ is the dimensionality of the face embedding space. We will clarify them in the manuscript and rewrite some equations (e.g. Equation 3) for better readability and aesthetic appeal.

We are glad that Reviewer qiUe appreciates our work in terms of originality, quality, clarity and significance. Here we respond to your questions as follows. Hopefully it will address your concerns.

Weakness 1: Additional tests on further diversified real-world datasets could strengthen the generalization claims.

Response: Thanks for the suggestion. To show the generalization of ID$^3$, we further run tests on RFW and obtain the following comparison results against the best SoTA model (IDiffFace) other than ID$^3$:

Note that RFW (Racial Faces in the Wild) is a comprehensive diverse benchmark for face recognition. It is notable for its diversity, capturing the facial variations within and across different racial groups, including but not limited to Caucasian, Black, Asian, and Indian individuals. It includes faces from different age groups, genders, and a variety of lighting and pose conditions, ensuring a realistic and challenging test bed for face recognition algorithms.

From the table above, we observe that ID$^3$ outperforms IDiffFace by clear margins, which suggests its effectiveness and generalization to different groups of age, pose, ethnics, etc.

Weakness 2: It would be beneficial to have details on the computational demands and scalability of the model when deployed in practical, real-world scenarios.

Response: Thanks for the suggestion. The time complexity of our proposed Algorithm 3 is $O(mNT)$, where $N$ is the number of identities to be generated, $m$ is the number of images per identity and $T$ is the diffusion steps. The execution runtime of Algorithm 3 (for generating a face dataset that contains 500,000 face images with 10,000 identities) is 17.36 hours. This is a lot more efficient than manual effort which usually takes months to establish a dataset. The scalability of ID$^3$ is linear with $m$ and $N$, which suggests that ID$^3$ is promising in generating larger-scale datasets for synthetic face recognition when deployed in practical, real-world scenarios.

Question 1: What measures have been taken to ensure the model's robustness against diverse ethnic and age groups, given the model's reliance on identity embeddings?

Response: ID$^3$ models attributes in two ways. For those attributes that can be explicitly disentangled from identity, such as age and pose, ID$^3$ take explicit measures by treating age as one of conditioning signals, which renders ID$^3$ aware of age variations when generating ID-preserving faces accordingly. This makes ID$^3$ robust to diverse age groups. For those attributes that intertwine with identity, such as ethnics, ID$^3$ take implicit measures by automatically and flexibly capturing them through model training. Specifically, recall that ID$^3$ assumes access to the pretrained ArcFace, on the manifold of which dissimilar groups are separated and similar groups are gathered in terms of ethnics. Through end-to-end training, this characteristic acts as supervisory signals that are propagated back to the diffusion model of ID$^3$ (see Figure 1 in the main text). This makes ID$^3$ robust to diverse ethnic groups.

Question 2: Are there potential improvements or variations in the diffusion model that could further enhance identity preservation without sacrificing diversity?

Response: The proposed model ID$^3$ is inherently a conditional diffusion model conditioned on face attributes. Face attributes may include many aspects. Some can be explicitly disentangled from identity while others are implicitly intertwined with identity. Therefore, ID$^3$ models the former as explicit conditioning signals and models the latter as part of face image features. In this paper, for simplicity, we only show the effect of two major face attributes that are disentangled from identity: age and pose. More such attributes can be added into our generation framework, such as expression. When more such attributes are incorporated into ID$^3$ explicitly, ID$^3$ becomes aware of these attributes upon which ID$^3$ utilizes the remaining information from face image features for better ID preservation enhancement.

Question 3: How does the model perform under constrained computational resources, and are there any strategies for optimizing its efficiency?

Response: The proposed model ID$^3$ is inherently a conditional diffusion model (DDPM) that normally costs much computational resources (e.g. ID$^3$ takes 21G GPU memory for inference; iDiffFace takes 24G GPU memory for inference). When deployed under constrained resources, one can use many efficient methods that aim to optimize DDPM inference efficiency such as DDIM [1], PLMSSampler [2] and SD (Spectral Diffusion) [3] while maintaining SFR performance.

References

[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising diffusion implicit models." arXiv preprint arXiv:2010.02502 (2020).

[2] Liu, Luping, et al. "Pseudo numerical methods for diffusion models on manifolds." arXiv preprint arXiv:2202.09778 (2022).

[3] Yang, Xingyi, et al. "Diffusion probabilistic model made slim." Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2023.