CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition

W1: Thank you for your insightful suggestion and positive feedback. We assume the benefits of the semi-hard training face images could be attributed to:

**(1) **easy training samples are typically images where the face is clear, well-lit, and faces the camera directly, and thus training on such easy samples would not allow the trained FR models to be able to generalize for face images with large pose/age/expression variations and different lighting conditions/backgrounds that are frequently happened in real-world applications. AdaFace[3] also mentioned that easy samples could be beneficial to early-stage training, while hard sample mining is needed for achieving generalized and effective FR models;

(2) hard samples normally contain noise data. Specifically, FaceNet[28] demonstrate that the hardest sample mining using a large batch size leads to hard convergence and produces inferior performance. This is because training with very hard samples may not allow FR models to learn effective features but focus on cues apart from facial identities;

(3) Semi-hard samples generated by CemiFace mostly contain large posed faces but less face-unrelated noises. We also evaluate the training epochs needed to reach the highest AVG performance for easy samples ($m=0.7$), semi-hard samples($m=0$) and extreme hard samples ($m=-0.5$). Easy samples take 10 epochs to reach the best AVG and 20 epochs to produce the training loss of 0; Semi-hard samples take much longer (38 epochs) to provide the highest AVG while the final training loss is around 3; and FR models training on extreme hard samples could not converge.

Due to the real inquiry center is not available, we further calculated the average similarity (using ArcFace to fairly compute them, i.e., avoiding information from adaface and cosface) between randomly paired images, based on random selected 200 identities from CASIA-WebFace, DCFace and CemiFace, where real face images in CASIA-WebFace provide average similarity of 0.51, while face images generated by our CemiFace has a similarity of 0.48. However, face images generated by DCFace are more different from the real face images. with a much easier average similarity of 0.57.

We have also additionally added an experiment to demonstrate the actual similarity to the inquiry data in the PDF Tab. 2. It shows that our generated face images exhibit larger distances (lower similarity) to the inquiry centers than the DCFace. We have also calculated the number of face images belonging to different similarity groups for CemiFace and DCFace in the PDF Tab. 3, indicating that our CemiFace tends to generate images showing lower similarities to their identity centers (i.e. all samples are semi-hard), while DCFace containing more easy samples.

W1: Low GtR is not attributed to real inquiry data. For instance, even with the same synthetic data DDPM, our CemiFace would surpass the previous state-of-the-art method DCFace inquired by DDPM data (clearly illustrated in the upper part of Tab. 4), and DDPM provides close performance to real-data. In fact, low GtR is attributed to the way of constructing the facial dataset, i.e. containing center-based semi-hard samples.

To calculate GtR with fair comparison with other SOTA methods, we train the DCFace released data to generate close AVG performance to their paper by using CosFace loss. This is because the training SFR code of previous methods is not available which prevents us from reproducing the results. Notably, our reproduced DCFace is higher than the results reported in [23]

---

W2: Only images belonging to one identity can be obtained from each input query regardless of the input m value. Despite that face images generated from the same query exhibited large differences, our approach defines them with the same identity label for the later SFR model training. Consequently, the number of identities is fully decided by the number of inquiry face images, which is mentioned in Fig. 1 stating that: "With our proposed CemiFace, each inquiry image finally forms a novel subject.". We will further explain this in the revision.

---

W3: (1) high inter-class variations: Each inquiry face image is selected to be highly independent on other inquiry images. Specifically, we follow DCFace to use a pre-trained FR model to keep samples with a threshold of lower 0.3. We also elaborated this in lines 287-289.

(2) high intra-class variations: high intra-class variations are ensured by (a) changing the similarity condition $m$, as a small input similarity $m$ results in the generated semi-hard images belonging to the same identity having long distances to the identity center; and (b) the face images of the same identity generated by CemiFace are distributed in all directions from the identity center, which can be observed from supplementary material TSNE Fig. 7. This is guaranteed by randomly sampled Gaussian noises $\epsilon$ input to the diffusion model, which exhibit a large variation. As a result, both properties would ensure the generated face images of the same identity are almost evenly distributed in a sphere that has a relatively large radius, and thus they would have high intra-class variations.

---

W4: Despite that our approach achieved slightly worse FID performance than the previous state-of-the-art DC face, face images generated by our approach still result in a better face recognition performance, suggesting that the semi-hard face images generated by our CemiFace compensate for their slightly worse FID compared to face images generated by DCFace with a mix of easier and hard samples. However, CemiFace still significantly outperforms DigiFace in this FiD. On the other hand, high-quality data is essentially needed, as discussed in Section 4.2.2 (lines 263-282, Table 4) and supplementary material Sections B.1 & B.2

We rephrase the last sentence in B.3.3 as: 'Our method doesn’t intend to generate images similar to the distribution of CASIA-WebFace, but to construct a discriminative dataset that is conducive to providing highly accurate FR performance'

---

W5: $F_1$ and $F_2$ are two stacked linear layers. Here, $F_1$ projects the input similarity m to a latent feature $C_\text{sim}\in \mathbb{R}^{512}$, and then $F_2$ projects the feature concatenating $C_\text{sim}$ and identity embedding to a condition vector $C_\text{att} \in \mathbb{R}^{128} = F_{2}(cat(E_{id},F_{1}(m)))$ including both the input similarity and identity information. We have provided more details about Figure 3 and the training pipeline in general response G1 and the PDF.

---

W6: we kindly disagree. High inter-class variation is ensured by filtering the inquiry data with a cosine similarity lower than 0.3 provided by pretrained FR model, following DCFace (mentioned in lines 288-289). As for high intra-class variations, our proposed diffusion model can generate high-variation samples belonging to each identity. Please refer to the answer of W3.

---

W1: In the last part of the Introduction section, we have clearly and specifically listed four contributions of our work, including: 1. a new and crucial finding; 2. a technical contribution (i.e., CemiFace face image generator) inspired by the finding; 3. an application contribution of our proposed technical approach; and 4. the effectiveness of our approach. Based on your suggestion, we will additionally add a paragraph at the beginning of the Method section to guide readers as: In Section 3.1, we first investigate the relationship between sample similarity and their effectiveness in training FR models, presenting the finding that samples with certain similarities (i.e., center-based semi-hard samples) to their identity centers are more effective for training FR models on a real dataset and subsequently devise a toy experiment to validate it. Inspired by our findings, we propose a novel CemiFace, a conditional diffusion model that produces images with various levels of similarity to an inquiry image in Section 3.2. Specifically, Section 3.2.1 introduces how we construct the similarity condition which is fed to diffusion model to guide the generation, and discusses the $L_{SimMat}$ to require the generated sample to exhibit a certain similarity degree to the inquiry image. In Section 3.2.2, we then present how to use our diffusion model to generate a synthetic face dataset given a fixed similarity condition $m$ and a set of inquiry images

---

W2: As clearly presented in line 174 (above the Eq. 10), Eq. 10 is not related to the time step, instead it represents the loss for reconstructing the identity of the inquiry image when $t \rightarrow 0$, which is the first part of our Time-step Dependent loss. As presented in line 177, Eq. 12 (rather than Eq. 10) defines our $L_\text{SimMat}$ inspired by the Time-step Dependent loss (DCFace[23]), which employs $\gamma_{t}=\frac{t}{\mathbf{T}}$ to make the sample have different similarity property at different time step $t$. We will re-phases lines 173-174 as: we employ the Time-step Dependent loss [23] with different time step t at Eq. 12, specifically firstly an identity loss for recovering the identity of the original inquiry image x, which will be applied to produce original facial embedding when the time step $t\rightarrow 0$.

---

W3: Please refer to the general response G1, where we have rephrased the pipeline and Fig. 3.

---

W4: Ethical and privacy issues are the top priority of our study. While a crucial goal for developing SFR is to eliminate the privacy issue that lies in the face recognition dataset, current SFR approaches are not capable of refraining from privacy risk while providing effective recognition performance. Compared with previous SOTA approaches DCFace[23] and IDiffFace[27] approaches, our method already reduces the privacy risk by avoiding using labelled face recognition images for the model training. These are clearly presented in supplementary material Section C. More importantly, we emphasize that the model development protocol in this paper strictly follows the same protocol of previous peer-reviewed related studies (e.g., the DCFace and IDiffFace) published in top journals and conferences. Thus, we believe our study would not trigger ethical and privacy-related issues.

---

W1: We first define some basic calculation complexities:

Time Step $T$: This represents the total number of time steps required for a complete diffusion process.

UNet Complexity $C_{\text{UNet}}$: The UNet model accepts the input image and outputs the estimated noise.

Pretrained Face Recognition Model $C_{\text{ID}}$ and Similarity Condition $C_{m}$ : These are processed by two sequential linear layers.

The backpropagation and weight updating time complexity for the UNet is given as $B$. Since the pretrained face recognition model is fixed, there is no computational complexity associated with it during backpropagation.

Forward Diffusion Process:

The forward diffusion process consists solely of adding noise to the clean image over $T$ time steps, resulting in a calculation complexity of $O(T)$.

Denoising Process:

The denoising process also consists of $T$ time steps to gradually denoise a noisy image. This process involves a UNet and conditions to output the estimated noise, resulting in a calculation complexity of $O(T \cdot (C _{\text{UNet}} + C _{\text{ID}} + C _{m}))$.

Training Step:

A training step includes a forward diffusion step, a denoising step, and model training (including loss calculation and backpropagation). Note that the model training only involves one time step. The overall complexity of training a CemiFace model by one step is given by:

$O(N \cdot (C_{\text{UNet}} + C_{\text{ID}} + C_{m} + B))$

where $N$ is the number of samples, note the loss calculation involves the similarity comparison between the estimated image $\hat{x_{0}}$ and the original image $x_{0}$, thus the pretrained FR is adopted 3 times in each training iteration. For a standard diffusion model, the complexity is:

$O(N \cdot (C_{\text{UNet}} + B))$

Accumulated Training Iterations:

When accumulating all the training iterations $I$, the calculation complexity becomes: $O(I \cdot N \cdot (C_{\text{UNet}} + C_{\text{ID}} + C_{m} + B))$

Generating One Sample:

To generate one sample, the process involves denoising random noise from time step $T$ to 0. Since the forward diffusion process and loss calculation are no longer needed, the complexity is: $O(T \cdot (C_{\text{UNet}} + C_{\text{ID}} + C_{m}))$

Generating a Dataset:

For generating a dataset with $L$ identities and $S$ samples for each identity, the complete calculation complexity is: $O(L \cdot S \cdot T \cdot (C_{\text{UNet}} + C_{\text{ID}} + C_{m}))$

On a single A100 GPU, (1) generating a sample with 50 time steps takes 0.9 seconds; (2) One CemiFace training step (one inquriy image) takes 0.6 seconds; (3) Training for one epoch takes 45 minutes; (4) It takes 10 epochs to converge with a total time of 7.5 hours. (5) It takes 16 hours to generate a dataset of 10,000 identities, with each having 50 images. We have mentioned the computational cost in supplementary material Sec. A.1, lines 473-480.

---

W2: The overall process not only depends on m but also relies on the input inquiry image. The conditions $C_{att}$ sent to diffusion models are determined by $m$ and the inquiry data using a linear projection $F_{2}$. The selection of inquiry images is dependent on the rule that the inquiry images should be unblurred, non-occluded, appropriately posed and independent of each other (discussed in Sec. 4.2.2, supplementary material B.1 and B.2). It is also required by SFR to ensure high inter-class variation. Then the generation $m$ directly determines the final similarity property of the samples inside each identity group, we find the optimal $m$ is scalar number 0 and mixing the generation $m$ would bring worse performance(lines 232-243).

---

W3: The pseudo-code is given in the PDF, Algorithm 1 and Algorithm 2

---

W4: The revised figure is appended in the PDF Fig. 1, while its description is provided in the general response G1, which will be added to the main text.

---

Q1: In Tab. 6, the results of all competitors except DCFace$\dagger$ are obtained from their original publications. Here, we additionally train a SFR model based on the released synthetic face images/dataset generated by the SOTA DCFace$\dagger$ model and the CosFace loss, to facilitate a fair comparison with ours. This is because DCFace hasn't released AdaFace-based SFR training code and details, and thus we were not able to reproduce it for our SFR model training. For this SFR training, as described in lines 215-229, we use IR-SE-50 as backbone and CosFace loss for learning facial embedding, where the hyperparameters are set the same as the standard CosFace.

---

L1: Although most papers only discussed their limitations in their conclusion section due to page limitation, our submission has detailed limitations not only in the conclusion but also: (1) Sec. 4.2.2 (lines 276-295), Sec. B.1 and B.2 of the supplementary material of the main submission (i.e., high-quality inquiry data is essential); (3) Sec. C of the supplementary material (i.e., privacy issues); and (4) Sec. D.1 of the supplementary material (i.e., the training of our CemiFace depends on the pre-trained FR model, and thus its performance also relies on the performance of this model).

---

W1: The displayed similarities are the input cosine similarities, based on which the displayed face images were generated. However, the actual similarities between the generated images and their inquiry images may not be exactly the same as the input cosine similarities as DL models typically cannot generate perfect/exact outputs. The actual cosine similarities between generated and inquiry images are provided in the PDF Tab. 2, measured by the IR-50 network pretrained using AdaFace loss (line 154).

---

W2: The $\epsilon$ is a random Gaussian noise image $\\epsilon\sim N(0,1)$ fed to our diffusion model (Line 145). We will explain this in the revision.

W3: We assume the mentioned training loss is Eq. 3 rather than Eq. 2. We will modify and cite as: we follow the previous SOTA SFR studies (DCFace[23] and IDiffFace[27]) to choose the same generic diffusion loss [20,21,22], ensuring the reproducibility of our approach and its fair comparison with DCFace and IDiffFace.

W4: The theories of yours and ours are the same but described in different forms. We treat all face images of each subject as an N-1 dimensional sphere with its center representing the subject-level identity center. Then, the spheres of all subjects can be combined in an N-dimensional sphere, where each subject-level sphere is a cluster. We will follow your suggestion to rephrase these sentences

W5: The ranges for each group are listed in the Tab. 1 of the PDF. Here, the splitting boundary for each group varies across different identities.

W6: Please refer to the general response G1.

W7: $F$ should be $F_{i}$ for representing the linear projection operation, and thus $F_1$ and $F_2$ are two stacked linear layers. Here, $F_1$ projects the input similarity m to a latent feature $C_\text{sim}\in \mathbb{R}^{512}$, and then $F_2$ projects the feature concatenating $C_\text{sim}$ and identity embedding to a condition vector $C_\text{att} \in \mathbb{R}^{128} = F_{2}(cat(E_{id},F_{1}(m)))$. Subsequently, the $C_\text{att}$ is fed to our diffusion to control its face image generation via a cross-attention (CA) as:

$

CA(Q,K,V,K_{c},V_{c})=SoftMax(\frac{QW_{q}([K,K_{c}]W_{k})^{T}}{\sqrt{d}})W_{v}[V,V_{c}]

$

where ${\mathbf{C_{att}}}$ is treated as the key $K_{c}$ and value $V_{c}$ (same as DCFace) to influence the latent representation extracted from the input noisy image (treated as the query, key and value $Q=K=V$) for generating the final face image. Please also refer to the general response G1 for details.

W8: $\beta_{t}$ starts from 0.0001 to 0.02 controlled by time step $t$-based linear schedule, while $\alpha_{t}=1-\beta_{t}$ (line 101)

W9: Sorry for the confusion. We put two curves in the same figure due to the limited space. We will separate the two curves into two smaller figures.

W10: As DCFace hasn't released its AdaFace-based SFR training code and details, we were not able to reproduce it for our model training. Thus, lines 216-220 and lines 300-311 fairly compare ours with DCFace by adopting the same pre-trained AdaFace model to train our diffusion generator, and then employing the same CosFace loss for both ours and DCFace's SFR models training. Results show that our CemiFace still outperformed the SOTA DCFace. Based on your suggestion, Tab. 4 in the PDF additionally provides results achieved by using unified CosFace. Specifically, we apply a model pre-trained by CosFace to train both our generator and DCFace generator, and employ the same CosFace loss for their SFR models' training. Due to limited rebuttal time, we only include the results for the 0.5M setting, but will present results for all data volumes in the revision.

W11: We provide ROC curves for CASIA, DCFace and CemiFace in 3 data volumes in the PDF Fig. 2. The FR model trained on our CemiFace generated dataset with the best curve (largest area), while the real face dataset provided inferior results than these SFR methods (CemiFace and DCFace). When FAR=1e-3, the TAR result achieved by our self-implementation on real-dataset is around 90, which exceeds previous works (ArcFace gives around 60 in [a]]). Importantly, the ROC of the IJB-B and IJB-C are typically not provided in neither the related SFR studies [23,24,25,27] nor many of the discriminative FR methods (e.g., CosFace[1], Face Transformer[b], boundaryface[c]) when using CASIA-WebFace for training.

[a]Federated Learning for Face Recognition with Gradient Correction

[b]Face Transformer for Recognition

[c]BoundaryFace: A mining framework with noise label self-correction for Face Recognition

Q1: We rephrase it as: We did not provide the AVG result achieved for setting the similarity interval (Table 2) close to zero (continuous similarities), as this training setting leads our model to generate same images when inputting different $m$ values. We assume that an extremely small similarity interval prevents the model from effectively learning the differences between each level of similarity

Q2: It is a typo and should be [-1,1]

L1: Eq.3 does not directly compare the input and reconstructed images. Instead, it compares the input noise and the estimated noise (Line 98). The overall loss Eq. 13 adopts $\lambda$ to balance between the diffusion focuses on comparing input and estimated noises for generating clean images (Eq.3 $L_{MSE}$) and the similarity between the generated samples and inquiry image (Eq. 12). Given the relationship between the estimated $\hat{x} _{t-1}$ and real $x _{t-1}$ as follows, it is symmetric and valid if the $\epsilon'$ is accurately estimated.

$

\hat{x} _{t-1} \approx \frac{x _t - \sqrt{1 - \alpha _t} \epsilon'}{\sqrt{\alpha _t}} = x _{t-1}

$

L2: Please refer to W1, where $m=0$ produced images of an actual average similarity of 0.2854, making the generated dataset semi-hard. We will update this table in the revised version.