HyperFace: Generating Synthetic Face Recognition Datasets by Exploring Face Embedding Hypersphere

We think that three questions raised by the reviewer are potentially answered in our reply to the raised weaknesses:

• Reply to Question #1: Please see our reply to weakness #1 about uniform distribution and face recognition manifold which makes theoretical analysis difficult. We would like to note that in the revised version of the paper, we further explored the complexity of our optimization in Appendix A and proposed a stochastic optimization that reduces the complexity in Appendix B. We provide thorough theoretical and experimental analyses where our experimental results meet our theoretical predictions.
• Reply to Question #2: Please see our reply to weakness #2, #3, and #4. We added a larger dataset with 50k identities in Table 3 and also provided more experiments in the appendix of the revised version of the paper (Appendix C: synthetic datasets at scale, Appendix D: idenitity leakage and recognition performance, Appendix E: additional ablation study).
• Reply to Question #3: Please see our reply to weakness #5.

We hope we could adequately address the reviewer's concerns. In case the reviewer found our reply convincing, we appreciate it if the reviewer can increase their scores and rating. We are happy to continue the discussion if any part remains unclear.

We thank the reviewer for their time in reviewing our paper and for their valuable comments. We are happy that the reviewer found our paper well-written and easy to follow. We are also delighted for the reviewer's positive feedback on the generalization across different validation sets of our method with promising satisfactory improvements on benchmark datasets. Below, we tried our best to address the concerns raised by the reviewer:

• Reply to Weakness #1 (Uniform Distribution): While equation (1) is focused on optimization over the entire hypersphere (n-ball), the face recognition manifold does not necessarily cover the entire hypersphere. For this reason, we proposed a regularization in equation (2) which tries to keep our HyperFace optimization on the manifold of the face recognition using embeddings from embeddings from a gallery of face images. Therefore, we cannot theoretically study the distribution of points on the entire hypersphere, and instead we need to consider the distribution of points on the manifold of the face recognition over hypersphere, which is not trivial and requires accurate estimation of the manifold.

• Reply to Weakness #2 (HyperFace Optmimization): Following the reviewer's suggestion, we added an ablation study in Appendix E and studied the case where random points are used as reference embeddings for generating dataset (i.e., without HyperFace optimization). As the results in our new experiment in Appendix E show, our HyperFace optimization results in a synthetic dataset with better quality which achieves superior performance.

• Reply to Weakness #3 (Experiments and Discussions): We tried to include an extensive study on different parameters in our paper. Following the reviewer's comment we revised our experiments and further discussed our results. In the revised version of the paper, we carefully considered all experiments suggested by reviewers and expanded our analyses. We also explored the complexity of our dataset generation in Appendix A. We proposed a stochastic optimization, which reduces the complexity, and provided in-depth theoretical and experimental analyses in Appendix B. We also provided more experiments in the appendix of the revised version of the paper (Appendix C: synthetic datasets at scale, Appendix D: idenitity leakage and recognition performance, Appendix E: additional ablation study). In case any part remains unclear, we appreciate it if the reviewer can clarify the missing part and the required discussion.

• Reply to Weakness #4 (Additional Experiment): Following the reviewer's suggestion, we added another experiment, in which we increased the number of identities to 50,000 in Table 3. The results indicate improvements in the performance of our model over different benchmarks. Therefore, we cannot conclude saturation on 50,000 identities, and further experiments are required.

While increasing the number of identities requires more computation time, we should note that our new generated dataset with 50,000 identities has already the largest number of synthetic images compared to all synthetic datasets in the literature in Table 1. We should note that we also added a new section in appendix (Appendix C), where we compared our method with largest synthetic datasets in the literature. Our dataset with 50k identity and 3.2M is larger than synthetic datasets compared from the literature and achieves competitive performance with previous large-scale synthetic datasets as reported in Appendix C.

• Reply to Weakness #5 (Identity Leakage): We thank the reviewer for raising this point and suggesting this interesting experiment. Following the reviewer's suggestion, we conducted a new experiment in which we excluded images which have high similarity with real images. Then, we use the cleaned dataset to train a new face recognition model and benchmark the performance of the trained face recognition model. The results show that the new face recognition models achieve comparable recognition performance. We added a new section to Appendix D and discussed it in detail.

We appreciate the reviewer for their engagement in the discussion. We are happy that our reply could address some of the initial concerns. Below, we tried to address the new points mentioned by the reviewer:

• Reply to Point #1 (Uniform Distribution): As mentioned earlier in our Reply to Weakness #1 (Uniform Distribution), our method does not only include optimizing equation (1), but we have an additional regularization term in our optimization in equation (2). The regularization term in equation (2) prevents uniform distribution of points over the hypersphere. In fact, the regularization term tries to keep the solution space for our optimization close to the manifold of face recognition embeddibgs on the hypersphere instead of the entire surface of the hypersphere. Therefore, equation (2) cannot result in a uniform distribution over the entire hypersphere. Our ablation study in Table 6 also shows that our regularization in equation (2) improves the recognition performance of the trained face recognition model.

• Reply to Point #2 (largest dataset): Our dataset with 50k identities has 3.2M images, which is the largest dataset in terms of number of images. As mentioned by the reviewer, DigiFace has 110k identities but with 1.22M images. While we acknowledge that DigiFace has more identities, in our Reply to Weakness #4 (Additional Experiment) we stated that our new generated dataset with 50,000 identities has already the largest number of synthetic images (i.e., 3.2M). We should note that our datasets (with 50k identities and even 10k identities) far outperform DigiFace on all benchmarks. In Appendix C, we also provided a benchmark with the largest available version of all datasets, where our dataset achieves competitive performance with state-of-the-art in the literature. Please note that in the paper, we have not mentioned that our dataset is the largest dataset in the literature.

• Reply to Point #3 (Maximum Number of Unique Identities): We would like to note that Figure 3 of reference [3] (i.e., DCFace paper) mentioned by the reviewer is on the capacity of face generator models (DiscoFaceGAN and unconditional DDPM). In fact this figure does not provide an answer to the maximum number of unique identities that can be generated by DCFace. In particular, for Figure 3 in the DCFace paper, Kim et al. [3] generated 10,000 images with two face generator models (DiscoFaceGAN and DDPM) and compared all the generated images. By changing the threshold of the face recognition model, they plotted Figure 3 in [3]. Therefore, even in the DCFace paper (CVPR 2023) the question of the maximum number of unique identities has not been addressed. While in Figure 3, a maximum of 10k identities are counted, Kim et al. [3] also published a version of the dataset with 60k identities. Therefore, the maximum number of identities that can be generated by DCFace is not evident in Figure 3 of the DCFace paper [3]. As a matter of fact, the maximum number of unique identities is more studeied for the capacity of face generator models, such as [A]. However, we believe for answering to this question for synthetic datasets, we need to increase the number of identities in generated datasets and train face recognition with larger datasets. As reported in the paper, by increasing the number of identities, we still could not observe saturation in performance, and therefore we cannot ensure the maximum number of unique identities in our method. However, generating a larger dataset requires more computation and time, while our method with 50k identity already achieves competitive performance with state of the art.

[A] Boddeti, et al. "On the biometric capacity of generative face models", IEEE International Joint Conference on Biometrics (IJCB), 2023.

We would be glad to continue the discussion if any part remains unclear.

• Reply to Weakness #3 [Major] (Optimization): Following the reviewer's suggestion, we conducted a new experiment and used other optimization techniques. We added the results to in Appendix E of the revised version of the paper. As the results in our new ablation study show, solving HyperFace optimization with different optimizers leads to comparable performance.

• Reply to Weakness #4 (Number of Images): While by default, we generated 64 samples per identity, we reported an ablation study for different numbers of samples per identity, including 50 samples. Comparing the results reported in the ablation study with Table 1, we can see that still with 50 samples per identity, our method achieves comparable performance. Considering the reviewer's concern, we updated the results in Table 1 with 50 samples per identity (500k images). However, still with 500k images, the ranking of synthetic datasets over different benchmarks remains unchanged.

• Reply to Weakness #5 (Visualization for Intra-class Variations): In Figure 1 of the paper and also Figure 4 of appendix, we illustrated sample face images from our dataset, including 8 samples for each subject to illustrate the intra-class variation. Following the reviewer's comment, in the revised version of the paper, we added 64 images of two synthetic identities in Appendix G to illustrate intra-class variations in our dataset.

• Reply to Weakness #6 [Minor] (Caption of Fig 2): Figure 2 provides a general overview of our method. Following the reviewer's suggestion, we extended the caption and provided more details of our data generation in the caption of this figure.

• Reply to Weakness #7 (Pseudo-code for the entire process): Following the reviewer's suggestion, we added a pseudo-code for the entire data generation process in Appendix F of the revised version of the paper.

We hope we could adequately address the reviewer's concerns. In case the reviewer found our reply convincing, we appreciate it if the reviewer can increase their scores and rating. We are happy to continue the discussion if any part remains unclear.

We thank the reviewer for their time in reviewing our paper and for their valuable comments. We are happy that the reviewer found our paper "good in writing and has a solid mathematical formulation". Below, we tried our best to address the concerns raised by the reviewer:

• Reply to Weakness #1 [Major] (motivation): In general, in each face recognition dataset (either synthetic or real), the variations in images are very important: especially, it is necessary to have a diversity in the identity (inter-class variation) and also variations for images of each subject (intra-class variation). As a matter of fact, having sufficient variations in a dataset has a direct impact on the generalization capability of the face recognition model (trained with that dataset) and its performance on different benchmarks. So, it can be useful to see how we can represent a face recognition dataset in a loower dimension (compared to image dimensions) and then see how we can use such representation to improve synthetic dataset generation. Previous work, e.g., [1,2], showed that we can study face images on embedding space of a face recognition model. The normalized embedding space of a pretrained face recognition model can shape a hypersphere (i.e., n-ball), and we can represent the face dataset on the hypersphere by using the embeddings of images. In order to have a larger variation in a dataset, we would like the face embeddings to cover the most areas on the hypersphere. Therefore, our question is how to distribute identities (using their embeddings) on the face hypersphere. This leads to a packing problem to find an optimum representation for a given number of identities. While representing face images on the hypersphere of a pretrained face recognition model has been used and studied in the literature [1,2], to our knowledge our paper is the first work that formulates the identity sampling for synthetic dataset generation as a packing problem. We would like to stress that intra-class variation can be improved by using conditional face generator models (for pose, light conditions, etc.) when generating the synthetic dataset. However, still the major problem is how to increase inter-class variation, which is the focus of our paper and we tried to see how we can improve inter-class variation assuming we can sample identites from face embedding hypersphere.

  • [1] Terhörst, et al. "On the (limited) generalization of masterface attacks and its relation to the capacity of face representations", IEEE International Joint Conference on Biometrics (IJCB), 2022.
  • [2] Boddeti, et al. "On the biometric capacity of generative face models", IEEE International Joint Conference on Biometrics (IJCB), 2023.

• Reply to Weakness #2 [Major] - Point 2 (Comparison with Arc2Face - Table 1): For a fair comparison in our experiments for Table 1, we fixed all hyperparameters and trained face recognition models for all datasets with the same configuration (backbone, loss function, batch size, number of epochs, etc.). Therefore, we could only compare with methods that have publicly available datasets. Whereas the authors of Arc2Face have not published their dataset, and we could not reproduce the results reported in their paper. There are also open issues on their GitHub repository, where several researchers reported that they could not reproduce their results for face recognition. Since our training setup is different, we believe the numbers reported in our experiments are not comparable with the values reported in the Arc2Face paper. The configuration for training face recognition in Arc2Face paper is neither available, which makes it more difficult to make a fair comparison.

• Reply to Weakness #2 [Major] - Point 1 (Comparison with Arc2Face - Ablation study): Following the reviewer's suggestion, we conducted a new experiment and used random embeddings (without HyperFace optimization) to generate a synthetic dataset. We also ensure that the similarity of each pair is below a certain threshold (0.3). As the results in Appendix E show, our HyperFace optimization results in a synthetic dataset with better quality which achieves superior performance.

• Reply to Question #5: Following the reviewer's feedback, we provided an evaluation of the complexity of our method in appendix A of the revised version of the paper. We also provided further theoretical and experimental analyses for reducing complexity in our optimization in Appendix B. We propose to solve the optimization with mini-batches which significantly reduces the complexity of our optimization and allows scaling dataset generation process. In our initial submission we discussed the complexity of our optimization as its limitation for scaling and mentioned how this can be improved, and following the reviewer's suggestion we provided in-depth theoretical and experimental analyses in Appendix B of the revised version of the paper.

• Reply to Question #6: For a fair comparison in our experiments for Table 1, we fixed all hyperparameters and trained face recognition models for all datasets with the same configuration (backbone, loss function, batch size, number of epochs, etc.). Therefore, we could only compare with methods that have publicly available datasets. Whereas the authors of Arc2Face have not published their synthetic dataset, and we could not reproduce the results reported in their paper. There are also open issues on their GitHub repository, where several researchers reported that they could not reproduce their results for face recognition. The configuration for training face recognition in Arc2Face paper is neither available, which makes it more difficult to make a fair comparison. However, considering the reviewer's concern, we conducted a new experiment, and as suggested by Reviewer 3 (YfZL), we used random embeddings (without HyperFace optimization) to generate a synthetic dataset. As the results in our new experiment in Appendix E show, our HyperFace optimization results in a synthetic dataset with better quality which achieves superior performance.

We hope we could adequately address the reviewer's concerns. In case the reviewer found our reply convincing, we appreciate it if the reviewer can increase their scores and rating. We are happy to continue the discussion if any part remains unclear.

We thank the reviewer for their time in reviewing our paper and for their insightful comments. We are happy that the reviewer found our paper well written, easy to read and understand, with interesting idea. Below, we tried our best to address the remaining concerns raised by the reviewer:

• Reply to Weaknesses: In the revised version of the paper, we added results for a larger version of our dataset with 50k identities. Moreover, we provide an in-depth discussion about the computation requirement in Appendix A and further improvements in Appendix B of the revised version of the paper. In our initial submission, we discussed the complexity of our optimization as its limitation and mentioned how this can be improved. In the revised version of the paper, we further explain our approach to reduce the computation required and propose stochastic optimization for HyperFace. We theoretically prove that stochastic optimization leads to similar results with less computation. In addition, we report numerical results for stochastic optimization which meet our theoretical analyses.

• Reply to Question #1: Reference embeddings and gallery embeddings can be independently initialized. They can also have different numbers of images. For example, Table 4 reports the recognition performance achieved for the face recognition model trained with datasets with 10k identities and optimized with different numbers of gallery images. The reference and gallery images can be generated by different generator models, such as StyleGAN or a diffusion model (e.g. LDM). We also reported an ablation study on the source of images in Table 5 of the paper.

• Reply to Question #2: Yes, for a fair comparison we fixed all training configurations (backbone, loss function, batch size, numbers of epochs, etc.) and trained new face recognition models for all datasets.

• Reply to Question #3: Following the reviewer's comment, we updated our model in Table 1 and used a dataset with 500k images. With 500k images, the ranking of synthetic datasets over different benchmarks remains unchanged. It is noteworthy that we also have an ablation study on the number of images in Table 2 of the paper.

• Reply to Question #4: Following the reviewer's suggestion, we expand Table 1 with more baselines with larger datasets in Appendix C. As the results for comparing previous datasets at scale show, our synthetic dataset achieves competitive performance with the largest synthetic datasets in the literature. The larger version of DCFace does not achieve the best performance on any benchmark. Langevin-DisCo achieves significant improvement with 30k identities, however, the authors of Langevin-DisCo have reported lower performance for 50k identities, indicating limitations in further scaling in Langevin-DisCo. In Table 3 of the revised version of the paper, we scaled our dataset for 50k identities. The results show improvement in our performance on the majority of benchmarks (without saturation), which shows our dataset can be further scaled. Our dataset with 50k identity and 3.2M is larger than synthetic datasets compared from the literature and achieves competitive performance with previous large-scale synthetic datasets as reported in Appendix C.

Dear Reviewer UATk,

We thank you for your review and valuable comments, which helped us improve the quality of our paper. We carefully considered all the points and tried our best to address the concerns raised by the reviewer in our reply and the revised version of the paper. While we still have not received any feedback on our reply from the respected reviewer, we are gladly open to further discussions.

• Reply to Question #1: We provide an in-depth discussion about the computation requirement in Appendix A and further improvements in Appendix B of the revised version of the paper. In our initial submission, we discussed the complexity of our optimization as its limitation and mentioned how this can be improved. In the revised version of the paper, we further explain our approach to reduce the computation required and propose stochastic optimization for HyperFace. We theoretically prove that stochastic optimization leads to similar results with less computation. In addition, we report numerical results for stochastic optimization which meet our theoretical analyses.

• Reply to Question #2: We thank the reviewer for raising this interesting question. To our knowledge, this experiment has not been explored in previous work on synthetic data for face recognition, and in fact requires more efforts to retrain the face generator model. We will consider this experiment in our future work.

We hope we could adequately address the reviewer's concerns. In case reviewer found our reply convincing, we appreciate it if the reviewer can increase their scores and rating. We are happy to continue the discussion if any part remains unclear.

We thank the reviewer for their time in reviewing our paper and for their insightful comments. We are happy that the reviewer found our paper with a well-formulated optimization problem, efficient solution, extensive experiments, and ethical considerations. Below, we tried our best to address the remaining concerns raised by the reviewer:

• Reply to Weakness #1 (Limited scale): Following the reviewer's suggestion, we added a new experiment and further increased the size of the dataset to 50k identities in Table 3. We should note that our dataset with 50k identities and 64 samples per identity (3.2M images) is larger than publicly available synthetic datasets compared from the literature. The results in this table demonstrate that we can still increase the number of identities and scale our dataset generation without saturating the performance:

We also provided a comparison with synthetic datasets at scale in Appendix C of the revised version of the paper. In addition, we provided an in-depth discussion of the computation requirement in Appendix A and further improvements in Appendix B of the revised version of the paper. In particular, we propose stochastic optimization for our approach and theoretically prove that the stochastic optimization leads to similar results. In addition, we report numerical results which meet our theoretical analyses.

• Reply to Weakness #2 (Inter-class variation): We acknowledge the reviewer's comment that the main focus of our work is on increasing inter-class variation in the synthetic datasets. However, we would like to stress that increasing inter-class variation is less explored in the literature, whereas there are extensive studies in the literature on increasing intra-class variation in face image generation. For example, training conditional generator models, or using ControlNet can easily help to improve intra-class variation in the image generation process. Meanwhile, we still used a simple yet effective approach to increase intra-class variation for our synthetic dataset, where we proposed to add a Gaussian noise controlled by hyperparameter $\beta$ to the embedding of each synthetic identity in the image generation process. The added noise to the embeddings simulates the change in embedding caused by image variation (such as lightning conditions, etc.). We also provided an ablation study on the effect of hyperparameter $\beta$ on the performance of the generated dataset in Table 7 of the paper.

Following a concern raised by Reviewer RHfp, in a new revision, we revised Solving the HyperFace Optimization in Section 2.2, and further explained the previous studies on the Tammes problem. As mentioned earlier, the equation (1) of our paper represents the Tammes problem. The Tammes problem is, however, an open problem and the optimal solutions for this problem are studied for small dimensions and a small number of points. Nevertheless, for large dimensions and a high number of points there is no closed-form solution. There are different approaches for solving this problem (such as geometric optimization, numerical optimization, etc.) for large dimensions and a high number of points. However, for a large dimension (e.g., 512) and a very large number of points (e.g., 10k identities) solving this problem with geometric optimization or numerical optimization is computationally very expensive. Hence, we solve this problem with a gradient descent approach, which allows us to solve the optimization with a reasonable computation resource. It is noteworthy that in Appendix A, we report the computation required to solve our optimization with our method and further reduce the computation with a stochastic optimization in Appendix B, where we demonstrated theoretically and empirically that stochastic optimization reduces the complexity while resulting in a comparable performance.