We appreciate your comments and pointing out two new papers. However, we want to emphasize that ID3 and CemiFace were released on arXiv (not officially published) on Sep. 26 and Sep. 27, which was close to the ICLR abstract submission deadline (Sep. 27) and the full paper submission deadline (Oct. 1). Hence, we think it is unfair to either compare our paper with these two or suppose the algorithms in our paper were borrowed from these two papers.

Answers of Weaknesses

Weakness 1.

The proposed model, Vec2Face, which can convert vectors to images while preserving the ID information, is the reason we can generate an unlimited number of identities, not PCA. The reason we use PCA is that we found it can sample better/stronger identity vectors, resulting in accuracy improvement.

Unlike Arc2Face, which fine-tunes a stable-diffusion-v1.5 on WebFace42M, we train our model with only 1M images and we can achieve higher accuracy, showcasing the efficacy of the proposed work.

As for ID3, we added its result because we saw that literature before the full version submission. We will add it to the related work, but it is impossible for us to borrow ideas from it because we trained our model for over 1000 epochs and the HSFace400K has 20M images. Both are non-trivial to be achieved in this short period.

Weakness 2

Yes, we do. The genuine distribution in Figure 4 and intra-class similarity distributions in Figure 5 of the Appendix show that the average intra-class similarity of HSFace10K is better than most of the prior synthetic datasets. Moreover, a pre-trained FR model (M_{FR}) is used in Algorithm 1 and Equation 5 to ensure that the output of AttrOP is identity-preserving.

Weakness 3

We use three different datasets due to privacy concerns reported in [1]. If we use WebFace4M identities to learn PCA, it might be easier for Vec2Face to generate identities that are close to the identities in WebFace4M due to the leakage. In fact, the results below show that using WebFace4M to learn PCA does benefit the accuracy, albeit incrementally. As for using the FR model trained with Glint360K, we want to use the SOTA FR model to generate good image embeddings. It can be any other model or a special design to map vectors to images with a well-defined pattern.

Weakness 4

We retrained the Vec2Face model with CASIA-WebFace, generated a new HSFace10K dataset, and the accuracy is shown below.

We use 1M images because we found that 0.5M images are insufficient for the ViT-based architecture, as the ViT model is known to be data-consuming [2]. Specifically, when using randomly sampled vectors (sampled from N(0, 1), not from PCA), the images are not well-structured. With an underfit model, it can still outperform DCFace, showing the efficacy of the proposed algorithm.

Weakness 5

Thanks for pointing out this paper to us. Their study on choosing the correct similarity range for image generation is interesting and inspiring. This was the headache when we tried to find a good \sigma combination for intra-class vector sampling. However, based on the analysis in DCFace, we don't think they can generate an unlimited number of well-separated identities (80K at most). The analysis of Vec2Face vs. Arc2Face shows that even with identity conditions, it is still hard for the diffusion model to generate well-separated identities. Since Vec2Face allows us to easily scale the dataset, our accuracy of 93.52% with 300K identities remains the SOTA performance.

We believe combining the core idea of Vec2Face and CemiFace can be worth trying. This will be explored in our future work.

Weakness 6

Vec2Face generates images from vectors, so the input is a vector. Identity vectors are randomly sampled from the learned PCA. The pipeline is: 1) randomly generating identity vectors, 2) perturbing sampled identity vectors, and 3) feeding these vectors to Vec2Face to obtain images. For AttrOP, the input vectors are from step 2).

Weakness 7

We will use the FR model trained with WebFace4M to measure the identity separability, but it should not change the observation.

Weakness 8

We will add it to the Appendix.

Weakness 9

Eq. 5 is used in AttrOP (Algorithm 1), which is used at the inference stage. The training loss is Eq. 3.

Answers of Questions

The labeled face datasets are not a must because we only use the features and images, not the labels.

Reference:
[1] This Face Does Not Exist... But It Might Be Yours! Identity Leakage in Generative Models
[2] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Dear Reviewer LGwK,

We hope your work goes smoothly, and thank you for pointing these two papers out again.

We admit that these two works report comparable accuracy (0.13% and 0.3% higher) to ours, but the highest accuracy (93.52%) we achieved in this paper is still state-of-the-art, due to the proposed method allowing us to generate an unlimited number of well-separated synthetic identities, which is not applicable to these two new papers. This is a big breakthrough in this area and can be combined with any other novel methods, e.g., CemiFace, to generate a better synthetic dataset.

Although it is unfair to compare with these two papers, we have put a lot of effort into addressing your concerns and questions.

We believe that the proposed method and the achieved functionality are still valuable to the field and genuinely hope you can reconsider this paper's quality and contributions.

We appreciate your decision!

Answers of Weaknesses

Limited Scale of Synthetic Data

These are the summaries of the number of identities in the prior works. The proposed HSFace400K is already ~4x more than the second largest dataset in this table. Moreover, based on Figure 5, the actual number of well-separated identities in previous works is much less than these reported numbers.

We have sampled 1M well-separated identity vectors. We only use 400K to form a dataset because, as shown in Table 2, using more does not improve accuracy. The proposed algorithm is the first method that allows us to effectively generate this large number of well-separated identities, and having a larger number of identities is recognized as an important factor for FR dataset generation. Thus, this is one of the major contributions rather than a weakness of our work.

Lack of Comparison with Larger Datasets

We do not compare with larger-scale datasets because the highest accuracy we achieved, although higher than prior works, is not higher than CASIA-WebFace. This indicates that many problems still need to be investigated and solved, but generating a large number of well-separated identities is not on the list because this work has solved it. We will try to add WebFace4M and Glint360K results to the paper.

Unclear Practical Motivation

The images in the existing training sets are web-scraped from the Internet, which raises severe privacy concerns. Hence, the regulations, such as GDPR, CCPA, PIPL, BIPA, CUBI, have been established to restrict biometric data usage. Given these concerns and similar to previous works, the goal of this work is to generate high-quality FR training sets with identities that do not exist to address the privacy issue. We will clarify the motivation in the introduction for people who do not know this field well.

Missing Citation

Thanks for pointing this out and we will add it in our related work.

Answers of Questions

Choice of Pre-trained FR Model

Using a pretrained FR model is a common choice in this field, such as SynFace, DCFace, IDiff-Face, Arc2Face, CemiFace, ID3, etc. Different from these works, we use a FR model is because it can provide well-grouped features, not for evaluating the image reconstruction. The proposed model aims to leverage the potential of high-dimensional space to generate well-separated identities and what it does is convert vectors to images. Hence, it is possible to get rid of the FR model from the pipeline as long as the high-dimensional vectors have well-defined patterns and can map to the images. This is the first work achieving the feature-to-image generation, so investigating how to drop the FR model is not in our scope. Overall, FR model is not a must but a convenient option for this first try.

Number of Identities

We have generated 1M well-separated identity vectors, which can be converted to images. However, we stopped at 400K because HSFace400K does not bring more accuracy on test results than HSFace300K. There are two main reasons to explain this: 1) ViT-Base might not be powerful enough to generate high-quality images, and 2) the intra-class identity consistency is not as high as that of the real dataset. Despite these problems, the proposed datasets achieve state-of-the-art accuracy, showcasing the efficacy.

Attribute Control During Inference

There are two main approaches for facial attribute editing: Style transfer and ControlNet. The former cannot achieve explicit attribute control and can lose identity information. The latter needs additional training and external models when a new attribute editing is needed. AttrOP allows us to do plug-and-play attribute operation, with identity preservation and no additional training. In this paper, we control quality and pose by using a pose and quality estimator. This means, any other attributes can be controlled, either separately or combined, in the same way with different attribute models.

The limitation of AttrOP is that when creating an image with extreme attribute conditions, e.g., head pose = 90 degree, it needs more time to search for a proper vector. Fortunately, a dataset does not need a huge number of faces with extreme attributes, so it is not much of an issue.

Dear Reviewer G33d,

Thanks for your effort in reviewing our work.

We believe there is a misunderstanding of the motivation of this work, which led you to think our work is not worth acceptance.

Yes, there are datasets with real web-scraped images available that can be used to train good face recognition models. However, these web-scraped images raise a severe privacy concern. Hence, a lot of works try to generate synthetic identities to form a dataset that can be used to train a face recognition model. Our work has the same motivation.

With this motivation, our work has a big contribution to solving the limitation of well-separated identity generation. Also, the proposed datasets achieve state-of-the-art accuracy.

We sincerely hope you can reconsider the contributions of this work and welcome any further questions.

We genuinely appreciate your valuable comments and hope our feedback answers the questions properly.

Answers of Weaknesses

The purpose of this work

The images in all the existing large training sets are web-scraped from the internet. This causes a severe privacy issue for those identities in the dataset. Hence, similar to the motivations in the previous works, we aim to generate large-scale face recognition (FR) training sets with identities that do not exist to avoid this privacy issue. Moreover, letting FR models obtain higher accuracy from the generated FR dataset than previous works is another goal. We will clarify this in the abstract and introduction sections.

Do the proposed HSFace datasets have the identities in the training set of Vec2Face model?

In the discussion section (line 512-518), we found that there were some training identities at the first stage of identity sampling, but we dropped those training identities when we assembled the datasets. Therefore, all the FR models in this paper were trained using identities that do not exist. Moreover, we want to emphasize that our approach is the first work that can generate this large number of well-separated synthetic identities. It is a big breakthrough in this field, because our algorithm solves the limitation of generating large numbers of well-separated identities found in previous works. This allows the following work to compare with larger versions of real FR training sets.

InceptionV3 trained on ImageNet is not a good candidate for reporting Freche Distance

Thanks for pointing this out and we will add these references in our paper. In addition, besides the FID measurement, our dataset is the best at identity separability, identity consistency, and most importantly FR accuracy on test sets. These metrics are really important in synthetic dataset generation.

Some minor typos

Thank you so much! We will fix that typo and double-check the verbal. The "five test sets" in our paper points to LFW, CFP-FP, CALFW, CPLFW, and AgeDB-30, which are the commonly used test sets in FR model performance evaluation. We will make it clearer in the paper.

Answers of Questions

Question1
Yes, we plan to publish the code, model weights, and datasets. The link will be shown in the paper after the publication.

Question2
It is an interesting research direction! The purpose of using a FR model is that we need the characteristics of the extracted features. Specifically, the image features of the same identity should be clustered together and be away from other identities, so that the Vec2Face model can learn how to map high-dimensional features to their images. With this requirement, using randomly perturbed vectors to train Vec2Face is possible, as long as it is a one-vector-to-one-image mapping. However, it needs extensive experiments to find out how to properly map the perturbations to the intra-class image variations. We love this suggestion and it can be one of our future works.

Question3
For the ID loss, it was reduced rapidly during training, down to 0.01 within 10 epochs. We have tried to use other face recognition models and the trend is the same, so it have marginal effect if use the other FR models for the ID loss. We will add it to the Appendix.

Question4
We only use the FR model trained with Glint360K, and its performance is shown in Table 1 of section A.7 in the appendix.

Question5
We are curious about this too and these are the results.

It shows that adding more synthetic data does not bring more accuracy imporvement. It might be because the the domain gap between generated images and real images. We think a stronger generative model might mitigate it.

Question6:
With a much smaller generative model and a training set that is 1/42 the size of Arc2Face's dataset,, we believe this is not the ceiling of the proposed model. However, our hardware does not support us to do larger scale training. Moreover, one big improvement is that the proposed method allows us to generate unlimited number of well-separated identities, so that we can easily scale the dataset up.Generating more images is trivial but not generating more well-separated identities. Eventually, 93.52% is the highest accuracy our work achieves and it is the state-of-the-art with no doubt.

Dear Reviewer oM4G,

We appreciate your suggestions and hope everything goes well with you. Since we both put our effort into the reviewing and rebuttal, we would like to know if our answers address your concerns well and if it is possible to increase the score.

We are here for any additional questions!

Sincerely, Authors

The goal of synthetic dataset generation in the FR field

We want to clarify the goal of the field that this paper is related to.

Problem - the web-scraped data raises severe privacy issues.

Millions of images are web-scraped online without getting permission from those identities in the dataset, raising privacy concerns in the face recognition field. This issue is related to all the existing large-scale datasets, including the Glint360K and WebFace4M you mentioned.

Solution - use synthetic identities (identities that do not exist)

The most intuitive way is to generate a dataset that does not contain any real identities. This is what our work and related work are trying to achieve. Thus, to answer your question "What is the advantage of this synthetic dataset over Glink360 or a subset of WebFace4M?", our datasets do not contain any of the real identities, so there is no privacy issue.

Experiment for illustration

Step 1: We randomly generate 5M vectors that have a similarity value less than 0.3 compared to each other.
Step 2: We use Vec2Face (trained with CASIA-WebFace, for fair comparison with other methods) to generate images with the AttrOP to ensure the ID of the generated images aligns with the sampled vectors.
Step3: We calculate the cosine similarity between the features of the generated identities and the identity vectors of the real datasets.
Step4: We use different similarity threshold values to measure the identity leakage. 1.27% means that 1.27% of the 5M identities have a similarity value higher than 0.4 when compared to the WebFace4M identities. It is small and easy to drop. When we generate the image for each identity, we add small perturbations so that the identity information is not lost. All these ensure the identities in our dataset do not exist in these real datasets.

Limitations of existing works

1. Cannot achieve the same average accuracy on test sets. However, our work achieves better performance than CASIA-WebFace on several test sets at the same dataset scale, marking the first such achievement in this field. These results are in Table 1 of the main paper and the tables under Reviewer XvoX.
2. Cannot generate a large number of well-separated identities. This limitation is solved by the proposed Vec2Face model.
3. Cannot maintain intra-class consistency while increasing the intra-class variation. This problem is first revealed in our work, so there is still work to follow.
4. The approach highly relies on the pre-trained FR model. This is a common limitation of all works in this field, but it is understandable because the field has just begun addressing this problem.

Your concerns

The main purpose of Arc2Face was personalization and editing

We know, but it has state-of-the-art performance, so we need to compare our result with it.

Is the method effective on its own, or are the results largely due to the priors and additional data?

Here are the FR results for the Vec2Face model trained with CASIA-WebFace, ensuring a fair comparison with other works.

As we mentioned under Reviewer LGwK, 0.5M data makes the model underfit, so the performance is not as good as the results reported in the paper. First, in 0.5M scale (DCFace, ID3, HSFace10K), our model still has state-of-the-art accuracy because ID3 became available on arXiv after the ICLR deadline. Second, our model allows us to scale the dataset up to increase the FR accuracy, which is not possible in any existing papers. With this short period of time, we only scale the dataset to 4M with 80K identities and the accuracy is significantly higher than ID3. Third, since CemiFace has a severe identity leakage issue, we are not comparing with it. Details can be found under Reviewer LGwK.

please revisit my third question for further clarification.

Within this short period of time, we are not able to retrain another model to address this concern, but we will do this experiment and include the results in the Appendix.

Benefit of our work vs. existing datasets and models

1. Dataset: The proposed datasets do not have real identities, which solves the privacy issue.
2. Model: Although the accuracy gap has been reduced in this work, we admit that this gap is still large, and thus there is no advantage from the model performance perspective when comparing with the existing FR systems. However, eliminating this accuracy gap is the final goal of this field and we are still progressing toward it. This work makes a big contribution in the identity generation aspect, accelerating the progress to the final goal.

We sincerely thank you for recognizing our contribution to this field!

Answers of Weaknesses

Using different quality evaluators

Using other FIQA measures in AttrOP is definitely worth trying! We will extend this work to a longer journal and this suggestion will be included in future work.

Precision and recall

These are the values of precision and recall for reconstructing LFW and Hadrian images. These two metrics might be good for evaluating the object generation results (e.g., ImageNet), but face recognition is more class/ID-oriented rather than pixel-oriented. We think evaluating the model by cosine similarity, as shown in Figure 1 of the Appendix, is a better indicator for the FR model training task.

Providing genuine and impostor distribution in Figure 4 of the supplementary material

We appreciate your suggestion! We will update Figure 4.

Evaluating the approach on a more complex and larger-scale set

These are the accuracy results (TPR@FPR=1e-4) on IJBB and IJBC. Surprisingly, we find that the model trained with HSFace10K outperforms the model trained with CASIA-WebFace.

To ensure the correctness of the code, we run the same measurement on a 5M scale. The accuracy value of the WebFace4M is consistent with that reported in the original paper.

Details of patch-based discriminator

We refer to the code at this link to form the discriminator: https://github.com/lucidrains/parti-pytorch/blob/main/parti_pytorch/vit_vqgan.py#L171. It is trained from scratch. We will provide a more detailed description of it in the Appendix.

Answers of Questions

Question 1

We think the biggest novelty of this work is demonstrating feature-to-image generation rather than image-to-image generation for the first time. The comparison between Vec2Face and Arc2Face results shows that feature-to-image generation is more appropriate for synthetic FR dataset generation. Moreover, this allows us to generate an unlimited number of well-separated identities and control the intra-class similarity and variation by controlling the sampled vectors. As for the fMAE, it is a novel design but can be replaced by other advanced architectures. The key differences between MAE, FMAE[1], and ours are that 1) the input is a 2-D feature map, achieved by feature projection and expansion, and 2) we use replace/fill the masked positions by image feature rather than mask tokens, so the model learns image generation purely from the image features. This guarantees the characteristics of the feature, e.g., the ID/class information, can be transferred to the image. We will elaborate on it in the Appendix.

Question 2

First of all, both inter-class separability and intra-class variation are important, because improving both factors increases accuracy when they are not already large. We are not sure if we need to determine the best balance between inter-class separability and intra-class variation, but we observe that the datasets with high separability have lower intra-class similarity than the real dataset, e.g., the mean value of intra-class similarity of DCFace is 0.52, HSFace10K is 0.64, and CASIA-WebFace is 0.76. This suggests that the next research direction should focus on maintaining large intra-class variation and inter-class separability while increasing intra-class similarity. We will add this conclusion to the main paper.