We thank the reviewer for the time and thoughtful feedback to improve our work.

• As suggested, we will merge Figures 1 and 3 in the final version for clarity and as we need to save space to include further experiments.

• Please note we are trying to alleviate the need for large-scale datasets, unlike current approaches that we compare our method does not require any pre-trained FR which on its own requires large-scale datasets (we are the first to do so).

• Due to space constraints, we focused on state-of-the-art methods but will include SFace and SynFace in the background section (2). Notably, DCFace and IDiffFace, which we primarily compare with, outperform SynFace and SFace.

  Additionally, please note that:

  • SynFace: Built on DiscoFaceGAN with auxiliary networks and 3DMM. In contrast, our method mixes in the generator’s condition space with uniquely set parameters (not interpolations, based on Algorithm.1 ). SynFace interpolates in input and label space like the original Mixup paper (Eq.3, Sec. 3.4).

  • SFace: Uses class conditional StyleGAN-ADA but does not mix in condition space as found by our Algorithm. 1, instead of generating images with the same conditions (Sec. II-A) and employing a combined output loss.

We appreciate the reviewer’s insights and will reflect these points in the manuscript. Please note that to the best of our knowledge, we are the first to do conditioning mixing as described in Alg.1.

Finally, we kindly ask the reviewer to check the additional experiments on separate datasets (WebFace10K) included during the rebuttal, further validating our findings. Also, we added an explanation on why we think our method works in the general response.

Dear Reviewer ujs9

In response to your comments, we have addressed the following:

• Outlined what distinguishes our method from the references you mentioned.
• Demonstrated the utility of our approach (beyond architecture level improvements), even when relying on the FR dataset, while showcasing its clear performance boost.

Please let us know if further clarifications are needed.

We sincerely thank the reviewer for their time and valuable feedback.

• Both the generator and discriminator were trained solely on CASIA-WebFace. The numbers (e.g., 335M, 805M) indicate how many times the generator processed the 0.5M CASIA images with different noise scales, with 335M corresponding to ~700 epochs. Diffusion model training involves creating and denoising noisy image versions, and this reporting aligns with standard practices (e.g., EDM, EDM2, RIN). We will clarify this in the manuscript.

• Please refer to the general comment on why we think our method is boosting the performance of the discriminator.

• Regarding comparisons, we focused on state-of-the-art methods due to space constraints but will include SFace and SynFace in the related work section:

  • SynFace: Built on DiscoFaceGAN with auxiliary networks and 3DMM. In contrast, our method mixes in the generator’s condition space with uniquely set parameters (not interpolations, based on Algorithm.1 ). SynFace interpolates in input and label space like the original Mixup paper (Eq.3, Sec. 3.4).

  • SFace: Uses class conditional StyleGAN-ADA but does not mix in condition space as found by our Algorithm. 1, instead of generating images with the same conditions (Sec. II-A) and employing a combined output loss.

• We thank the reviewer for spotting these typos and will try to clarify this in the manuscript.

Finally, we kindly ask the reviewer to check the additional experiments on separate datasets (WebFace10K) in the general response included during the rebuttal, further validating our findings.

Dear Reviewer 1S4a,

We sincerely appreciate your time and effort in reviewing our paper.

In response to your feedback, we have addressed the following:

• Clarified that the same dataset was used and demonstrated consistent improvements by redoing experiments on a new dataset.
• Provided an explanation for why our method works, supported by experiments on Separation of Classes and Compactness (Please refer to general response).
• Highlighted what distinguishes our simple and effective method from the references you suggested.

Please let us know if you have any further questions or need additional clarifications. We would be happy to assist.

Thank you for the thorough response, additional experiments, and a more detailed explanation of the method.

Regarding the illustration of your method, I am not convinced that introducing the additional class M-N necessarily increases the cosine distance between M and N. By this logic, introducing more classes to a margin-loss discriminator would always lead to better separation between existing classes. Is this something you verify in your experiments by comparing e.g. distances between class centroids in the embedding space?

Regarding the results on LFW, CFP-FP, CPLFW, AgeDB, CALFW, and IJB-C, could you help me understand the difference between AugGen, $D^{aug}$ and AugGen, $D^{aug}$ (Ours)? Is the former trained on CASIA, and the latter on WebFace? Why are the gains not materializing for AugGen (Ours) CASIA in the first table?

Overall, I have to concede that the method does lead to improvements in some cases. However, I think the lack of theoretical grounding and a convincing explanation of why (and when) that is the case makes it hard to recommend acceptance. I raised my score to reflect the new experimental insights.

We appreciate the reviewer’s time and effort in reviewing our work.

To address the reviewers' concerns, we have added an explanation and illustration to justify our method, which will be included in the final manuscript. We would like to clarify that our method does not generate new information but produces images that enhance the discriminator's capabilities, aligned with margin losses commonly employed in FR tasks. As requested, we performed an additional experiment on a less class-imbalanced dataset and observed similar improvements. We kindly ask the reviewer to check the general response. We felt it necessary to describe our method from the fundamentals, as practices such as seed changes, while impactful (please look at the variances ~0.1–1%), are not commonly emphasized in the FR community. We hope this explanation will encourage their broader adoption. Additionally, our results demonstrate statistically significant improvements, with up to a 5% performance boost on challenging IJB-B and IJB-C benchmarks. Regarding the references mentioned by the reviewer:

• Hong et al., 2023: This work uses an unconditional GAN and cycle consistency approach, differing fundamentally from our simpler class-conditional generator. Their results on CIFAR (a smaller dataset) contrast with our improvements demonstrated across eight benchmarks and 10,000 classes.
• This Dataset Does Not Exist, 2019: This method optimizes in the z-space with hard sampling, whereas our approach mixes in the condition space (Algorithm 1). This makes our method more practical for diffusion models which optimizing with respect to input noise is challenging.
• Stable Diffusion Dataset Generation, 2024: This approach requires fine-tuning Stable Diffusion 2.0 for datasets like CIFAR and MNIST. Our study avoids relying on pre-trained networks, emphasizing fundamental exploration instead. We sincerely thank the reviewers for their thoughtful feedback and will include these references in the related work section, highlighting their distinctions from our approach.

Answer to Questions:

Indeed we explored various mixtures of synthetic and real images, including different numbers of generated classes and samples per class, as detailed in Table 5 of the paper.

About Ethics:

To train an FR system, datasets with face images across various identities are necessary. We primarily use CASIA-WebFace, a widely adopted dataset for synthetic FR generation in recent works such as DCFace (CVPR 2023) and ID³ (NeurIPS 2024).

We sincerely thank the reviewer for their follow-up and interest in our work.

• Experimentally, we observe that models performing well with AugGen better reproduce $D^{\mathrm{orig}}$ (i.e., $D^{\mathrm{repro}}$), evidenced by lower $FD_{\mathrm{dinov2}}$ and $FD_{\mathrm{discriminator}}$. Here, FD (Frechet Distance) how close the distribution of the reproduced set is to the original data, with subscripts indicating the extractor used. We propose including these observations in the final manuscript, alongside our experiments linking $D^{\mathrm{auggen}}$ to final performance (Please refer to appendix for the current version of this study). Given the complexity of generative models, establishing a theoretical link to the discriminator's capabilities remains challenging and is left for future work. To our knowledge, we are the first to address this problem in this manner. Notably, even MixUp (introduced in 2017) lacks a clear theoretical explanation for its effectiveness to date. For reference, see:

  • https://arxiv.org/pdf/2010.04819
  • https://linkinghub.elsevier.com/retrieve/pii/S0952197623019759

• We apologize for the typo, corrected in the rebuttal table and to be clarified in the final manuscript. AugGen from Dataset refers to augmentations generated solely using WebFace10K or CASIA-WebFace. While IR-101 outperforms IR-50 on CASIA-WebFace on LFW, CFP-FP, ... (first table), our method surpasses CASIA-WebFace IR-50 using the same IR-50 backbone for AugGen training. We argue that enhancing discriminator capabilities via generative models can rival architectural improvements. Model trained on AugGen using IR-50 outperforms even the model trained on the solely original dataset using IR-101 by up to 4% on IJB-B and IJB-C and consistently surpasses IR-50 models trained only on the original dataset across all studied benchmarks like LFW, CFP-FP, and CPLFW. This finding is a valuable contribution to the community.

Verification of Hypothesis:

Here, we validate the hypothesis in two ways:

• Calculating the average inter-class Similarity between mixed classes before and after applying AugGen samples, separation of classes).
• Average intra-class similarity and standard deviation, Compactness of samples within each class.

Separation of Classes

We verified this hypothesis experimentally. We compared two models: $F_{\mathrm{AugGen}}$, trained on the Original Dataset + AugGen, and $F_{\mathrm{Before}}$ (baseline), trained only on the Original Dataset. For each AugGen mix (e.g., M-N mix), we calculated the similarity of the mixed classes (M and N) using both models across 10,000 classes, then averaged the absolute similarities. Our hypothesis is confirmed if $F_{\mathrm{AugGen}}$ produces lower average similarity (corresponding to higher $\theta_{\mathrm{ours}}$, as shown in the figure). The results support this:

[Updated, finished averaging over all the dataset]

Compactness

Like previous section, we used the $F_{\mathrm{AugGen}}$ and $F_{\mathrm{Before}}$ to extract embeddings for Mix classes (like M and N) for all mixed classes ($\sim$ 10K). Here we are reporting the per class average similarity within each class (every two different samples of the same class) and averaged it over all the mix classes. We do the same and reporting std for each class and averaged it over all mix classes:

Here we are demonstrating that the compactness (please refer to how the ovals are more compact in the illustration ) of the embeddings are increased after applying our AugGen samples, further boosting the effectiveness of the margin losses.

This finding validates the core idea driving our work.

We also added this section in the general response as we find your comment insightful for the verification of our hypothesis for other reviewers.

Dear Reviewer y7jg,

We have further verified the question you raised about our explanation: "Introducing more classes to a margin-loss discriminator improves separation between existing classes."

For details, please see the new comment in the general response: Adding More AugGen augmentation makes the model better.

Thank you for your feedback.

We appreciate the time and effort the reviewer put into reviewing our manuscript:

About the references:

• [ref1] Inducing Optimal Attribute Representations for Conditional GANs: This method focuses on attribute editing rather than dataset augmentation for boosting FR performance. It uses networks such as Graph Convolutional Networks and co-occurrence matrices, whereas our approach employs a simple class-conditional generator for augmentation without additional complexities.

• [ref2] STGAN: This method introduces attribute editing (e.g., adding glasses) by using attribute difference codes in its loss function. Unlike our approach, it does not generate augmentation datasets by mixing identities in the generator’s condition space. Our method relies on a pre-trained class-conditional generator without requiring optimization of the generator itself.

Neither of these methods incorporates offline condition-space mixing as we do. Using Algorithm 1, our mixing parameters are calculated in a few forward passes without complex optimization, making it practical even for state-of-the-art diffusion models.

Additional Clarifications:

• We tested both torch’s nn.Embedding layer and one-hot condition vectors. The one-hot vectors are projected into a 768-dimensional condition space by a Magnitude Preserving Linear Layer following EDM2 observations. We will release the dataset and code to facilitate further adoption and we propose to add more details on the conditioning mechanism in the appendix.
• As requested, we conducted experiments with another dataset, a subset of WebFace4M (WebFace10K). Please check the general response, briefly, we trained a generator from scratch, sampled 10K identity mixes, and augmented this with the original dataset, WebFace10K in this case. This experiment confirmed similar performance improvements, as described in the opening section.
• Lastly, as requested, we added an explanation of why we think our methodology works and we will incorporate this into the Methodology section of the paper.

Dear Reviewer 1Ykd,

Thank you for your valuable feedback and for taking the time to review our paper.

As per your suggestions, we have addressed the following :

• Clearly articulated what sets our simple yet effective method apart.
• Provided an explanation for why our method works, supported by experiments (Separation of Classes and Compactness).
• Applied our method to an additional dataset, demonstrating consistent effectiveness.

Please let us know if you have any further questions or require additional clarifications. We are happy to provide further insights as needed.

Adding More AugGen augmentation makes the model better

By the above explanation we expect that adding more AugGen classes enhance model performance. To verify this, we extended the experiments from Table 4 (stopping at 10K new samples) to 30K in increments of 10K. The generator for the new dataset was trained over ~230 epochs. The results are summarized below.

Results on LFW, CFP-FP, CPLFW, AgeDB, CALFW

Here we are observing that adding more mix classes from our AugGen enhances the overall model's performance.

We had the same observations for IJB-B and IJB-C.