AugGen: Synthetic Augmentation using Diffusion Models Can Improve Recognition

We sincerley thank the reviewer for the detailed feedback and suggestions, and also for the time and effort invested in reviewing and improving our manuscript.

Weakness 1: Novelty

We agree that a primary contribution of our work lies in the novel system design and its empirical validation, rather than the invention of entirely new standalone components. Yet we believe that this constitutes a significant addition to the literature. Our work demonstrates that a carefully designed and pragmatic integration of existing techniques can surprisingly outperform more complex approaches. As we show in our response to Q1 (below), our simple approach surpasses a much more intricate, end-to-end guided alternative. We argue that demonstrating that simplicity can be more effective is a valuable contribution, especially for practical applications in a field like face recognition where reliability and reproducibility are paramount.

Question 1/Weakness 2: Why grid search and heuristic?

We initially attempted to optimize $\alpha$ and $\beta$ with respect to our metric $m_{total}$ (which is differentiable via $f_{\theta_{dis}}$), but the unstable gradients and the multi-step nature of diffusion inference made this approach diverge. We explored alternative techniques, including Classifier-Guidance [3R3] (using the discriminator to guide the generator) and GGDPM [3R2] but results were not promising.

Therefore, we adopted a simple grid search, which is simple, stable, and generalizable across generator types (e.g., multi-step autoregressive models), where gradient-based guidance is often not applicable [3R6].

To demonstrate this, we present some parts of our results from experiments with classifier guidance, where we encourage the denoising process to produce samples that lie between certain classes in the discriminator's embedding space. The update rule is: $\tilde S\_\theta = S\_\theta - \lambda\_t \mathbf{g}\_t, \qquad \mathbf{g}\_t = \nabla\_{\mathbf{x}\_t} \mathcal{L}(f\_{\theta\_{dis}}(\mathbf{x}\_t),\bar{\mathbf{e}})$ where $\mathbf{g}\_t$ is the gradient guiding the sample $\mathbf{x}\_t$ towards a target embedding $\bar{\mathbf{e}}$ via $f\_{\theta\_{dis}}$. The rest of the notation follows Section 3. Despite some effort for hyperparameter tuning, we could not exceed the performance of our simpler AugGen approach.

As the table shows, even with classifier guidance added, the performance did not surpass the baseline AugGen.

Optimizing via few-step distilled variants

One natural difficulty for end-to-end optimization is the multi-step inference process. We tested recent methods that distill generation into a few steps (e.g., Adversarial Score Identity Distillation [3R4, 3R5]). While distillation improved the performance of the discriminator trained the on reproduced data ($D^{\mathrm{repro}}$), applying guidance or AugGen to the few-step distilled models resulted in a significant drop in image quality, likely due to mode collapse from the adversarial distillation loss (Please see the answer to the next question too.)

We propose to include these negative results and failure cases in our Appendix to guide future work.

Question 2: Can we use different generator types?

VAEs, Diffusion and Flows

This is a very good question. Theoretically, both VAEs and Diffusion Models train a generator with a maximum likelihood or ELBO objective; for a detailed derivation, please see [3R1]. We chose to use a diffusion model primarily because the methodology is more mature, and there are stable empirical procedures for both training (e.g., SNR-based weighting for high-resolution images) and inference (e.g., faster samplers like DPM-v3).

The same can be said for Flow Matching. More specifically, methods like Gaussian Flow Matching (used in Flux and SD3) can be directly formulated as a diffusion model under a v-prediction parameterization.

GANs

The main difference lies with GANs, whose objective is not formulated as an ELBO or ML. During our experimentation, we attempted to train StyleGANv3 from scratch on our datasets (CASIA-WebFace and WebFace160K), as no publicly available models were trained on these specific FR datasets, and we aimed to avoid any information leakage from external data like FFHQ. However, as it is well known, GANs are very difficult to train, and our training runs were divergent despite using the settings provided by the original authors. Furthermore, a primary concern with GANs is mode collapse. This makes them an unfavorable choice for our goal, which is to explore out-of-distribution generation. This is especially important for long-tailed datasets like CASIA-WebFace, where modes in the tail would likely not be recovered by a GAN-based generator.

We propose to include a discussion of these design choices in the Appendix

Question 3 / Weakness 3: Simplification of text

Thanks for spotting the typo [L126] and for the feedback. We reworked the presentation, especially Section 3, to improve readability.

We will clarify the notation in the paper to distinguish vectors, matrices, and higher-order tensors, consistent with the implementation.

[3R1] Kingma, Diederik, and Ruiqi Gao. "Understanding diffusion objectives as the elbo with simple data augmentation." Advances in Neural Information Processing Systems 36 (2023): 65484-65516.

[3R2] Guo, Yingqing, et al. "Gradient guidance for diffusion models: An optimization perspective." Advances in Neural Information Processing Systems 37 (2024): 90736-90770.

[3R3] Dhariwal, Prafulla, and Alexander Nichol. "Diffusion models beat gans on image synthesis." Advances in neural information processing systems 34 (2021): 8780-8794.

[3R4] Zhou, Mingyuan, et al. "Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation." Forty-first International Conference on Machine Learning. 2024.

[3R5] Zhou, Mingyuan, et al. "Adversarial score identity distillation: Rapidly surpassing the teacher in one step." arXiv preprint arXiv:2410.14919 (2024).

[3R6] Yu, Qihang, et al. "Randomized autoregressive visual generation." arXiv preprint arXiv:2411.00776 (2024)

We sincerely thank the reviewer for their thoughtful feedback and valuable suggestions, which have greatly helped us improve the manuscript.

Weakness 1: Compute cost

We propose to add a discussion to the paper about this important aspect. The proposed method adds a non-trivial one-time training cost, but this is amortized as it yields a model that is both more accurate and more efficient at inference.

We present a cost breakdown below. Augmenting the data lets the smaller IR-50 backbone outperform the much larger IR-101 model (~1.9×FLOPs and ~1.7×parameters) trained on the original data (Table. 1.) Crucially, our final model retains the low inference cost of the IR50 backbone while outperforming the IR101 model, which is vital for real-world deployment where cumulative inference costs quickly surpass the one-time training expense [2R3].

Variances are calculated as the pooled standard deviation from the results reported in Table 1.

This demonstrates a favorable trade-off: we accept a higher, fixed training cost to produce a superior model that is cheaper to deploy. We will release our source code and trained models to further mitigate this cost for the community.

Weakness 2: Usage of generalist models like GPT-4o, FLUX, Gemini for dataset generation

This is an important aspect that we propose to clarify in the introduction section. State-of-the-art generalist models may generate higher-quality images, but they come with significant constraints (see below). This is the key motivation for our development of a self-contained method (independent of external datasets, commercial APIs, or existing models).

• License restrictions. Generalist models like GPT-4o and Gemini have restrictive usage policies [2R1, 2R2] that prevent their use in sensitive or commercial applications like face recognition.

• Unknown training data & consent issues. Many generalist models are trained on private data, where subject consent cannot be guaranteed. This poses a major concern for face recognition systems, medical applications, and other sensitive use cases—an issue our work explicitly avoids.

Weakness 3: Why is synthetic data less sample-efficient than real data?

This is an excellent question. Our approach does not claim to introduce entirely new information; rather, it strategically re-uses information from the original dataset to find hard samples the discriminator may have missed during initial training. We offer two primary reasons why more synthetic samples are beneficial.

1. Recovering Missed Information. The generator's role is to explore the data distribution and produce hard-to-classify samples. Presenting a larger volume of these generated samples naturally increases the probability of exposing the discriminator to these challenging examples, thereby improving its robustness.

2. Generator Imperfection. As shown by the performance of a model trained only on reproduced data ($D^{\mathrm{repro}}$) versus the original data ($D^{\mathrm{orig}}$) (i.e., Table.1) our generator does not perfectly replicate the real data distribution. This slight imperfection means each synthetic sample may carry less informational value than a real one, necessitating a larger quantity of augmented data to inject the same amount of useful information.

Ultimately, our results demonstrate that even state-of-the-art discriminator models (usually used in face recognition), which are already highly optimized, can be significantly improved without adding any new real datasets or pre-trained models.

[2R1] https://openai.com/policies/creating-images-and-videos-in-line-with-our-policies

[2R2] https://ai.google.dev/gemini-api/terms#use-restrictions

[2R3] https://www.breakingviews.com/columns/big-view/ai-boom-is-infrastructure-masquerading-software-2025-07-23/

We thank the reviewer for the thoughtful feedback and following up to our rebuttal. As requested, we present below additional experiments with state-of-the-art generative models.

Open-weight model: Flux-1-Kontext-dev

We tested Flux-1-Kontext-dev, the best available open-weight image-to-image model currently available (see benchmarks at https://lmarena.ai/leaderboard/image-edit).

Only a single input image is supported due to implementation constraints from the diffusers library (https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/flux). To preserve identity context, we concatenate between 1 and 6 images of two random identities (similar to AugGen) and test various prompts. One of the best-performing prompts was the following.

"There are two columns in this image, each corresponds to images of a distinct identity. Generate a morphed image combining cues from both identities to create a new, perceptibly distinct identity."

However, results were generally underwhelming. Even with our best prompts, the resulting images are mostly super-sampled versions of the input images, with very little variation in appearance or identity.

OpenAI gpt-image-1

We also evaluated OpenAI’s gpt-image-1 (currently top performer on image-to-image tasks). We tested it on a small set of images, since cannot upload the whole dataset to a third-party server (we need to ask for permission from original data curators). We report metrics from a strong face recognition system (IR101 trained on WebFace4M):

• RID1 x RID2: average unique similarities between images of two different original identities.
• RID1-RID2 intra: average intra-class similarity within each original identity class.
• SID intra: average intra-class similarity of generated synthetic images.
• SID x all RID1-RID2: average similarity between synthetic and original images.

Key observations:

• SID x all RID1-RID2 > RID1 x RID2. Synthetic samples resemble their sources, acting as hard examples.
• SID intra > 0.3. Effective identity preservation in synthetic samples.

These results suggest that gpt-image-1 has some potential for generating synthetic images with similar effects as our approach. Its downside is the high cost (~USD 10 for 140 images in HQ) which severly limits its use at scale, compared to our approach. This is especially important since we showed in Table 2 (main paper) that a relatively large amount of synthetic data is needed to improve performance compared to real data.

Please let us know if this answers your question.

We sincerely thank the reviewer for the thoughtful feedback and suggestions, and for the time invested in reviewing and improving our manuscript.

We highlight that the paper contains appendices that provide additional support for our claims and address most of the reviewer's questions. We included these additional results in the final version.

Weakness 1

One-hot conditoning. We generate interpolated faces using soft conditioning vectors (i.e. interpolation between one-hot vectors). Formally, let $\mathbf{c}$ be the condition vector and $\mathbf{W}$ be the learned condition embedding matrix. The selection of an embedding is: $\mathbf{c}^T \mathbf{W}$. If we set $\mathbf{c}$ as a soft vector such as $[0.7, 0, 0.7]$, the resulting condition vector is a linear combination: $0.7 \times \mathbf{W}_1 + 0.7 \times \mathbf{W}_3$.

Constraint on condition space. Despite being trained end-to-end using only one-hot condition vectors and without explicit regularization on condition space, the learned condition embeddings exhibit meaningful structure. Our method effectively exploits this structure to generalize to unseen conditioning vectors.

We will include this clarification in the final version.

Weakness 2: Training dataset

Why CASIA-WebFace and WebFace160K? We selected CASIA-WebFace to ensure a fair comparison with prior synthetic face generation methods (e.g., DCFace, VariFace, and Vec2Face), as these methods frequently use it as a training dataset. Our approach, which relies solely on CASIA-WebFace, notably surpasses these previous methods that typically incorporate additional datasets or sources of information. However, due to identified inconsistencies within CASIA-WebFace—detailed in our paper—we introduced WebFace160K, a carefully curated subset of WebFace4M. We employed this new dataset to further validate our method's effectiveness and recommend its adoption by the community.

Will it work with a single image per identity? To effectively capture identity-preserving features, every face recognition training requires multiple images per identity. Our method trains both the generator and discriminator from scratch, making this requirement particularly crucial for learning meaningful identity representations—especially given the margin-based discriminator loss. Similarly, the generator’s condition space benefits from multiple samples per identity. We indirectly evaluated scenarios with few images per identity using CASIA-WebFace, which has a heavy-tailed distribution (Table 4), including identities with as few as two images. However, our approach may be less effective when strictly limited to a single image per identity, due to insufficient identity-specific variation. In fact, any face recognition training, especially those using margin-based losses, would suffer under such a strict constraint, as margin losses fundamentally rely on multiple examples per identity to enforce their key design principles.

Privacy protection. We claim to mitigate privacy concerns primarily because, using a SOTA discriminator and our method, we can reduce the dataset size required to train a SOTA discriminator (with same performance) by up to $40$% (Table 2), thereby mitigating privacy risks associated with collecting large datasets.

Why not MS1M or larger datasets? Please note that large datasets like MS1M have been retracted due to privacy concerns, as they were often collected without consent from the identities [1R1]. Thus, for legal and reproducibility reasons, we refrained from using this dataset, despite its continued online availability. Additionally, while we validated our method on another dataset-a subset of WebFace4M- (for first time in Synthetic Dataset Generation community), training a generator directly on the full-scale dataset was beyond our available computational resources.

Weakness 3: Novel identities or hard samples?

This is an excellent question. Our goal is to generate synthetic samples that are maximally beneficial for training the discriminator. While the synthesized identities have some similarities to their sources, we have demonstrated qualitatively that they represent distinct identities. By design, the generated samples are similar to their source classes. This similarity makes them effective hard samples for the discriminator. The discriminator must not only distinguish between different source classes but also differentiate between real images and our generated samples, which share many cues with their sources yet are distinct.

Our results show that, with a limited amount of data, our approach achieves a performance boost equivalent to adding 1.7x the original dataset's volume (Table 2). This means one would need to add 70% more real data to observe the same performance gain achieved by our method. thereby, as mentioned previously, mitigating privacy risks associated with collecting large datasets.

Weakness 4: Selection process for finding $\alpha$ and $\beta$

To determine the optimal $\alpha$ and $\beta$, we randomly selected 1,000 pairs of identities from the $\binom{N}{2}$ possible combinations [L246], where $N$ is the number of classes in the source dataset (e.g., CASIA-WebFace or our WebFace160K). We tested different random sets of pairs and observed a negligible impact on the optimal values, which remained $(\alpha=0.7, \beta=0.7)$ with a step resolution of 0.1. We added a short ablation study for this process in the final version.

Weakness 5: Notations

Thanks! We fixed the notations to be coherent throughout the paper.

Question 1: Representative face images in most scenarios?

Qualitatively and quantitatively (please refer to the next question), we observe that the generated images are plausible both when the source identities are similar (high cosine similarity) and when they are very different, such as having distinct ethnicities or ages (low cosine similarity). We provide examples for both cases in Figures 12 and 13 of our appendix. For instance, in Figure 13a, the source identities differ significantly in gender, ethnicity, and age. Nevertheless, the AugGen samples (third column) are not only plausible faces but also maintain strong identity consistency down the column, indicating a stable, novel identity has been generated.

Question 2: Similar and different face interpolation

This is a very interesting question. Through extensive exploration of identity selection strategies, we found that mixing identities that are further apart in the discriminator's embedding space yields a greater performance improvement. When we perform mixing (using Alg. 2) on identities that are already similar, the performance gain is lower compared to when we select identities that are highly distinguishable from each other. We have summarized these results in the following table:

We propose to include a more detailed analysis of this finding in the appendix.

[1R1] https://www.vice.com/en/article/microsoft-deleted-a-facial-recognition-database-but-its-not-dead/

Please let us know if you have any additional questions. Thank you for your time and consideration.