Salient Concept-Aware Generative Data Augmentation

Question: In the introduction, the paper refers to synthetic image generation as a form of distillation. Based on this, can generative data augmentation be understood as a process where knowledge from the salient concept-aware embedding model is distilled into the classifier via the synthesis model? From this perspective, I am curious whether using the features from the salient concept-aware embedding model directly for classification would yield results comparable to those obtained through generative data augmentation.

Answer: Our work is motivated by practical challenges that researchers often need to train expert classifiers on domain-specific datasets that are typically limited in quantity, class-imbalanced, or require adaptation to new classes with very few examples. To address this problem, we use foundation synthesis model with world knowledge to generate images with diverse incidental features to improve the robustness of the downstream classifier that can focus on the salient concept. We can think of it as a form of indirect knowledge distillation.

However, existing text-to-image or GDA approaches often fail to generate faithful images due to either the limitations of textual descriptions or inability to incorporate task priors. The role of SCA is to introduce the task priors to the synthesis model, achieving optimal fidelity-diversity trade-off. In this way, the foundation synthesis model serves as a task-aware source of world knowledge that the downstream classifier can effectively learn from.

It’s also important to note that the proposed SCA training mechanism is not a silver bullet for learning to isolate salient concepts with exceptional precision. It is an effective mitigation strategy to reduce conflicts between image and text prompts during synthesis, eventually helping achieve a better fidelity-diversity trade-off in the generated data.

Therefore, we suspect that the performance of features from SCA models alone cannot be comparable to that of our augmentation method. To validate this, we evaluate the performance of the SCA embedding model by probing its frozen features with classification head. For a fair comparison, we used the OpenAI CLIP‑B/16 backbone for SCA probing (borrowed from the experiment requested by Reviewer MqZv to compare DINO with CLIP), since our downstream classifiers are also trained from OpenAI CLIP‑B/16.

Our augmentation method out-performs the SCA embedding by an average of 3.31% over 8 datasets.

Question: In the comparisons for conventional, long-tail, and OOD classification, is the amount of synthetic data generated by each method kept consistent? If not, I would appreciate it if the authors could report the amount of synthetic data produced by each method.

Answer: For CutMix and Mixup, we apply it to every training sample in the mini-batch. For GDA methods, we generated synthetic data with their public code and use them to train our downstream classifiers. The classifier training protocol outlined in section 4.1 is the same for all methods, except for the synthetic-to-real ratio K and the probability of sampling synthetic images P during training. We follow the settings from original papers for the choices of K and P described below:

• For our method, we use P=20% for ImageNet and iNaturalist, and P=50% for all other FGVC dataset. In figure 4 of the manuscript, we show that our method is robust for P in the range from 20% to 50%. We set K=5 all datasets.
• For Real-Guidance, we use P=50% and K=5 for all datasets.
• For Da-Fusion, we use P=50% and K=10 for all datasets.
• For Diff-Mix, we use P=10% and K=5 for all datasets.

In addition, we guarantee that at least 250 synthetic data are generated for each category, applicable to all methods. This is to make sure that enough data are generated for few-shot experiments and under-represented tail categories in LT experiments.

Question: The proposed approach still requires finetuning both the salient concept-aware embedding model and the synthesis model for each new dataset, which limits its generalizability and falls short of achieving truly optimization-free general data augmentation.

Answer: We acknowledge that the proposed approach requires fine-tuning both the salient concept-aware embedding model and the synthesis model for each new dataset. However, it is important for our method to remain functional for samples from previously unseen or underrepresented classes within the same domain as the training data. For example, if the training dataset contains various dog breeds but excludes bulldogs, the model should still recognize a bulldog as a distinct and relevant class and accurately preserve its concept during the synthesis process. This is partially demonstrated in our LT and OOD experiments in Table 4 of the manuscript, and we will dive deeper in this rebuttal.

We consider an “open-set” setup where our embedding and synthesis models were trained using only 80% of randomly selected species from the iNat training set, and the remaining 20% of species were held out and only used for evaluation. We compare our method with two peer personalized image generation methods as recommended by Reviewer MqZv.

The Top-5 accuracy of the external timm leaderboard classifier on the generated images (Gen. Top5) drops moderately from 87.48% (in Table 5 of the manuscript) to 84.19%, primarily due to the challenge of generating samples for unseen classes. However, despite this constraint, our approach still outperforms all peer methods across concept preservation and text-image alignment metrics, demonstrating its superior ability to extrapolate fine-grained visual concepts beyond the training distribution. This aligns with findings from our long-tail and OOD experiments in Section 4.2, where our model demonstrated strong performance for underrepresented/unseen categories.

[1] IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models, Ye et al.

[2] ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation, Wei et al.

Question: However, the method is expensive to train; IIUIC, it requires fine-tuning of the SCA and the diffusion model for each classification task.

Answer: We acknowledge that our approach is more computationally intensive than classical augmentation or existing generative data augmentation (GDA) methods, as it involves training a SCA embedding model and fine-tuning a diffusion model. As shown in the table below, on an 8×A100 GPU setup, training takes approximately 20–30 minutes for smaller fine-grained datasets (e.g., ~10K images), and up to 3 days for large-scale datasets such as iNaturalist 2021 (2.7M images). We argue that this cost is practical for most application use-cases, especially for scenarios where curating high-quality in-domain data is prohibitively expensive. Our method offers a promising solution, giving consistent performance improvements over baseline GDA methods.

Question: Comparison to other off-the-shelf adapters is required.

Question: I wonder how the method improves over augmentation that utilizes other pre-trained diffusion adapters, e.g.: [1] IP-Adapter [2] BLIP-Diffusion [3] ELITE

Answer: We compare our method with IP-Adapter [1], ELITE [3], E4T [4], , FastComposer [5].

In line with Table 5 of the manuscript, we conduct evaluation using the iNaturalist dataset. For each method, we generated synthetic data and measured concept preservation using DreamSim, diversity with Vendi, and text–image alignment using CLIP-T. Additionally, we evaluate Top-1 and Top-5 accuracy on the synthetic data using an external classifier from the TIMM leaderboard (denoted as Gen. Top-1 and Gen. Top-5). Our method consistently outperforms other off-the-shelf adapters across all metrics, with 41.1% and 2.56 improvements on Gen. Top-1 accuracy and CLIPT scores, respectively.

[1] Ye et al., “IP-Adapter: Text-Compatible Image Prompt Adapter for Text-to-Image Diffusion Models,” arXiv:2308.06721, 2023.

[3] Wei et al., “ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation,” ICCV 2023.

[4] Gal et al., “Encoder-Based Domain Tuning for Fast Personalization of Text-to-Image Models,” TOG 2023.

[5] Xiao et al., “FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention,” arXiv:2305.10431, 2023.

Question: Is the Domain Expert Embedding Model trained together with the diffusion model?

Answer: The domain expert embedding model is trained separately and kept frozen during the diffusion model fine-tuning.

Question: Is it trained separately for each task?

Answer: Yes, the SCA embedding model as well as the diffusion model are trained separately for each task.

Question: L. 128-131: Why CLIP Image features are not suitable for the task? Unlike DINO, CLIP is trained under contrastive settings between images and text such that it captures the important concepts in the image.

Answer: Our experimental findings show that DINOv2 consistently outperforms CLIP by a significant margin across all eight datasets. We include the iNaturalist results alongside the off-the-shelf adapter results in the table above. Here we fine-tune the synthesis model to be compatible with each image embedding model, following the same procedure used with SCA. By switching from DINOv2 to Clip, the CLIPT score is dropped by 0.0082 and the Gen. Top-1 score is dropped by 21.92%. We suspect the reason is that CLIP is optimized for global semantic consistency with high-level language descriptions, making it less effective for fine-grained visual tasks such as image generation that require understanding of low-level details. In contrast, DINOv2 is trained with a self-supervised objective targeted for vision task only. This observation is also supported by recent work [6], where they show in Table 2 that aligning diffusion model representations with DINO embeddings leads to superior performance in generative tasks, compared to CLIP embeddings.

[6] Zhang et al., “Representation Alignment for Generation: Training Diffusion Transformers is Easier Than You Think,” ICLR 2025.

Question: Additionally, I found the scope of the method to be limited: could the proposed data augmentation help to improve generative personalization tasks (e.g., 1, 2), or other discriminative tasks?

Answer: [Generative Personalization Tasks] Our method could potentially improve personalized image generation methods that rely on image encoders to extract image prompt information (e.g., [1], [3–5]). Prior work has mainly focused on redesigning multimodal attention in diffusion models, and our approach is orthogonal to them as we improve the image encoder to reduce conflicts between irrelevant visual features and text prompts, resulting in images that more faithfully align with the intended concepts.

Due to the new rebuttal policy, we cannot include qualitative comparisons. Informally, we observe that our method achieves better text‑prompt alignment than IP‑Adapter and greater visual quality than ELITE. For example, given the prompt "Brilliant purple flower, aged brick wall, gentle shadows" and an input image of a violet on green grass, ELITE outputs a pink-purple blend which does not look like flowers. IP-Adapter outputs a counterfactual image of a violet on a stone road with surrounding green grass. Our method faithfully generates a violet near a red brick wall under sunlight.

The primary focus of this paper is on discriminative tasks, motivated by practical challenges in niche domains where expert models are only trained on small or class‑imbalanced domain-specific datasets, a regime where our method shines. Extending our approach to generative personalization is an exciting direction for future work.

[Discriminative Vision Tasks] Our current synthesis pipeline is built on a diffusion model (SDXL), which is not well-suited for complex vision tasks such as text-rich VQA or GraphQA. There are other interesting experiments for our method to demonstrate: a) broader applicability to diverse domains, b) application to multi-modal models.

a) broader applicability to diverse domains: We perform a few-shot experiment on Places365, a dataset for abstract scene understanding rather than the object-centric datasets we primarily focused on. The evaluation is done on its validation set. We follow the same the N-way 10-shot experiment setting in Table 2 of the manuscript. The result is shown below:

We report classification accuracy (%) using 10-shot training samples on Places 365 validation set. Our method out-performs peer method by 1.53% and 1.78% in Top-1 and 5 accuracy, respectively, demonstrating that our method is also effective on datasets of abstract scene as well.

b) application to multi-modal models: Classification is still an important task for modern multi-modal models that often require fine-tuning domain-specific dataset to achieve production-level performance. To demonstrate the effectiveness of our generation method in this context, we explore multi-modal visual classification by performing supervised fine-tuning on an MLLM using FGVC datasets. In this setup, we tune only the projector and vision encoder and keep the LLM frozen. The classification problem is reformulated as a 26-choice QA task, where the ground-truth label is always included among the choices. An prompt example is provided below: You are an expert classifier. Based on the visual evidence in the image, select the most appropriate category. Please choose ONE of the following: A. beef_tartare B. creme_brulee ... Z. tacos Please respond ONLY with your answer wrapped in this format: <answer> LETTER </answer>.

We report classification accuracy (%) under zero-shot and N-way 10-shot setting with Qwen2.5-VL-7B backbone. The zero-shot performance is not satisfactory on majority of the FGVC datasets. When fine-tuning the model with augmented data, our method out-performs Diff-mix by 3.98% on average across 6 FGVC datasets.

Question: The core of this paper is the loss term in the Salient Concept-Aware Embedding Model (Eq. 1). It looks like a tuned version of a contrastive loss (based on cosine similarity with margins) in a contrastive learning framework. It is unclear to me where the 'salient' part comes from in this formulation: I think it is closer to a max-margin positive feature vs negative feature learning scheme.

Answer: Salient concept learning is from the combination of the proposed loss and the structure of the training data. For example, consider two images of the same bird species, one standing on a tree, the other on a rock. Cross-entropy loss is happy as long as the prediction is correct, even if the model encodes incidental features (e.g., trees or rocks). In contrast, our margin-based loss encourages intra-class compactness in the embedding space, penalizing incidental variations that introduce noise.

We would like to note that SCA is not a silver bullet for isolating salient concepts. It is an effective mitigation strategy to reduce conflicts between image and text prompts during synthesis, eventually helping achieve a better fidelity-diversity trade-off in the generated data.

We agree with the reviewer that our formulation can be viewed as a form of max-margin learning. But we want to highlight the role of the angular loss: image embeddings are first normalized before being fed into the adapter and cross-attention layers of the diffusion model. This normalization stabilizes gradient flow during fine-tuning and better aligns the image embeddings with the text embeddings. Our angular margin–based loss is specifically designed to optimize these normalized embeddings (see Lines 144–145 of the manuscript). In contrast, embeddings trained with cross-entropy typically require post-hoc normalization and gives inferior performance in our experiments (see Table 6 of the manuscript).

Question: The 0.73% accuracy improvement in the conventional setting is underwhelming. One would expect that having a good data augmentation to obtain more diverse data should improve the results more, even in the traditional setting without the focus on the few-shot or long-tail cases.

Answer: We agree with the reviewer that a 0.73% improvement over peer methods in the conventional setting is modest in absolute terms. Another perspective is to compare relative gains: the current SOTA GDA method improves performance over the vanilla baseline by 0.855%, and our method achieves a 1.585% improvement. This is an 85% relative performance gain over the SOTA method, which is not marginal.

It is also true that the conventional setting is more challenging to improve: with fixed task complexity, as datasets become more balanced and diversified, there is less room for robustness improvement through augmentation (e.g., a model pretrained on web-scale data may show little to no further gain on the easy task distinguishing cats and dogs). Our work is motivated by practical challenges in niche domains, where expert models must be trained on domain-specific datasets that are often small, class-imbalanced, or updated incrementally (e.g., with new categories over time). In these settings, our method is a good choice for practitioners as it consistently outperforms baseline GDA methods across conventional, few-shot, long-tail, and out-of-distribution classification tasks.

Question: Figure 2: for image (c), it seems the proposed model SCA learns some sort of style transfer—for the given image (f), it treats the black mask as the texture of the fly and draws a black fly/moth in image (c). This is slightly different from what is expected if the learned embeddings are truly about either the fly or the background flowers. Can the authors clarify what's happening here when a salient object is masked out?

Answer: This is a very interesting observation that we didn’t pay attention to before. We suspect that the SCA model is influenced by learned priors from the training data, driving it to interpret the black masked region at the center into one of the species from the iNaturalist dataset. To validate this we computed the SCA embedding of the masked image (Fig. 2(f)) and compared it with the average class embeddings across all species in iNat dataset. The top nearest matches were:

1. Atolmis rubricollis (Insects), cosine similarity: 0.31
2. Lucidota atra (Insects), cosine similarity: 0.19

Both of them are black-colored insect species. Given this result, we agree with the reviewer that our model’s behavior is analogous to style transfer. Furthermore, we think the SCA embedding encodes key foreground information from the masked image (Fig. 2(f)) including style, color, textual of the concept of a dark insect. Conditioned on those, the synthesis model is able to reconstruct the final generated image (Fig. 2(c)).

Question: In some experiments (such as Section 4.3), only a small subset of the dataset is selected. Is there a reason not to test on all fine-grained datasets?

Answer: We used the two largest, representative and most challenging datasets (ImageNet and iNaturalist) in our experiment setting to analyze the quality of the generated images in Table 5 of Section 4.3. We complete the Table 5 with the remaining 6 FGVC datasets. We will include the following results in the supplementary of the revised manuscript.

Question: Focuses exclusively on fine-grained visual tasks; applicability to broader or non-visual domains remains unverified.

Answer: Our focus on fine-grained classification tasks is motivated by practical challenges in niche domains where expert models must be trained on domain-specific datasets that are often small or class-imbalanced. GDA is particularly valuable in these settings, as it can generate faithful and diverse samples by leveraging world priors learned from synthesis models, thereby improving robustness of downstream classifier. However, our current synthesis pipeline is built on a diffusion model (SDXL), which is not well-suited for complex vision tasks such as text-rich VQA or GraphQA. There are other interesting experiments for our method to demonstrate: a) broader applicability to diverse domains, b) application to multi-modal models.

a) broader applicability to diverse domains: We perform a few-shot experiment on Places365, a dataset for abstract scene understanding rather than the object-centric datasets we primarily focused on. The evaluation is done on its validation set. We follow the same the N-way 10-shot experiment setting in Table 2 of the manuscript. The result is shown below:

Table 1: Classification accuracy (%) using 10-shot training samples on Places 365 validation set. Our method out-performs peer method by 1.53% and 1.78% in Top-1 and 5 accuracy, respectively, demonstrating that our method is also effective on datasets of abstract scene as well.

b) application to multi-modal models: Classification is still an important task for modern multi-modal models that often require fine-tuning domain-specific dataset to achieve production-level performance. To demonstrate the effectiveness of our generation method in this context, we explore multi-modal visual classification by performing supervised fine-tuning on an MLLM using FGVC datasets. In this setup, we tune only the projector and vision encoder and keep the LLM frozen. The classification problem is reformulated as a 26-choice QA task, where the ground-truth label is always included among the choices. An prompt example is provided below: "You are an expert classifier. Based on the visual evidence in the image, select the most appropriate category. Please choose ONE of the following: A. beef_tartare B. creme_brulee ... Z. tacos Please respond ONLY with your answer wrapped in this format: <answer> LETTER </answer>."

Table 2: Classification accuracy (%) under N-way 10-shot setting with Qwen2.5-VL-7B backbone. We report the zero-shot and fine-tuned performance. The zero-shot performance is not satisfactory on majority of the FGVC datasets. When fine-tuning the model with augmented data, our method out-performs Diff-mix by 3.98% on average across 6 FGVC datasets.

Question: Fails to analyze computational costs of training the SCA embedding model, a critical factor for adoption.

Answer: Our method involves two training stages: (1) training the SCA embedding model, and (2) fine-tuning the synthesis model using SCA embeddings; training the SCA model requires about 20% of the time needed to fine-tune the synthesis model. As shown in the table below, on an 8×A100 GPU setup, training both components takes approximately 20–30 minutes for smaller-scale FGVC datasets of size ~10K, and up to 3 days for iNat2021 with 2.7M images, the largest dataset we have experimented with. We argue that this cost is practical for most application use-cases, especially for scenarios where curating high-quality in-domain data is prohibitively expensive. Our method offers a promising solution, giving consistent performance improvements over baseline GDA methods.