GeNIe: Generative Hard Negative Images Through Diffusion

[R4,W3] Visualizations. The PCA representations of DINOv2 in Figure 5 could be improved and expanded. DINOv2's representations are excellent, and their PCA is usually very insightful.:

We are not sure if we understand your suggestion perfectly, particularly w.r.t. "improving and expanding" the PCA vizualizations. In Fig. 5, we use PCA to project DINOv2's representations to a 2D space (by selecting the top-2 eigen valued principle components) for effective visualization of our generative augmentations. The sparse distribution of points between an interval of $r$ values in both images corroborates our intuition of a rapid transition between the source to target category. Additionally, we have included more visualizations similar to Figure 5 in the appendix (see Figure A5). From what angle do you suggest we improve this? We would be happy to follow through and make changes per your remarks.

[R4,Q1] Visualizations. Improve visualization of the PCA over DINOv2's embeddings as suggested in Weaknesses section. Also, increasing the resolution of images across all figures would help fully appreciating the variations in image generation.:

Thank you for your feedback. We have enhanced the resolution of Figure 5 and will ensure that all figures are improved for the final manuscript.

---

[R4,Q2] Data augmentation. Run experiments with data augmentation policies combining both geometric and photometric transformations, as suggested in Weaknesses section.:

Sure. Please refer to our explanations and the Table in response to [R4,W2] above.

---

[R4,Q3] Datasets. In the supplementary material Table A1, the augmentation is evaluated on fine-grained classification by training a SVM on DINOv2's pretrained ViT-g. This evaluation is quite insightful and would deserve being integrated in the main paper. To make space, consider relocating some few-shot table contents to the supplementary. It would also be very interesting to see the same evaluation in the non-few-shot setting (i.e. training with the full training set), which could be directly compared to the linear evaluation results reported by DINOv2. Note that for Aircraft, DINOv2 evaluates on the variants categories.:

Thanks for your suggestions. We have now moved Table A1 to the main manuscript, as now indicated by Table 2 in Subsection 4.2.

Since our approach leverages diffusion models for generating augmentations, the process is time-intensive when applied to the full training set for both the baselines and our method. As a result, we focus on the few-shot setting in this paper to ensure feasibility and efficiency.

---

[R4,Q4] Minor remarks on formatting. Consider using multicol to improve table readability. Please write ViT-g instead of ViT-G for DINOv2's model.:

Thank you for your feedback. We have updated "ViT-g" to "ViT-G" as suggested. Additionally, we will enhance the tables to improve readability based on your recommendation.

[R4,W1] Method section: My main concern regards the automatic noise level selection process. It involves generating multiple images with various noise levels and selecting the one where the semantic shift is the highest. This seems highly inefficient given the time required to generate these images.:

• Thanks for this remark. As we highlight in our limitations, we acknowledge that GeNIe is comparatively slower than traditional augmentation methods, while standing on par with (or even faster in the case of barebone GeNIe, due to partial reverse process) other generative methods. Optimization/Efficiency of diffusion based models is active line of research making approaches like our more favorable in the future due to their superior in performance and capacity.

• When it comes to GeNIe-Ada, in practice we only probe $r \in [0.5, 0.9]$ (means $5$ paths, with steps of $0.1$), and for each of which we go through partial reverse diffusion process. E.g. for $r=0.5$ we do $25$ steps instead of standard $50$ steps of Stable Diffusion. This practically breaks down the total run-time of $`GeNIe-Ada`$ to roughly $2-3$ times any standard reverse diffusion.

• Lastly, as discussed in Section 5, this latency might be even irrelevant/negligible when it comes to offline augmentation scenarios.

---

[R4,W2] Data augmentation. I am confused about the weak and strong augmentation policies GeNIe is compared to. Are strong augmentations (random color jitter, random grayscale, Gaussian blur) applied in addition to weak augmentations (random resize crop, random flipping)? Otherwise, why are the weak augmentations solely geometric, and the strong ones solely photometric? A balanced policy combining both geometric and photometric transformations could be a better contender. For example, it could consist of random resize crop (with e.g., a scale of 0.4), random flipping, color jittering, and maybe MixUp or CutMix on top.:

Thanks for the constructive suggestion. Following your remark, we have conducted new experiments with a combination of weak (geometric) and strong (photometric) augmentations, as indicated by Mixed in Table A1 of the revised manuscript. Additionally, we also experiment with Mixup and Cutmix on top of these, as indicated by Mixed + Mixup and Mixed + Cutmix in Table A1. We notice marginal improvements of upto $0.6$% by using the Mixed augmentations either with or without the CutMix, MixUp counterparts.

ResNet-18

---

ResNet-50

We're happy to know that you find our responses detailed and thoughtful; and that you recognize our efforts in answering your questions.

[R4,Q1] I also appreciate the inclusion of experiments on fine-grained datasets in the main paper. However, it is somewhat unfortunate that the method cannot be feasibly applied to full datasets as small as CUB or FGVCAircraft, which consist of approximately 10k images.

Thanks for the remark. We would like to clarify that generating augmentations for the full training set of CUB200 or FGVCAircraft is infeasible for the duration of the rebuttal period. However, we would be happy to include this experiment in the final version of the revised manuscript. To enhance the efficiency of our augmentation process, we propose a simple solution in Section 4.3. Instead of considering every possible pair of source and target classes, we leverage a confusion matrix to focus on augmenting pairs where the source and target exhibit high confusion (please refer to Section 4.3 for more details). We will apply this method and present our results on the complete training datasets of CUB200 and FGVCAircraft in the final version of our manuscript.

Moreover, following your remark, we have now included a more elaborate section on computational complexity of GeNIe and GeNIe-Ada in the appendix Section A7, Table A4 of our revised manuscript (also shown in the table below). As can be seen in the table below , GeNIe is approximately $1/r$ times faster than the base diffusion model (referred to as the Txt2Img augmentation baseline). This empirically corroborates with the total number of denoising steps using in GeNIe vs. Txt2Img. Since, GeNIe-Ada scans for the best hard-negative in $r\in[0.6,0.8]$, it incurs a computational cost of $\approx 2.2\times$ the Txt2Img. Note that the runtime for GeNIe-Ada reported in this table also includes the runtime of performing a batched forward pass through a ResNet-50 feature extraction backbone.

Computational Complexity

[R4,Q2] I am grateful for the additional experiments with mixed augmentations. The very low difference in performance between the mixed augmentations and the purely photometric or geometric strategies is surprising. What is the specific set of augmentations (and their strengths) used for the mixed strategy?

For the mixed strategy, we used random resize crop with scaling $\in [0.2, 1.0]$, horizontal flipping, random color jitter with probability = 0.8, random grayscale with probability = 0.2 and Gaussian blur with probability = 0.5. These augmentations and their hyperparameters have been used widely in self-supervised learning such as MoCo[1] and SimCLR[2].

References:

[1] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[2] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

[R4,Q3] I find the PCA visualizations in the paper insightful but believe an expanded analysis could offer further depth. Performing PCA over transitions involving multiple source/target image pairs, such as various "mushroom" and "volcano" images, could reveal whether transitions form specific lines between individual pairs or represent a broader front between semantic categories.

This is an excellent experiment. Thank you for your suggestions. To establish this further, we now choose $10$ samples of a source class of Mushroom and generate GeNIe augmentations with the target class of a Volcano. The generated images corresponding to each $r \in [0,1]$ are passed through a DINOv2 encoder and their embeddings are projected onto their $2$ principle eigen vectors using PCA. The trajectories extracted from each of these $10$ source images is depicted collectively and individually in Fig A5. Each trajectory demonstrates a smooth progression from the source to the target class, with a sparse distribution of embeddings typically observed within the range $r∈[0.4,0.6]$. This is also observed in the plot on the bottom-right side of the figure where all trajectories are collectively plotted. Here, however, there is no clear range of $r$ where a sparse distribution of points can be observed. This suggests that the optimal $r$ value varies for each source image, likely due to inter-sample differences within the Mushroom class. By operating on individual source images and specific target prompts, GeNIe-Ada effectively identifies the best hard-negative example for each sample.

Furthermore, based on your suggestions, we conducted an experiment in which GeNIe was applied twice to transform between a source image and a target category. For this experiment, images from the ''mushroom'' category were used as the source, and ''volcano'' served as the target category. In the first step, we applied GeNIe using a mushroom image as the source and a volcano prompt as the target. In the second step, we reversed the process: the GeNIe-generated volcano image from the first step was used as the source, with the target prompt set to mushroom. Importantly, using a smaller noise ratio, $r$ during the generation of the volcano image helps preserve more low-level visual features from the original mushroom source image. Consequently, when the roles of source and target are flipped in the second step, the final image retains a stronger resemblance to the original mushroom source image for lower noise ratios. This phenomenon is visualized in Fig.A6 of the revised manuscript. As shown, a lower noise ratio during the first step results in the preservation of more visual features, leading to a final image that more closely resembles the original mushroom source.

We would be more than happy to provide further clarifications and revisions if you have any more questions or concerns, and if not we would greatly appreciate it if you would please re-evaluate our paper's score. Thanks again for your reviews and questions which have helped us to improve our paper!

We're happy to know that you find our responses detailed, insightful and reasonable.

[R4,Q1] I thank the authors for their detailed responses to my comments. The new visualizations are insightful, and I am quite surprised that the trajectories in DINOv2's PCA are always the same for a given source class and target prompt. Could the authors provide further insights into this observation?

Thanks for the remark. We would like to clarify that the DINOv2 PCA representations shown in Fig. A5, A6 illustrate a variance amongst the 10 different trajectories belonging to the same source class and target prompt. Fig. A5 illustrates that while each of the trajectories demonstrate a gradual transition of semantic category from the source to the target class, they still maintain variations from each other. This is further observed in the bottom-right plot of Fig. A5 which collectively visualizes all the trajectories, since there is no clear range of $r$ where a sparse distribution of points can be observed. We attribute this difference in trajectories to the inter-sample variances of images belonging to the same Mushroom class. Since GeNIe-Ada operates separately on each individual source image and target class text-prompt, it facilitates the selection of the best hard-negative per sample per text-prompt.

Remark/Request: We are glad to hear most of your concerns are already addressed. We have done our very best and will continue doing so for all other reviewers as well. If we may kindly ask you, is there any further action we can take which would help further increasing our score to a solid accept instead of a marginal one? This would be a deal-breaker to give our work a chance if you also believe the community would benefit from its acceptance and impact; if so, we are more than happy to take further actions to address your remaining concerns. Thank you again for all the constructive feedback and your considerations!

Thanks for the continued thoughtful and engaging conversation.

[R4,Q1] The transition speed differs between the 10 trajectories, but they all follow almost exactly the same path from the source to the target class.

Thanks for clarifying your remark. We agree that the transition speed differs, but they all follow almost the same path from source to target class. This can be explained by the fact that at any partial noise ratio $r$ along the semantic trajectory, the class represented by the image can only be one of the two given classes - either the source or the target (text-prompt) class. This is illustrated in Figs. A6, A7, A8 and A9, where we qualitatively demonstrate the generated images corresponding to different noise ratios $r \in [0,1]$. Since DINOv2 is pre-trained on a large-scale image dataset, it can reliably identify the semantics of an image and embed it in its representation. Given this, we argue that all the 10 trajectories follow a similar semantic path. The extent/magnitude of the variances in generated images corresponding to partial noise ratios throughout the denoising process has been studied in SDEdit[1], and is outside the scope of this work.

References:

[1] Meng, Chenlin, et al. "Sdedit: Guided image synthesis and editing with stochastic differential equations." arXiv preprint arXiv:2108.01073 (2021).

[R3,W1] They didn't give more contribute to the model itself. It's just an application to generate images.:

A common trend in generative AI space to leave the pretrained foundation model untouched to avoid artifacts such a hallucination and catastrophic forgetting. We follow the same line of thought and we believe this becomes even more customary on the way forward. That said, GeNIe is still a novel generative data augmentation model that uniquely leverages a diffusion-based approach to combine two distinct sources of information—a source image and a contradictory target prompt—to generate hard-negative images. To the best of our knowledge, no other generative data augmentation method specifically explores the effectiveness of generative models in creating hard-negative samples. Furthermore, we introduce a novel method to effectively sample the optimal noise level, eliminating the need for manual parameter tuning and further enhancing GeNIe's performance.

---

[R3,W2] Limited number of downstream cases.: [R3,Q1] Could you add more experiments on some other mainstream tasks? Do you have comparisons with other models for data augmentation? Such as GAN?:

Thanks for the remark. We would like to clarify and re-iterate that we already evaluate our proposed method on a variety of image classification tasks, summarized as follows:

Few-shot classification:

• Evaluated on mini-ImageNet (Table 1)
• Evaluated on Tiered-ImageNet (Table 3).
• Long-tail classification: Assessed on ImageNet-LT (Table 4).
• Fine-grained few-shot classification: Conducted on four datasets—CUBS200, Cars196, Food101, and Aircraft (Table 2).

As such, we believe extensive experimental evaluation is one of the strong aspects of this paper. The use of generative models, such as GANs, has been extensively explored in the past for downstream classification tasks. In this space, our contribution lies in investigating the effectiveness of hard-negative samples generated by generative models. To the best of our knowledge, no other method explores the use of generative models, such as SDv1.5, in generating optimal hard-negatives for classification. Importantly, the choice of the generative model is orthogonal to our work; one could use a GAN instead of a diffusion model, provided the model supports text-based image editing.

For our investigation, we selected open sourced state-of-the-art diffusion models that are publicly available. To validate the effectiveness of hard-negative samples generated by GeNIe, we compared our approach against four diffusion-based data augmentation baselines:

• Img2Img (described in Line 256)
• Txt2Img (described in Line 262)
• DAFusion (described in Line 267)
• Cap2Aug (described in Line 287)

Additionally, we compared GeNIe to traditional augmentation techniques, including strong data augmentation, as well as methods like CutMix and MixUp (explained in Line 290).

Considering the four downstream tasks and four diffusion-based baselines, we believe our evaluation framework is comprehensive and substantiates our claims regarding the effectiveness of GeNIe and its hard-negative samples. Additionally, we further ablate the impact of different generative models in Table 5 to show effectivness of GeNIe across different generative models.

We furthur include additional experiments to demonstrate the effectiveness of GeNIe in scenarios where category labels are unavailable and compare its performance with the DAFusion baseline (Table A3).

Could you suggest specifically which additional tasks/datasets we could include in our experimental evaluations? We will try to include these in our responses to the best of our capabilities given the time-limit of the rebuttal period.

Dear Reviewer eEnJ,

Firstly, many thanks for your rigorous and constructive feedback. The deadline for reviewer-author discussion is approaching soon. We have put in tremendous effort in compiling a detailed response trying our very best to address all your concerns (please see the revised manuscript PDF and our P2P responses). If you are convinced and happy with our responses, please kindly consider re-evaluating/raising your final score; please also let us know if you have any further questions or concerns; we'll be more than happy to address those.

Many thanks for your insightful feedback.

Best regards, Authors.

[R1,W1] The exact method for GeNIe is not detailed in the main script:

Thanks for suggesting this. Barebone GeNIe is pretty simple. It uses a base Stable Diffusion (SD) model, takes as input a source image, selects a partial denosing level ($r$), which is optimized and automated by $`GeNIe-Ada`$, and a prompt to a different target class than that of the source image. This enables GeNIe to generate optimal hard-negative augmentations per source image and target class. We elabore details of our method in Lines 182-205. Following yor suggestion, we have now added a Pytorch-style pseudocode for GeNIe in Section A.5, Algorithm 2 of the appendix of our revised manuscript.

---

[R1,W2] The proposed GeNIe is a simple data augmentation method on the latent space of Stable Diffusion, which lacks the novelty of the method except for modifying noise levels for better augmentation.:

We respectfully disagree with the reviewer. GeNIe is a simple, yet novel, generative data augmentation approach that uniquely leverages a diffusion-based models to combine two distinct sources of information—a source image and a contradictory target prompt—to generate hard-negative images. To the best of our knowledge, no other generative data augmentation method specifically explores the effectiveness of generative models in creating hard-negative samples. Building upon GeNIe, we introduce a novel method (GeNIe-Ada) to effectively sample the optimal noise level, eliminating the need for manual parameter tuning and further enhancing GeNIe's performance. While we acknowledge that GeNIe is a straightforward approach (unlike the more complex GeNIe-Ada), it demonstrates strong effectiveness across a wide range of benchmarks, as highlighted in our work.

---

[R1,W3] GeNIe-Ada requires an additional classifier to search the decision boundary between two categories.:

Thanks for the comment. We would like to emphasize that GeNIe-Ada only requires the downstream task's feature extractor/backbone ($f_{\theta}$), and not its classifier. We have clarified this in the revised manuscript. The backbone is used to extract the image-embeddings $Z_r$ corresponding to their partially-noised images $X_r = `STDiff`(X_S, P, r)$, as illustrated in Algorithm 1 and Section 3. This backbone is pre-trained on the few-shot training set and is thus already available during few-shot testing. This does not introduce any additional requirements specific to our generative hard-negative augmentation method.

[R1,Q1] Is the classifier used for GeNIe-Ada the same as the one trained for the target task? How does the performance of this classifier affect the effectiveness of GeNIe-Ada?:

Thanks for the insightful question which prompted the following experiment. Yes, in all experiments presented in this paper, we use the same backbone for $f_{\theta}(.)$ that is subsequently fine-tuned for few-shot classification tasks. However, to analyze the effect of the backbone feature extractor $f_{\theta}$ on selecting the optimal hard-negative using GeNIe-Ada, we use an IN-1k pre-trained DeiT base instead of the UniSiam pretrained ResNet backbone. However, we still utilize the same ResNet backbone for few-shot classification. We have now included Table A2 in the appendix which demonstrates the results of this experiment. We notice a marginal improvement of upto 0.7% when using GeNIe-Ada+DeiT-B as compared to GeNIe-Ada, which uses the UniSiam pre-trained ResNet backbone. This suggests that there is still potential to develop more effective strategies for selecting noise ratios to further enhance GeNIe. However, in this paper, we limit our exploration to GeNIe-Ada and leave these improvements for future work.

---

ResNet-18

---

ResNet-50

---

[R1,Q2] Is there no information leakage by using stable diffusion in few-shot classification for mini-ImageNet and tiered-ImageNet?:

This a great comment. To substantiate our understanding per your remark, we have run a set of experiments on the few-shot setting as summarized in Fig. A3 of the appendix. To set the scene, we use pretrained image encoder as an oracle (DeiT-Base) to extract the latent embeddings corresponding to the train (i.e. support) set, test (i.e. query set) and augmentations generated by GeNIe. Fig. A3 in the revised manuscript demonstrates the distribution of distances between train-test and augmentation-test pairs across 600 episodes. As can be seen, the (mean of the) distribution of augmentation-test pair is higher that that of train-test pair indicating that the augmented samples are indeed different from the test sets (based on the strong assumption of train and test sets being mutually exclusive). This is further illustrated in the last column of Fig. A3 on a UMAP embedding plot of a random episode where the embedding of train, test and augmentations are plotted. Here again there is noticeable separation between the augmentation and test samples as compared to train and test samples.

---

[R1,Q3] Does using images from all other classes as the source image in line 352 mean that source images are selected from the support set of the other 4 classes, or all other 63 classes (for mini-ImageNet)?:

We use the source images from the support set of the other 4 classes for augmentation in a given episode, and not from all the other 63 classes, thus preventing leakage of data across episodes.

Dear Reviewer X612,

Firstly, many thanks for your rigorous and constructive feedback. The deadline for reviewer-author discussion is approaching soon. We have put in tremendous effort in compiling a detailed response trying our very best to address all your concerns (please see the revised manuscript PDF and our P2P responses). If you are convinced and happy with our responses, please kindly consider re-evaluating/raising your final score; please also let us know if you have any further questions or concerns; we'll be more than happy to address those.

Many thanks for your insightful feedback.

Best regards, Authors.

[R1,Q1] GeNIe-Ada uses an auxiliary classifier backbone to find the proper noise level that transitions from the source category to the target category occur. It seems more soundness for me to use VAE encoder used for Stable Diffusion in this case to get the latents. Are there any rationales for using an auxiliary classifier?

That is a great remark. The rationale behind using the donwstream (few-shot) task pre-trained backbone is to utilize its class-aware/relevant latent space. On the other hand, the VAE used in SDv1.5 is pre-trained using an image-reconstruction task, which is not the most relevant for embedding class-cognizant information in the latent space. For eg: UniSiam uses a mutual-information maximization loss between images and their augmentations, thus allowing the representations to be instance-discriminative/class-aware. Also, since this backbone is anyway used for test-set few-shot classification, we re-use this backbone for GeNIe-Ada to avoid introducing any additional resource requirements.

Following your suggestion, we empirically evaluate the use of VAE latent space for GeNIe-Ada. We conduct an experiment where optimal samples in GeNIe-Ada are selected using the latent space of the VAE encoder used in SDv1.5. As illustrated in the table below, the few-shot accuracies of GeNIe-Ada with a VAE noise selector backbone reduce as compared to the version with the UniSiam pre-trained backbone. We attribute this drop in scores to the inability of the VAE backbone in identifying meaningful class-relevant semantic information present in the generated images. We attribute this to the inherent nature of VAE's pre-training task - image reconstruction. On the other hand, the UniSiam pre-trained backbone involves discriminative pre-training, a task that is more relevant and apt to the downstream task of image discrimination/classification. Additionally, since we already have access to this pre-trained backbone for few-shot image classification, this is not considered as an additional resource requirement.

ResNet-18

---

ResNet-50

[R2,W1] Generative models, while powerful, face limitations in their applicability, particularly with GenIE, as they assume prior knowledge of target categories, making them more effective for background augmentations but less so for category-specific tasks, which require comprehensive generative knowledge and limit generalization compared to approaches like SiSTA.? I would like to see comparisons or discussions regarding the strong assumptions that GenIE makes in this context. This is an important discussion because even simpler methods like mixup do not require explicit knowledge of target categories, which GenIE's reliance on diffusion models necessitates.:

That's an insightful remark. We agree that GeNIe assumes prior knowledge of target categories for the base diffusion model, in order to generate hard-negative examples. Following your suggestion, we have conducted additional experiments to demomstrate the robustness of our method in scenarios where the target class is unknown to the diffusion model. This additional experiment has been added to Table A3 in Section A6 of the appendix of our revised manuscript. We use textual inversion to fine-tune the diffusion model (embeddings) on few images belonging the unknown target class. This fine-tuning allows us to learn embeddings specific to the target class, enabling the generation of the desired hard-negative examples. Note that once the diffusion model is fine-tuned, the procedure to generate hard-negatives using partial noising and a contradictory text-prompt remains the same. As can be seen in the table below, GeNIe+TxtInv performs significantly better than DAFusion baseline. It is important to note that, in this case, we do not utilize any information about the target category labels. DAFusion also employs textual-inversion-based fine-tuning; however, it does so without generating hard-negative samples. This indicates that GeNIe is effective even in scenario where the diffusion model is unaware of the target-class.

Thank you for pointing out the missing related works; we have added them to the revised manuscript. To clarify, we compared our approach to DAFusion (diffusion-based), which is similar to SiSTA (GAN-based) in that both fine-tune generative models to address distribution mismatches between downstream tasks and generated samples. However, our primary focus in this paper is on investigating the effectiveness of hard-negative samples, which is independent of the limitations of generative models themselves. As demonstrated in Table A3, this limitation can be mitigated by fine-tuning diffusion models to learn embeddings corresponding to the target categories. GeNIe can then leverage these embeddings to generate highly effective hard-negative images, outperforming the baseline model, such as DAFusion.

---

ResNet-18

---

ResNet-50

---

[R2,W2] Similarly, the diffusion models used in the paper are large-scale foundation models trained on LAION. This raises questions about the improvements: it is unclear whether the gains are due to the proposed method or the large-scale pretraining of the diffusion model. It would also be important to discuss relevant work, such as (Tian, Yonglong, et al. "Stablerep: Synthetic images from text-to-image models make strong visual representation learners." Advances in Neural Information Processing Systems 36 (2024).), which uses Stable Diffusion to generate synthetic positive samples for contrastive self-supervised learning training.:

This a great comment.

• To substantiate our understanding per your remark, we have run a set of experiments on the few-shot setting as summarized in Fig. A3 of the appendix. To set the scene, we use pretrained image encoder as an oracle (DeiT-Base) to extract the latent embeddings corresponding to the train (i.e. support) set, test (i.e. query set) and augmentations generated by GeNIe. Fig. A3 in the revised manuscript demonstrates the distribution of distances between train-test and augmentation-test pairs across 600 episodes. As can be seen, the (mean of the) distribution of augmentation-test pair is higher that that of train-test pair indicating that the augmented samples are indeed different from the test sets (based on the strong assumption of train and test sets being mutually exclusive). This is further illustrated in the last column of Fig. A3 on a UMAP embedding plot of a random episode where the embedding of train, test and augmentations are plotted. Here again there is noticeable separation between the augmentation and test samples as compared to train and test samples.

• Additionally, following your suggestion on discussing Stablerep, we have now included this work in the related works of our revised manuscript.

• To highlight the effectiveness of GeNIe, GeNIe-Ada as compared to just using large-scale pretraining of the diffusion model, we compare our performance with the Txt2Img baseline as well. The Txt2Img augmentations are generated by simply using class names as text prompts for the SDv1.5 model. As indicated in Tables 1,2,3,4, GeNIe, GeNIe-Ada consistently outperform the naive Txt2Img baseline by up to 3%, across all datasets, tasks and N-way, K-shot scenarios. This empirically demonstrates the effectiveness of using hard-negatives through GeNIe, GeNIe-Ada as compared to simply using generative augmentations through large-scale pretrained SDv1.5.

[R2,W3] Lastly, it would be interesting to see if synthetic image augmentations can be used alongside general mixup, similar to (Graikos, Alexandros, Srikar Yellapragada, and Dimitris Samaras. "Conditional Generation from Pre-Trained Diffusion Models using Denoiser Representations." BMVC. 2023.), which has shown that synthetic images from a conditional diffusion model can provide complementary benefits to mixup.:

Thanks for the remark. Following your suggestion, we have included an additional experiment to assess the few-shot testing performance of UniSiam with our GeNIe augmentations alongside mixup, in the appendix of the revised manuscript in Table A1. We notice a drop in performance of upto $0.9$% when MixUp is used along with GeNIe. This aligns with the general trend of drop in performance when using CutMix or MixUp, as already reported in Table 1.

ResNet-18

---

ResNet-50

Dear Reviewer dDiR,

Firstly, many thanks for your rigorous and constructive feedback. The deadline for reviewer-author discussion is approaching soon. We have put in tremendous effort in compiling a detailed response trying our very best to address all your concerns (please see the revised manuscript PDF and our P2P responses). If you are convinced and happy with our responses, please kindly consider re-evaluating/raising your final score; please also let us know if you have any further questions or concerns; we'll be more than happy to address those.

Many thanks for your insightful feedback.

Best regards, Authors.

We are happy that our responses have convinced you. We sincerely thank you for raising your score!

However, we would like to further understand if any of the other reviewers share your concerns on the "unknown" classes. To the best of our understanding, we notice a similar but not exactly the same question raised by reviewer X612 around potential data-leakage from the LAION pre-training set. To this aim, we have presented a global response to you and reviewer X612, addressing the applicability of GeNIe on unknown classes and clarifying our stance around potential data-leakage from the LAION dataset.

Could you please check this global response and let us know if we have sufficiently addressed all your concerns? If not, we'd be happy to address them in the course of the time remaining for the discussion period.

General Response to reviewers dDiR and X612:

3. To address the comment on potential data overlap, we would like to clarify that Fig. A3 demonstrates that GeNIe, GeNIe-Ada do not create augmentations that mimic the test-set images of miniImagenet. This is illustrated by the by the noticeable separation between the GeNIe augmentations and test samples as compared to train and test samples, in both, the UMAP embedding plot and the distance-distribution plots across all $r \in [0.5,0.8]$. Even if we do consider the case that LAION encapsulates images common in several downstream datasets, we empirically demonstrate that GeNIe and GeNIe-Ada go beyond the naïve methodology of generating test-set similar augmentations by generating hard-negatives that are not only different from the test-set (as indicated in Fig. A3), but are also significantly more effective in improving few-shot classification performance as compared to the Txt2Img baseline (as indicated in Tables 1,2,3,4).

All in all, we demonstrate that GeNIe and GeNIe-Ada do not naively augment the test-set by copying the LAION pre-training dataset, while also being effectively applicable in scenarios where the target-class information is "unknown" to the diffusion-model.

Remark:

We do believe that we have presented necessary and sufficient empirical evidence to support and substantiate our claims around your remarks. However, one last possible direction to experiment on would be to further enhance our experimental evaluation by incorporating datasets such as ChestX[1] or EuroSAT[2], where the dataset classes are significantly more novel to the SDv1.5 model. However, this requires more time than the rebuttal period allows and thus, we would be happy to include our experimental evaluations on one of these datasets in the final version of our revised manuscript upon acceptance. Thanks again for your valuable and constructive feedback!

References:

[1] Xiaosong Wang, et al. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE CVPR, pp. 2097–2106, 2017.

[2] Patrick Helber, et al. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.