GeNIe: Generative Hard Negative Images Through Diffusion

[hT8y][W1]: seemingly comparable performance of Txt2Img and GeNIe:

• Thanks for this comment. Indeed diffusion-based augmentation techniques offer a notable margin compared to traditional approaches (such as Cutmix and Mixup). We offer an enhancement of a diffusion based approach by proposing GeNIe and further automating and enhancing it through GeNIe-Ada.

• Besides, please note that GeNIe and GeNIe-Ada almost consistently outperform txt2Img, a few examples summarized in the following: (i) Few-shot classification scenarios, as demonstrated in Section 4.1 - Table 1 and Table 2. On miniImagenet, GeNIe and GeNIe-Ada offer improvements in the range of 1.3% - 3.1%, and in the range of 0.8% - 0.9% on tieredImagenet. (ii) Fine-grained classification, as demonstrated in Section A2 - Table A1. Here, GeNIe and GeNIe-Ada offer improvements in the range of 5.3% on the Aircrafts dataset and 6.3% on Cars196.

• Furthermore, our newest experiments in Table X1 and X2 (attached PDF) further corroborate the superiority of GeNIe compared to Txt2Img when adopting a more recent and smaller diffusion models (i.e. Stable Diffusion v3, SDXL-Turbo).

[hT8y][W2]: consistently choosing a large $r$. Considering other application such as OOD.:

• The Fig. 6 and Table 4 of Section 4.2 combined tell the story a bit better. Even though it might seem a larger $r$ would be a sensible option to opt for (at least on average), in practice such a choice will distance the model from capturing the context and low-level features of the source category. So, striking a balance between as low as possible choice of $r$ (to capture low-level source features) and as high as possible to result in a high Oracle score (confirming the correct target category) is crucial, and this is what GeNIe-Ada does.

• Note that the margin between GeNIe-Ada and GeNIe (for instance, with $r\geq 0.9$) becomes more pronounced when taking the $1$-shot setting into account, in all few-shot settings.

• Considering GeNIe for OOD applications is a great idea, and we have already started looking into it, following your remark. While we will definitely cite [Your Ref 1, on OOD] in the revised paper, we feel this requires more work than we can afford for this revision and would fit best into our future work.

[hT8y][W3]: Txt2Img with lager number of augmentations to make up for its relatively lower computation complexity:

• Thanks for the insightful remark. Given that GeNIe-Ada searches for the best hard-negative between multiple noise-ratios $r$'s, it naturally requires a higher compute budget than Txt2Img that only uses $r=1$. For this experiment, we use GeNIe-Ada with $r \in \{0.6, 0.7, 0.8\}$ to compare with Txt2Img. Based on this, we only have $3$ paths, with steps of $0.1$), and for each of which we go through partial reverse diffusion process. E.g. for $r=0.6$ we do $30$ steps instead of standard $50$ steps of Stable Diffusion. This practically breaks down the total run-time of GeNIe-Ada to approximately $2$ times that of the standard reverse diffusion (GeNIe-Ada: total $r = 0.6 + 0.7 + 0.8 = 2.1$ vs Txt2Img total $r = 1$). Thus, to be fair, we generate twice as many Txt2Img augmentations as compared to GeNIe-Ada to keep a constant compute budget across the methods, following your suggestion. The results are shown in Table X3. As can be seen, even in this new setting, GeNIe-Ada offers a performance improvement of $0.8\%$ to $1.9\%$ across different backbones.

• Note that GeNIe itself (due to partial reverse process) is actually faster than txt2Img, and yet consistently more accurate.

[hT8y][W4]: GeNIe is applicable to the fine-tuning stage rather than the pretraining stage:

• We agree, even though theoretically nothing stops one from applying GeNIe on a pretraining stage, especially in an offline augmentation setting where latency is of no concern. Note that, data augmentation is mostly beneficial in data deficient settings than the pretraining stage where data is typically abundant.

[hT8y][Q1-p1]: classification involving a larger number of classes:

• GeNIe can be applied to large number of classes, as we already do this in the long-tail ImageNet experiment (Section 4.2). However, as discussed in the paper, we use the confusion matrix to sub-select that source-target pairs in an efficient manner out of the large pool of classes.

[hT8y][Q1-p2]: using confusion matrix post training convergence:

• Thanks for highlighting this. In our ImageNet-LT experiments, we compute the confusion matrix on a held-out set split from the full training set. We will clarify this in the revised draft.

[hT8y][Q2]: incompatible source-target classes:

• Thanks for this comment. Semantic compatibility is indeed a difficult matter to quantify. That said, following your remark, Fig. X1 (see attached PDF) summarizes a few examples where source and target are incompatible. As can be seen, GeNIe starts from pizza and gradually transforms the image towards the volcano while capturing the low-level features of the pizza image. This transformation (semantic switch), however, in such incompatible cases seems to occur at larger noise ratios $r$ when compared to easier cases. We will include this figure in the camera ready version and elaborate on this point in the main text of the revised draft.

[hT8y][Q3]: failure cases of the algorithm:

• Great point. We are conscious that there are exceptional failure cases (where the right choice of $r$ is ambiguous due to the presence of a mixture of both source and target predominant visual features - e.g in Fig. X1, the top row) in which the automated noise selection process does not return an ideal outcome. We will discuss this point further in the Limitations Section in the revised draft.

We do hope this addresses all your concerns, please do not hesitate to let us know if you have any further suggestions.

$**[3KAK][W1]**$: $**How does$GeNIe**$ control which features are retained or changed$:

We instruct the diffusion model to generate an image by combining the latent noise of the source image with the textual prompt of the target category. This combination is controlled by the amount of added noise and the number of reverse diffusion iterations. This approach aims to produce an image that aligns closely with the semantics of the target category while preserving the background and features from the source image that are unrelated to the target.

• To demonstrate this, we have prepapred Fig X3 in the attached PDF. Here, we are progessivley moving towards the two key components of $`GeNIe`$: (i) careful choice of $r$ and (ii) contradictory prompt. The input image is a bird in a cage. The top row shows a Stable Diffusion model, unprompted. As can be seen, such a model can generate anything (irrespective of the input image) with a large $r$. Now prompting the same model with "a photo of a bird" allows the model to preserve low-level and contextual features of the input image (up to $r = 0.8$ and $0.9$), then for $r = 1.0$ it returns a bird but the context has nothing to do with the source input. This illustrates how a careful choice of $r$ can help preserve such low-level features, and is a key idea behind $`GeNIe`$. However, we also need a semantic switch to a different target class as shown in the last row where a hardly seen image of a dog in a cage is generated by a combination of a careful choice of $r$ and the contradictory prompt, leading to the full mechanics of $`GeNIe`$. This sample now serves as hard negative for the source image (bird class).

$**[3KAK][W2]**$: $**adding attributes from the target category ... a ``[dog] with [wings]''**$:

• Based on the reviewer's example, it appears they are suggesting adding attributes from the source category to the prompt. We should clarify the distinction between ambiguous examples and hard examples. For instance, using a prompt like "[dog] with [wings]" results in an image of a dog with wings (please see Fig. X2-[C]), but the label for this example remains unclear due to its ambiguity. Such ambiguous examples could potentially confuse the training process. In contrast, hard examples are those where the label is clearly defined. For example, a prompt like "dog in a cage" provides a clear context and should be labeled as "dog" for an animal classification task. We aim to generate these hard examples, where the correct label is unambiguous, to improve the clarity and effectiveness of the training.

• We suspect the main point from the reviewer's remark is to asses whether the design engineering of $`GeNIe`$ can be replaced by a standard Stable Diffusion model given a more elaborate prompt reflecting on the low-level contextual information of the source image? This is a great suggestion, even though we need to highlight that giving a contradictory prompt opposing the source image is by itself part of $`GeNIe`$'s proposition and novelty. That said, Fig. X2 does exactly that: comparing a standard Stable Diffusion with a more elaborate prompt "a dog in a cage" and $`GeNIe`$. As can be seen, the former can result in dog in a cage where neither dog nor the cage resemble those in the source image. On the contrary, $`GeNIe`$ does preserve the contextual features of the source image, as such generating effective/challenging hard negatives for the given source image.

$**[3KAK][W3]**$: $**advantage of$GeNIe**$ over image editing approaches in hard negative generation$:

• According to our main contributions stated in line 57 of the main manuscript, we do not claim novelty on using diffusion models for image editing. Instead, our primary contribution lies in leveraging these tools to generate $*hard negatives*$ for training. Note that advances in image editing techniques ([Your Ref 1] and [Your Ref 2]) is orthogonal to our contribution, and we believe that improvement in image editing techniques over time, could further enhance our results and increase the effectiveness of our approach. In that light, other diffusion based image editing techniques can also be used as the backbone engine of $`GeNIe`$, where the novelty is (i) to find the right noise threshold and (ii) to provide a contradictory prompt.
• Notably, we will cite both [Your Ref 1] and [Your Ref 2] in the revised draft as alternatives for an image editing backbone.

$**[3KAK][W4]**$: $**choice/effectiveness of source-target class pairs**$:

• You are right. Not all pairs will lead to examples informative for training. For the very same reason, in Lines 136 and 258, we discuss choosing the pairs of source and target using the confusion matrix of an initially trained model (e.g. for long-tail distributions).

We do hope this addresses all your concerns, please do not hesitate to let us know if you have any further suggestions.

$**[Q6Ew][W1]**$: $**slowness of `GeNIe-Ada**`$:

• Thanks for this remark. As we highlight in our limitations, we acknowledge that $`GeNIe`$ is comparatively slower than traditional augmentation methods, while standing on par with (or even faster in the case of barebone $`GeNIe`$, due to partial reverse process) other generative methods. Optimization/Efficiency of diffusion based models is active line of research making approaches like our more favorable in the future due to their superior in performance and capacity.

• When it comes to $`GeNIe-Ada`$, in practice we only probe $r \in [0.5, 0.9]$ (means $5$ paths, with steps of $0.1$), and for each of which we go through partial reverse diffusion process. E.g. for $r=0.5$ we do $25$ steps instead of standard $50$ steps of Stable Diffusion. This practically breaks down the total run-time of $`GeNIe-Ada`$ to roughly $2-3$ times any standard reverse diffusion.

• Lastly, as discussed in Section 5, this latency might be even irrelevant/negligible when it comes to offline augmentation scenarios.

$**[Q6Ew][W2]**$: $**crucial importance of$r$**$:

• We agree, and that is the reason behind automating this critical parameter (through $`GeNIe-Ada`$) in favor of efficiency.

$**[Q6Ew][W3]**$: $**relying on accessing a foundation model**$:

• We believe with the current upsurge of interest, such foundation models are going to become commodities in the near future, further capitalizing on the importance of methodologies like ours when it comes to data augmentation. Notably, $`GeNIe`$ outperforms other Diffusion based competitors adopting similar engine as we illustrate in Sections 4.1 and 4.2.

$**[Q6Ew][Q1]**$: $**ensuring augmented samples are not present in the test set**$:

• This a great comment. To substantiate our understanding, we have a run set of experiments on the few-shot setting. To set the scene, we use pretrained image encoder as an oracle (DeiT-Base) to extract the latent embeddings corresponding to the train (i.e. support) set, test (i.e. query set) and augmentations generated by $`GeNIe`$. Fig. X4 demonstrates the distribution of distances between train-test and augmentation-test pairs across $600$ episodes. As can be seen, the (mean of the) distribution of augmentation-test pair is higher that that of train-test pair indicating that the augmented samples are indeed different from the test sets (based on the strong assumption of train and test sets being mutually exclusive). This is further illustrated in the last column of Fig. X4 on a UMAP embedding plot of a random episode where the embedding of train, test and augmentations are plotted. Here again there is noticeable separation between the augmentation and test samples as compared to train and test samples.

$**[Q6Ew][Q2]**$: $**performance on a different diffusion model or one trained on ImageNet**$:

• Using much larger dataset such as LAION seems to be the prevalent choice for training Diffusion models. We had two concerns in not adopting models based on ImageNet: (i) to avoid limiting the taxonomy of classes to the ones present in Imagenet, but a much larger set (present in LAION); (ii) also considering the fact that some of our few-shot settings (e.g. tiered/mini-ImageNet) are derivatives of ImageNet itself.

• Following your suggestion, we have tried experimenting with both smaller as well as more recent diffusion models (please see the PDF, therein Table X1 and X2). More specifically, we have used Stable Diffusion XL-Turbo to generate hard-negatives through $`GeNIe`$ and $`GeNIe-Ada`$. Few-shot classification results on miniImagenet with these augmentations are shown in Table X1. The accuracies follow a similar trend to that of Table 1 in the main manuscript, where Stable Diffusion 1.5 was used to generate augmentations. $`GeNIe-Ada`$ improves UniSiam's few-shot performance the most as compared to $`GeNIe`$ with different noise ratios $r$, and even when compared to $`Txt2Img`$. This empirically indicates the robustness of $`GeNIe`$ and $`GeNIe-Ada`$ to different diffusion engines. Note that, Stable Diffusion XL-Turbo by default uses $4$ steps for the sake of optimization, and to ensure we can have the right granularity for the choice of $r$ we have set the number of steps to $10$. That is already 5 times faster than the standard Stable Diffusion v1.5 with $50$ steps used through the original submission. Our experiments with Stable Diffusion v3 (which is a totally different model with a Transformers backbone) also in Table X2 also convey the same message. As such, we believe our approach is generalizable across different diffusion models. It is to be noted that for SDv3.0 (Table X2), we test the few-shot classification accuracies on 200 episodes instead of the standard setting following 600 episodes. This leads to a higher standard deviation in the reported accuracy scores. We will report accuracy scores using the complete 600 episodes in the final draft of the camera ready version.

$**[Q6Ew][Q3]**$: $**ensuring generated images resemble natural ones**$:

• We agree that it is almost impossible to ensure every generated image necessarily lies in the manifold of natural images; however, the generated imaged are definitely closer to natural images when compared to traditional methods such as Cutmix and Mixup. We will modify the text to reflect on this accordingly.

• Regardless of generated augmentation lying in the manifold of natural images, we demonstrate the efficacy of $`GeNIe`$ on $7$ different datasets, irrespective of whether or not the downstream datasets have semantic overlap with the pretraining data of the diffusion model.

We do hope this addresses all your concerns, please do not hesitate to let us know if you have any further suggestions.

$**[qsxy][W1]**$: "$*happy with the paper, experiments, and presentation*$", $**selection of noise ratio$r$across different datasets**$:

• We are pleased with reviewer's positive feedback, also for finding our proposed ideas interesting.

• Thanks for the interesting remark. Indeed, in Appendix A2 Table A1 we provide further experimentation on fine-grained classification (as an example of other $*benchmarks*$) which we think corroborates that $`GeNIe-Ada`$ can handle such unforeseen circumstances to a good extent. Notably, throughout the paper we investigate the impact of GeNIe-Ada in $7$ different $*datasets*$(tiered and miniImagenet, CUB200, Cars196, Imagenet-LT, Food101, Aircraft) demonstrating the robustness of $`GeNIe-Ada`$ across datasets. That said, we are conscious that there are $*exceptional*$ failure cases (where the right choice of $r$ is ambiguous due to the presence of a mixture of both source and target predominant visual features - Fig. X1, top row) in which the automated noise selection process does not return an ideal outcome, as is already explained in response to reviewer $**hT8y**$. We will discuss this point further in our revised draft.

$**[qsxy][Q1]**$: $**different diffusion model and/or smaller dataset**$:

• Regarding smaller dataset, the smallest dataset we are aware of adopted for training diffusion models is ImageNet, and using much larger dataset such as LAION seems to be the prevalent choice. We had two concerns in not adopting ImageNet: (i) to avoid limiting the taxonomy of classes to the ones present in Imagenet, but a much larger set (present in LAION); (ii) also considering the fact that some of our few-shot settings (e.g. tiered/mini-ImageNet) are derivatives of ImageNet itself.

• Following suggestion, we have tried experimenting with both smaller as well as more recent diffusion models (please see the PDF, therein Table X1 and X2). More specifically, we have used Stable Diffusion XL-Turbo to generate hard-negatives through $`GeNIe`$ and $`GeNIe-Ada`$. Few-shot classification results on miniImagenet with these augmentations are shown in Table X1. The accuracies follow a similar trend to that of Table 1 in the main manuscript, where Stable Diffusion 1.5 was used to generate augmentations. $`GeNIe-Ada`$ improves UniSiam's few-shot performance the most as compared to $`GeNIe`$ with different noise ratios $r$, and even when compared to $`Txt2Img`$. This empirically indicates the robustness of $`GeNIe`$ and $`GeNIe-Ada`$ to different diffusion engines. Note that, Stable Diffusion XL-Turbo by default uses $4$ steps for the sake of optimization, and to ensure we can have the right granularity for the choice of $r$ we have set the number of steps to $10$. That is already 5 times faster than the standard Stable Diffusion v1.5 with $50$ steps used through the original submission. Our experiments with Stable Diffusion v3 (which is a totally different model with a Transformers backbone) also in Table X2 convey the same message. As such, we believe our approach is generalizable across different diffusion models.

It is to be noted that for SDv3.0 (Table X2), we test the few-shot classification accuracies on 200 episodes instead of the standard setting following 600 episodes. This leads to a higher standard deviation in the reported accuracy scores. We will report accuracy scores using the complete 600 episodes in the final draft of the camera ready version.

We do hope this addresses all your concerns, please do not hesitate to let us know if you have any further suggestions.