Teaching Humans Subtle Differences with DIFFusion

Thank you very much for your feedback. We are glad to read that you found our method novel and the core idea meaningful (along with 3qb4). We are also happy to find that you are satisfied with the experiments (along with Qza6). We hope we’ve answered your questions and addressed the weaknesses in the following sections.

“What is the reasoning behind the exact methodology?”

The goal of our method is to perform an edit to an image that only modifies the category, with minimal modifications to the instance as possible. As such, we must disentangle the category from the instance. We chose EF-DDPM as the inversion technique since the inversion “imprints” the original image into the noise maps better, and so when we perform our edit, it corrupts the instance of the image less. To capture category differences, we propose to perform simple algebra in the image conditioning space, which automatically captures the discriminative features since the subtraction of the representative vectors captures the key differences between the categories.

---

“Could you provide an experiment that captures performance for multi-class problems?”

While DIFFusion is currently framed around a binary source-to-target transformation, the mechanism of computing the diff arithmetic vector that discriminates between classes can be extended to multi-class scenarios. One approach involves designating the positive embedding as the target class mean and the negative embedding as the average of all other classes. This “one-vs-rest” approach would allow us to distinguish between features of one class against all other classes in the dataset. We demonstrate the results of this technique on an extended AFHQ dataset of 3 classes: cat, dog and wildlife, as an example. We trained 3 separate classifiers (cat vs dog+wildlife, dog vs cat+wildlife, wildlife vs cat+dog), and computed corresponding three sets of positive and negative average embeddings ((positive: cat, negative: dog+wildlife), (positive: dog, negative: cat+wildlife), (positive: wildlife, negative: cat+dog)). We then ran inference with these 3 corresponding sets of classifiers/positive+negative average embeddings. We evaluated flip rate and LPIPS on going from the validation sets of the negative classes (ex: dog+wildlife) to the positive class (ex: cat). We repeat this for going from cat+wildlife to dog, and cat+dog to wildlife. Since we can’t submit images, we report flip rate and LPIPS values, we will add the qualitative examples in the camera ready’s supplemental.

The column is the starting class and the row is the target class, where the first number is the LPIPS and the second is the flip rate.

---

“What is the trade-off between identity preservation and class change?”

The tradeoff between identity preservation (dominated by LPIPS) and class change (dominated by success-rate) is further analyzed in Appendix B.1, where we plot SR vs. LPIPS curves. Generally speaking, a higher AUC in these curves indicate an overall better classifier flip-rate and identity preservation to input.

---

“How do you guarantee alignment with scientific facts?”

Identifying whether the edits provided are reflective of dataset biases, artifacts, or true scientific discriminative features would require an expert. All three are very interesting, though. If it is an artifact or dataset bias, it is also good to know about it so that it can be avoided in future datasets that might be used for ML training. As a result, all three case scenarios are useful, but we agree that an expert is needed to disentangle between them.

---

“How does this method get influenced by CLIP’s biases?”

This is a good point. Fortunately, fine-tuning for a few steps ensures that the conditioning provided by clip ends up generating images that are in distribution for the categories, effectively overriding CLIP’s biases in large part. However, we agree that this should be an active area of research before this method gets deployed in the real world, and we plan to explore it in future work.

---

“How do you ensure the difference of means approach doesn't amplify false correlations in training data?”

This could definitely be the case. It depends on what is meant by “false”. For example, in the spurious correlations example, (see section 4.5) we show that “false correlations”, such as Dachshunds always existing in deserts and Corgis in jungles definitely influences the edit in the end, since Dachshunds are converted to Corgis, but the environment changes to slightly more “jungle”-like, and vice versa. The point of our method is that it will pick up on the strongest discriminative feature, and if that feature is “false”, then it will highlight this in the edit. Our method can be used to perform automatic dataset bias identification.

---

“What happens when class boundaries are not linearly separable in CLIP embedding space?”

We can make a reasonable assumption that it is linearly separable in a feature space. The choice of feature space is important, and if it is not linearly separable, there are many representation learning frameworks to make it separable.

Thank you very much for your feedback. We’re very glad that we were able to demonstrate our method as both a technique for explainability and a mechanism for machine-teaching, and that these were both interesting to you! We’re glad that you found our approach methodologically sound (along with reviewer KpBL). We’re also happy to hear that you found our experiments to be principled (along with Qza6 and FpZn). We hope that we’ve answered your questions and addressed the weaknesses in the following sections.

“How can we extend our method to multi-class settings?”

While DIFFusion is currently framed around a binary source-to-target transformation, the mechanism of computing the diff arithmetic vector that discriminates between classes, can be extended to multi-class scenarios. One approach involves designating the positive embedding as the target class mean and the negative embedding as the average of all other classes. This “one-vs-rest” approach would allow us to distinguish between features of one class against all other classes in the dataset. We demonstrate the results of this technique on an extended AFHQ dataset of 3 classes: cat, dog and wildlife, as an example. We trained 3 separate classifiers (cat vs dog+wildlife, dog vs cat+wildlife, wildlife vs cat+dog), and computed corresponding three sets of positive and negative average embeddings ((positive: cat, negative: dog+wildlife), (positive: dog, negative: cat+wildlife), (positive: wildlife, negative: cat+dog)). We then ran inference with these 3 corresponding sets of classifiers/positive+negative average embeddings. We evaluated flip rate and LPIPS on going from the validation sets of the negative classes (ex: dog+wildlife) to the positive class (ex: cat). We repeat this for going from cat+wildlife to dog, and cat+dog to wildlife. Since we can’t submit images, we report flip rate and LPIPS values, we will add the qualitative examples in the camera ready’s supplemental.

The column is the starting class and the row is the target class, where the first number is the LPIPS and the second is the flip rate.

---

“How do you account for CLIP embedding collapsing/misaligning/overcompressing small cues?”

Great point! We noticed this to be the case in some datasets too, particularly OOD datasets. This is where fine-tuning for a little bit (< 2000 steps) helps a lot. Without fine-tuning for example, the Butterfly and Retina datasets struggle because these types of images were likely not seen during CLIP’s initial training.

---

“Is the user evaluation statistically significant?”

We performed an independent two-sample t-test between the results of our method compared to unpaired results, and the best baseline compared to unpaired results. The null hypothesis that the p-value is measuring the likelihood is that the mean of the two underlying distributions is the same. For our result, that probability is less than 5% in both studies, whereas for the best baseline, it is above 80%. While the number of participants in our study is low, such a striking difference in p-values gives us confidence in our results. However, we agree that the number of participants is on the lower side and will do a larger study in our future work.

---

“Please quantify how the visual-algebra automatically isolates discriminative features, meaning are the ∆c directions regularized?”

The approach is discriminative because the difference of means is the same as an LDA classifier (linear discriminant analysis) with an identity covariance. However, our approach will work with any discriminative linear classifier as well, such as a logistic regression or linear SVM.

---

“Please provide a legend for $\omega$ and $T_{skip}$”

Thank you for pointing this out! We will update Figure 8 to label the $\omega$ there in the camera ready.

---

“Since selecting the first edit that flips prediction could bias toward minimal LPIPS, how would you verify no adversarial artefacts emerge with this approach?”

Since the conditioning space is CLIP image encoder’s latent space, and the diffusion model we used was trained to be conditioned on this space, the prior is to generate images that are on the natural image manifold. We verify this qualitatively.

---

“Please report standard deviations over the 5 $\omega$/$T_{skip}$ samples to gauge stability.”

$T_{skip}$ (Mean ± Std Dev) by Dataset and Manipulation Value

In Table 2, we reported results per single $\omega$, according to the best LPIPS. For AFHQ, $\omega=1.0$, and the rest are $\omega=2.0$.

---

“Can you provide the exact video frame rate and if participants could replay. Also clarify IRB approval ID.”

The frame rate was 1 FPS (so 2 second gif). Our study is under IRB approval, and we will release the approval ID for the camera ready.

---

“Please clarify if DIFFusion can chain two vectors, for example Monarch->Viceroy->Queen without any re-inversion?”

While the core mechanism of manipulating conditioning vectors could theoretically be extended, directly chaining multiple vectors without re-inversion is not explicitly supported by the current method. It would likely require either re-inverting the image after each transformation to obtain new noise maps for the intermediate image, or developing a more complex sampling strategy, which is a promising direction for future work.

---

“What is the computational cost of DIFFusion? How does it compare to existing methods / is it a blocker for real-time deployment?”

• For evaluation, as noted, we generate ten candidate edits per image to thoroughly explore the SR versus LPIPS trade-off against baselines. However, since $T_{skip}$ and $\omega$ are only being used during sampling, the inversion process is performed only once per input image. Therefore, the total computational cost for each image is actually 1×(inversion cost)+10×(generation cost), not 10×(inversion+generation).
• In our experiments and in the table above we can see that a choice of $\omega$=2 and $T_{skip}$ between 0.8-0.9 is usually near-optimal for most if not all datasets. Therefore, for real-world scenarios, the search space could potentially dramatically be reduced when running on a new domain.
• The effective per-image runtime (if run in a batch) is approximately 5 seconds on an A100 GPU.
• As diffusion models continue to evolve towards faster and more efficient architectures, such as distilled diffusion models (which can operate with only a few or even a single step), DIFFusion's runtime will inherently improve.

Thank you very much for your feedback. We are grateful that you found our experiments to be rigorously evaluated, both from a metrics point of view and from a user’s educational standpoint. We are glad that you found our experiments compelling and convincing (along with reviewer 3qB4 and FpZn), our results good (along with KpBL), and that the theory seemed reasonable to you. We hope we’ve answered your questions and addressed the weaknesses in the following sections.

“Why do the best baselines sometimes perform worse than a user analyzing unpaired data in the user study, raising concerns about the experiment's utility or the baseline selection?”

The chosen baselines perform well on familiar datasets where humans already understand class differences (e.g., AFHQ, CelebaHQ), confirming their appropriateness for counterfactual image generation. However, they struggle on scientific datasets (Table 2), likely due to visual corruptions that distract users. However, we can only conduct meaningful user studies on these challenging scientific datasets, where participants must learn new concepts, precisely where baselines fail. This methodological constraint actually highlights DIFFusion's value: when genuine learning is required on unfamiliar content, DIFFusion's minimal, identity-preserving changes that reveal true discriminative features become crucial for effective human learning.

---

“Could you please elaborate on the core technical novelty of DIFFusion, distinguishing its contributions from existing state-of-the-art image editing and counterfactual generation methods?”

DIFFusion introduces a novel perspective on diffusion-based image editing as a machine-teaching tool, generating counterfactuals to help humans learn class-discriminative features that are hard to verbalize. Unlike prior counterfactual generation work that relies on instance-level perturbations [1,2], DIFFusion uses simple latent-space arithmetic that enables realistic, fine-grained edits on limited data.

---

“Could the authors clarify the specific mechanisms within DIFFusion that mitigate these data requirements, especially regarding "aligned, abundant data"?”

DIFFusion avoids the need for unpaired data by defining a semantic shift between classes. The shift can be applied to any input to guide the generation toward the target class. To address data scarcity, DIFFusion (1) relies on the robustness of CLIP embedding averages, where only a few samples per class are needed to define meaningful class directions, and (2) uses LoRA when needed to fine-tune a large pretrained diffusion model with just tens to hundreds of examples.

---

"Furthermore, how would DIFFusion's performance, particularly its LPIPS, compare to baselines when abundant data is available, and would it still offer a significant advantage?"

DIFFusion's performance works better than the baselines even when abundant data is available. To clarify, the results in Table 2 are after training and computing the directions over the training set of each dataset.

---

[1] Guillaume Jeanneret, Loïc Simon, and Frédéric Jurie. Diffusion models for counterfactual explanations, 2022.

[2] Guillaume Jeanneret, Loïc Simon, and Frédéric Jurie. Adversarial counterfactual visual explanations, 2023.

Thank you very much for your feedback. We are glad that you found our method both simple and capable of strong and realistic counterfactual image generation (along with Qza6). We hope we’ve answered your questions and addressed the weaknesses in the following sections.

“What is the technical innovation?”

Our technical innovation is in unlocking the capability of performing subtle edits on limited data, which was previously not possible in scientific domains for example, as demonstrated via the baselines’ qualitative and quantitative results (see: Table 2).

---

“How can the model be extended to multi-class settings?”

While DIFFusion is currently framed around a binary source-to-target transformation, the mechanism of computing the diff arithmetic vector that discriminates between classes can be extended to multi-class scenarios. One approach involves designating the positive embedding as the target class mean and the negative embedding as the average of all other classes. This “one-vs-rest” approach would allow us to distinguish between features of one class against all other classes in the dataset. We demonstrate the results of this technique on an extended AFHQ dataset of 3 classes: cat, dog and wildlife, as an example. We trained 3 separate classifiers (cat vs dog+wildlife, dog vs cat+wildlife, wildlife vs cat+dog), and computed corresponding three sets of positive and negative average embeddings ((positive: cat, negative: dog+wildlife), (positive: dog, negative: cat+wildlife), (positive: wildlife, negative: cat+dog)). We then ran inference with these 3 corresponding sets of classifiers/positive+negative avg embeddings. We evaluated flip rate and LPIPS on going from the validation sets of the negative classes (ex: dog+wildlife) to the positive class (ex: cat). We repeat this for going from cat+wildlife to dog, and cat+dog to wildlife. Since we can’t submit images, we report flip rate and LPIPS values, we will add the qualitative examples in the camera ready’s supplemental.

The column is the starting class and the row is the target class, where the first number is the LPIPS and the second is the flip rate.

---

“Is the methodology novel or is it a straightforward application of existing methods and uses image manipulation strategies?”

While previous work can generate counterfactual images in common domains, as demonstrated in the quantitative and qualitative results, they don’t perform well in scientific domains. The technical novelty lies in developing a system that can automatically identify the key discriminative features, and apply them just enough to modify the structure without corrupting the instance, which is useful to teach humans subtle differences between categories. We believe that the simplicity of the method is its strength.

---

“What makes the proposed methodology different with respect to previous works in machine teaching for vision problems?”

Thank you for pointing this out! We view this work as highly complimentary. While [1] and similar machine teaching methods focus on optimal example selection for multi-class teaching, our approach fundamentally differs by generating synthetic counterfactual examples rather than selecting from existing data. The key distinction lies in our method's ability to automatically identify and visualize discriminative features that explicitly demonstrate the minimal edit necessary to flip the classifier’s prediction. However, we will make sure to cite this.

[1] Becoming the expert - interactive multi-class machine teaching. CVPR 2015.