Rethinking Visual Counterfactual Explanations Through Region Constraint

We sincerely thank the reviewer for the feedback and appreciation of the mentioned aspects of our paper. We address the reviewer's concerns below.

Weaknesses

W1. Only tested in a limited dataset (...). While ImageNet is considered as a very challenging benchmark for VCE generation (Augustin et al. (2022), Augustin et al. (2024)), we agree that the evaluation could be extended. To address that, we include three additional datasets to ensure a fairer comparison. Specifically, following recent works (Jeanneret et al. (2022), Jeanneret et al. (2023), Weng et al. (2024)), we evaluate RCSB on CelebA and CelebA-HQ, and provide an additional experiment on MNIST to showcase that our method works across the entire spectrum of resolutions and data complexities. These experiments include training I2SB for the inpainting task on each dataset prior to evaluation. Moreover, we adapt the implementation of DiME (Jeanneret et al. (2022), Jeanneret et al. (2023), Weng et al. (2024)) to ImageNet and FastDiME (Weng et al. (2024)) to both CelebA-HQ and ImageNet, to provide a broader comparison. For details of these extended experiments, please see the newly added section Other benchmarks in the Appendix. Overall, our method sets new SOTA in either 5 or 6 (depending on the case) out of 8 considered metrics. We also provide qualitative examples presenting the method's ability to create realistic RVCEs on face images. Regarding the MNIST dataset, we utilize it as additional proof that our method is able to handle shape modifications in addition to textural and color-based changes. For these results, we refer to the Shape modification section in the Appendix. We hope that this extension of our evaluation protocol effectively addresses the reviewer's concerns.

W2. From the results in the main paper, (...). We fully agree that including the mentioned numeric results in the main paper would benefit the reader. However, including additional tables in the main manuscript is not currently possible due to space constraints. Therefore, we include the additional results from W1. in the new sections mentioned above. Moreover, to support future research, we will shortly update the Appendix with quantitative results for all of the classifiers mentioned at the end of the experimental section. [Update: the results are now included in the Appendix].

W3. The use of FID score is misleading (...). We agree with the reviewer's concern that using FID as evaluation measure in this context might be misleading. We use it mainly to ensure a fair comparison with previous works, e.g. Jeanneret et al. (2023), which also identified this issue as FID may be biased by the unmodified original pixels. To alleviate it, also following prior works, our quantitative evaluation uses the sFID metric, which divides the original and generated data into independent splits. Next, FID is computed on splits where the synthetic images are independent from original samples, and averaged. This ensures that the evaluation is not biased by the unmodified original pixels. For a more detailed description of each metric, we refer all readers to subsection Metrics description in the Appendix.

W4. In the paper the authors state that the method (...). We agree that this claim lacked proper support in the initial version of the paper. To address that, we perform an independent user study focused on a. understanding which type of the explanation the users prefer (standard VCE vs. our RVCE) and b. verifying whether the users treat the possible interaction with the method as a useful and helpful addition. We briefly describe the results below. For more details, see the newly added User studies section in the Appendix, where an additional user study focused on improving the understanding of model's failure cases is also mentioned.

We thank the reviewer for the provided feedback and appreciation of the mentioned aspects of our paper. Regarding the reviewer's concerns, we detail our clarifications below.

Strengths

S1. (...) at least on a very small evaluation dataset (3 class pairs). Since a common part of the reviews was the aspect of an insufficiently comprehensive benchmark, we implemented our approach for additional three datasets: CelebA and CelebA-HQ to provide a fair comparison with all of the previous diffusion-based methods, and MNIST as a sanity check application, with which we also wanted partially to address the reviewer's concern about our method not being able to modify shapes. Specific details of these experiments are included in Other benchmarks and Shape modification in the Appendix. Regarding the MNIST experiment, we hope that the qualitative samples provide some proof that our method is able to modify shapes in addition to textures. We further address this aspect below.

Weaknesses

W1. The main contribution of the paper (...). While we fully acknowledge the reviewer's point, we must argue that the classification model does not have to rely on the described changes in overall shape/size, since it may be biased by how these classes (American chameleon $\leftrightarrow$ Common iguana) are represented within the training data. Following the definition of VCEs, one aims at obtaining a minimal semantic change that changes the model's decision, and the change in texture/color seems to simply be enough for the model to change the prediction.

We must also mention that both the region area and the trajectory truncation feature allow for controlling how the original content is preserved. Hence, setting the area to contain almost only the object of interest and picking low trajectory truncation will typically result in textural changes only. To address this concern, we provide new qualitative samples in the Shape modification section in the Appendix, which show that these parameters can be properly modified to result in changing the overall shape and size. Moreover, the samples are supported with a quantitative comparison to previous SOTA on three new tasks, which should mostly depend on modifying the object's shape instead of texture and/or color: pretzel $\rightarrow$ bagel, hatchet $\rightarrow$ hammer and paperknife $\rightarrow$ wooden spoon. We repeat the results from the mentioned section in the table below. Clearly, the performance of our method and its advantage over previous SOTA is largely preserved when compared with Table 1. from our paper. We hope that these extended results, together with qualitative examples, provide sufficient proof of our method's ability to modify entire shapes of objects.

Pretzel → Bagel

Hatchet → Hammer

Paperknife → Wooden Spoon

W2. The intro says that the goal of (...). We agree with the reviewer's point that the aspect of explaining the decision-making process of the classifier to the user was not backed in the original version. To address that, we perform two independent user studies, providing a comprehensive analysis of how our RVCEs are more helpful in understanding the model's reasoning, how the interactive explanation creation serves as added value to the explanatory process and whether our explanations help in understanding the misclassifications of the model.

The first study assessed the effectiveness of RVCEs compared to standard VCEs and the advantages of incorporating an interactive process. The results indicated that 86.6% of participants considered RVCEs more effective than VCEs for understanding and explaining the model's decisions. Key reasons included the ability to make localized semantic changes, enhanced interpretability, and stronger alignment with human intuition. Furthermore, 93.3% expressed a preference for our method over standard static interaction with VCEs when given the option to manually specify regions for RVCEs. This preference was largely driven by the ability to validate regions consistent with human understanding and incorporate input from domain experts.

We thank the reviewer for the insightful feedback and appreciating our paper's clarity. Below, we refer to the mentioned concerns.

Weaknesses

W1. The technical novelty is relatively limited. (...). To elaborate on the aspect of limited novelty, we must point to specific contributions of our work. First, RCSB is the first method to show that a more general class of generative models than SGMs (namely tractable Schrödinger Bridges) can be adapted to the problem of counterfactual explanation generation. Second, contrary to all previous diffusion-based approaches to VCEs, our approach builds from the ground up the practical approach to effective utilization of the context available from the outside of the region. There, the novelties stem from treating the entire process as an optimization procedure, which can be improved by ADAM stabilization and adaptive normalization - two crucial factors in highly increasing our method's effectiveness. Third, the proposed general definition of RVCEs combined with high effectiveness of RCSB allows for obtaining explanations where the region of interest was chosen independently from the classifier, e.g., through automated segmentation or even human interaction, which was not addressed by any of the previous diffusion-based approaches. We believe that these contributions constitute great technical novelty.

Moreover, the quantitative results highlight that our approach outperforms previous SOTA by a large margin. For example, we achieve up to 4 times better FID, up to 3 times better sFID, up to 2 times higher COUT and mach or exceed the best previous results in terms of representational similarity and efficiency. As part of the rebuttal, we extended our quantitative evaluation with two benchmarks, CelebA and CelebA-HQ, used by many previous diffusion-based approaches. There, we also provide new SOTA records, outperforming the previous one in 5 or 6 (depending on the case) out of 8 metrics. We believe that these results highlight the novelty brought by the effectiveness of our method.

Questions

Q1. The paper claimed that the (...). We agree that the aspect of confirmation bias could be more thoroughly analyzed in the paper. While we treat it rather as a starting point for motivating our method, here we describe it more deeply. Beginning with Fig. 1, the regular VCE may bias the user to think that the model's decision changed due to the most visible and the most intuitive changes of the bird, i.e. the modified feathers and head. However, since the VCE also modifies the branch, the background and the copyright caption, there is a risk that these changes influence the decision change more strongly, but are not recognized by users that do not investigate the explanation carefully. On the other hand, RVCEs alleviate this problem by not allowing modifications anywhere except a predefined region.

As part of the rebuttal process, we conduct two independent user studies, which we believe to deeply investigate many practical aspects of RVCEs, in particular the notion of confirmation bias and its presence in VCEs. The first study evaluated the usefulness of RVCEs compared to standard VCEs and the benefits of the interactive process. Results showed that 86.6% of participants found RVCEs more helpful than VCEs for understanding and explaining the model’s decisions. Participants cited reasons such as more localized semantic changes, improved interpretability, and better alignment with human intuition. Additionally, 93.3% preferred our method when comparing standard static interaction with VCEs to manually specifying regions for RVCEs. This preference was primarily attributed to the ability to validate regions that align with human understanding and to involve domain expertise.

In the second study, RVCEs were used to assist a different group in analyzing model failure cases and identifying the minimal semantic changes required for correction. Over 80% of participants found RVCEs effective for addressing the problem. Furthermore, 90.9% successfully identified the missing semantic features needed for the model to make accurate predictions and reported a better understanding of failure cases after reviewing RVCEs. All participants agreed that RVCEs were a valuable tool for explaining the model's decisions. Additional details can be found in the newly added User studies section of the Appendix.

Thanks for the detailed response. After comprehensive consideration of both the reply and the reviews by other reviewers, I maintain my rating score.

We sincerely thank the reviewer for providing such detailed feedback. We deeply appreciate the insightful analysis of our approach. Below, we first list new results on which we further base the response to the reviewer's concerns.

• We extend our evaluation scheme with three new datasets:

  a. CelebA-HQ following previous works (Jeanneret et al. (2022), Jeanneret et al. (2023), Farid et al. (2023)) which serves as a testbed for higher resolution ($256\times256$) face images ($30000$ samples).

  b. CelebA also following previous works (Weng et al. (2024) and the above) as a larger dataset of lower resolution face images ($128\times128$, $\sim200 000$ samples).

  c. MNIST as a proof-of-concept sanity check on very low resolution ($32\times32$) to further extend the difficulty spectrum.

  For each dataset, we first train I2SB on the task of inpainting the freeform (20%-30% area) masks, starting from a pretrained diffusion checkpoint (from the work of Lugmayr et al. (2022) for CelebA-HQ, Jeanneret et al. (2022) for CelebA and from scratch for MNIST). Specific details about the above experiments are now included in new sections in the Appendix: Other benchmarks and Shape modification.

• We perform two independent user studies:

  a. the first one focused on user preference and explanation usefulness when presented with standard VCEs vs. RVCEs, and the added value coming from the interaction with the explanation creation, where regions can be provided manually by the user.

  b. the second one verifying whether the users are able to better understand the misclassification of the model and identify minimal semantic change to correct it when presented with RVCEs generated by RCSB.

  The results are described in detail in a new section User studies in the Appendix.

• We adapt the implementations of DiME to ImageNet and FastDiME to CelebA-HQ and ImageNet to extend the evaluation further. For both methods, we perform an extensive grid search to find optimal hyperparameters on a small subset of images and evaluate the best configuration in a large-scale experiment.

• We provide additional results, where RCSB is evaluated using our automated region extraction approach with the use of lower-importance attributions (Appendix: Lower-level attributions).

• We provide a detailed comparison of RCSB to previous works (Appendix: Comparison to previous works) and thoroughly discuss its limitations (Appendix: Limitations).

Weaknesses

W1. The relationship to several prior works should be discussed more comprehensively.

Before referring to prior works, we first relate to the definition of RVCEs provided in Eq. 2 of our work, i.e. RVCEs are sampled from a distribution $p(\mathbf{x} \mid \argmax_{y'}f(y' \mid \mathbf{x})=y, (\mathbf{1} - \mathbf{R}) \odot \mathbf{x} = (\mathbf{1} - \mathbf{R}) \odot \mathbf{x}^*)$, where $\mathbf{x}^*$ - original image, $y$ - target class label, $\mathbf{R}$ - region, $f$ - classifier. Please note that this definition a. does not assume the dependence of $\mathbf{R}$ on any other components of the problem (e.g., the classifier $f$) and b. assumes that $\mathbf{R}$ is determined prior to the sampling process and kept fixed. We also emphasize that our understanding of inpainting follows the typical definition from the literature (see, e.g., Liu et al. (2023), Rombach et al. (2022)) and refers to the process of infilling after the region (mask) is provided.

The method ACE of Jeanneret et al. (2023) divides the explanation generation process into pre-explanation (I) and post-processing (II) phases (Algorithms 1. and 2. in their Appendix). Phase I is responsible for modifying $\mathbf{x}^*$ with diffusion model-based denoising and influencing it through a PGD attack with the classifier $f$. Phase II first extracts a binary mask based on the absolute change between $\mathbf{x}*$ and the pre-explanation. Then, the region of the mask is unconditionally (without the use of classifier $f$) inpainted with the diffusion model. Therefore, ACE explicitly assumes that $\mathbf{R}$ depends on $f$ and was not shown to work with $f$-independent regions (contrary to our method). Moreover, they do not perform conditional inpainting, since it depends only on the diffusion model, while our approach conditions it with $f$.

CelebA (smile)

CelebA (age)

On CelebA-HQ, the comparison remains largely the same. Our method stays very close to SOTA results of ACE in terms of FID, sFID and CD, while outperforming all other approaches on FVA, FS, MNAC, COUT and FR.

CelebA-HQ (smile)

CelebA-HQ (age)

Regarding the comment connected to combining RCSB with FastDiME, we note that the ablation study included in the main part of our paper contains an adaptation of Manifold Constrained Gradient (MCG, Chung et al. (2022)), which performs the task of inpainting using standard SGMs and adopts the Tweedie trick for the gradient at each step. There, the Jacobian is computed using autodifferentiation. Hence, this variant of MCG is effectively a combination of FastDiME (use of Tweedie, but with exact computation of Jacobian) with our automated region extraction approach providing a fixed region for the entire conditioning process.

W3. Overclaims regarding trustworthiness and 'causality' of the resulting explanations.

We thank the reviewer for such comprehensive analysis of the different aspects of RVCEs. We will try to clarify the claims made in the paper, as it is true that they may not have been conveyed properly.

I would like to thank the authors for this incredibly thorough and comprehensive rebuttal in such a short period of time! This must have been a huge amount of work and is much appreciated.

The various additional experiments do add a lot of value to the paper and provide significant further support to the arguments made by the authors, I believe.

The very detailed comparison with prior work is also very much appreciated; I believe this will be very valuable for future readers. Just one minor nitpick on this: the argument that some of the prior works do not fit exactly into the authors' Eq. 2 really seems quite superficial. E.g. FastDiME does conditional inpainting and could really also use any mask, e.g. LangSAM-based as suggested here. The differences in the core setup really seem quite negligible. This is fine - there are enough other novel contributions in this piece - but then they should also not be unnecessarily overstated. I would strongly suggest citing prior works that also perform conditional inpainting either in section 2 or the conditional inpainting part of section 3, which currently still reads as if the authors were the first to propose this. (Given that esp. FastDiME seems like the most closely related prior work, one could also ask whether this should not be included in the results in the main paper, but I will leave this decision to the authors' discretion.)

I do have some concerns about the interpretation of the newly added user study, however. I concur with everything that the authors write on the subject of the trustworthiness of the resulting explanations. I fully agree that the purpose of VCEs is neither to detect shortcuts nor to fully explain a model, that all the explanations generated in the paper are valid VCEs, and that baseline attribution maps also have many problems. However, there is a large risk when specifically designing an explanation method that is constrained to only change semantically meaningful features of an image: since such explanations align well with human intuition (as the authors themselves note), users will be strongly biased to assume that 'if the model has understood that changing the color of a lemon turns it into something like an orange, then it is likely that the model has well-internalized the concepts of lemons and oranges and that it will generally perform well'. However, as I indicated in my initial review and as the authors also seem to agree, this may be a false conclusion - the model can still be arbitrarily non-robust. In other words, the constraining of explanations to semantically meaningful variations may induce trust on the user's side that is unwarranted. This is actually reflected well in what the authors write about their user study: "The answers generally focused on the semantic change being more localized, easier to interpret and better aligned with human intuition". The fact that an explanation is 'aligned with human intuition' should be completely irrelevant for evaluating the utility of an explanation method: if the model does something very weird, I want to see that, and not an explanation that looks intuitively pleasing but does not reflect the model's actual decision process well. As far as I understand it, there was no evaluation in this study of whether the users actually understood the model's decision process better, or whether they just felt they did? As it stands then, the user study seems to show that users like these explanations, but not whether the explanations are any good in the sense that they really help them understand the model (better). See e.g. Davis et al. or Jacovi et al., specifically section 8.2 on the distinction between warranted and unwarranted trust.

Nevertheless, in light of the otherwise excellent rebuttal and significantly improved manuscript, I am already raising my score.

We are grateful to the reviewer for the insightful feedback. We discuss the mentioned topics below.

• [W1.] We agree that the choice of region extraction method is important and using any automated approach may introduce additional bias to the explanation, e.g., some attribution methods focusing on specific patterns of the image or LangSAM's understanding of particular objects. Sometimes, some form of bias may be regarded as beneficial to our approach. For example, since many attribution methods are very faithful to the model, using highest attributions in the proposed way will focus on semantic features that are most important to the prediction. Moreover, because our approach allows the user to provide the region manually, one can explicitly test any desired bias or get rid of the bias induced by an automated approach.
• [W2.] We also agree that combining generative models with guidance from the model of interest introduces inevitable entanglement. For example, one has to first ensure that the generative model is good enough to properly modify images. However, most of the previous works follow the specific path on basing their method on such entanglement (Augustin et al. (2022), Augustin et al. (2024), Jeanneret et al. (2022), Jeanneret et al. (2023), Jeanneret et al. (2024)), Weng et al. (2024)). Following this research direction, it seems that producing high-quality VCEs requires access to an accurate generative prior, which accurately approximates the original data distribution. This ensures that the signal from the classifier is used to produce a sample that resembles original data. We also agree that this combination might lead to conflicting modifications in the image. On the other hand, our approach is, to the best of our knowledge, the first diffusion-based method to show that the guidance signal may originate only from the classifier of interest, without the use of other regularization components, such as an $l_2$ loss or LPIPS (Zhang et al. (2018)), which might additionally bias the generation, complicating the explanatory process with classifier-unrelated features.
• [W3.] Following and extending our previous answer, the main difference between the described approach and our method is the use of a generative prior that keeps the explanation close to the data manifold. This is required by the definition of a VCE, as we want to obtain a minimal semantic change and avoid any form of adversarial noise. The method described by the reviewer would indeed change the classifier's decision, but simultaneously be far from the original image in terms of semantics. Such an approach is much less informative to the users, since they cannot identify semantic features that are important to the model of interest.

References

Maximilian Augustin, Valentyn Boreiko, Francesco Croce, and Matthias Hein. Diffusion visual counterfactual explanations. In Advances in Neural Information Processing Systems, 2022. Maximilian Augustin, Yannic Neuhaus, and Matthias Hein. Analyzing and explaining image clas- sifiers via diffusion guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

Guillaume Jeanneret, Lo¨ ıc Simon, and Fr´ ed´ eric Jurie. Diffusion models for counterfactual explana- tions. In Proceedings of the Asian Conference on Computer Vision, pp. 858–876, 2022.

Guillaume Jeanneret, Lo¨ ıc Simon, and Fr´ ed´ eric Jurie. Adversarial counterfactual visual explana- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16425–16435, 2023.

Guillaume Jeanneret, Lo¨ ıc Simon, and Fr´ ed´ eric Jurie. Text-to-image models for counterfactual explanations: a black-box approach. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4757–4767, 2024.

Nina Weng, Paraskevas Pegios, Aasa Feragen, Eike Petersen, and Siavash Bigdeli. Fast diffusion- based counterfactuals for shortcut removal and generation. In European Conference on Computer Vision, 2024.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595, 2018.

I would gratefully thank the authors for the rebuttal in the short period, its indeed a hard work. Part of my concerns has been resolved especially the difference between the attribution map generation and the VCE. The newly added user study is good to show the alignment between human's understanding and the generated mask. However, I still hold one concern regarding to the root of the method. From the user study, we can conclude that the method can help person to find the most influence region aligned with human's perception. It is doubtful if the method can explain the wrong behavior of the model especially when the model's prediction is influence by some factors similar to adversarial attach. But this might be a problem beyond this manuscript. I would like to raise my score.

• [W3.] Regarding the results for LangSAM and increased FID and sFID (Table 2. (A)), we follow the justification provided in lines 404-409. The regions extracted with LangSAM entirely cover the main objects representing the given class, which our method modifies to depict objects judged as a different class by the classifier. This naturally justifies that the distribution of the generated explanations shifts far from the original data distribution (for the initial class). While this shift is expected, it may suggest that the generated samples could be lacking in realism. However, as indicated by the S$^3$ metric in Table 2. (A), this does not seem to be the case, since the representational similarity of the images is well-preserved. Moreover, the additional influence on FID stems from the LangSAM masks typically covering an area greater than total 30%. This allows modifying a larger part of the image than, e.g., the results for the automated approach, further justifying the inflation of FID. Hence, we do not consider these specific results as an indication of our method's limitation, as other metrics still indicate its high effectiveness.

• [W4.] We agree that the confirmation bias aspect could be introduced better in the manuscript. Here, we describe it more deeply using the images from Figure 1. of our work. Specifically, previous VCEs modify the given image with no constraints on the region. This leads to an explanation that simultaneously modifies the bird's head, feathers, the branch, background and the text caption in the bottom-right corner. Confirmation bias refers to us, humans, having a tendency to search for or favor the information that confirms our prior beliefs. In this case, this may lead to, based on the explanation, concluding that the model bases its reasoning on the features that changed the most visibly and that are consistent with intuition, i.e., change in feathers and head. However, the change of the caption and branch are much less intuitive and visible, but may constitute the main reasons behind the model's new prediction due to spurious correlation of the data. Our method overcomes this limitation, since it restricts modifications only to the features of interest indicated by the prespecified region.

We hope that the above comments clarify the reviewer's concerns. Thank you for appreciating the well written paper and precisely set research goals. The proposed method accomplishes them while greatly outperforming previous SOTA and creates a new benchmark in the field of counterfactual explanations for image data. We are open to elucidating any further ambiguities and whatever remains unclear. We would also like to encourage the reviewer to increase the proposed score if sufficient justifications were provided or suggest some additional directions to help in that decision.

References

Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos Theodorou, Weili Nie, and Anima Anandkumar. I2SB: Image-to-Image Schr¨ odinger Bridge. In International Conference on Ma- chine Learning, pp. 22042–22062. PMLR, 2023.

Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1–10, 2022.

We sincerely thank the reviewer for the valuable feedback. Below, we address the mentioned weaknesses and concerns.

• [W1.] Thank you for suggesting this analysis. We agree that it is an insightful scenario. Please note that an experiment of this kind has already been performed and described in our paper (Discovering complex patterns with interactive RVCEs subsection, results in Table 2. (B) of our work), but since it was easy to overlook, we elaborate on it in the following. In the paper, we simulate user interaction for the region proposal by evaluating our method on randomly assigned freeform masks from the work of Saharia et al. (2022) (for examples of these masks, see the work of Liu et al. (2023)). Importantly, these masks represent blob-like shapes with similar shape and size to the regions obtained with, e.g., our automated approach, but are not semantically relevant to the image as they are assigned randomly across the entire dataset. Table 2. (B) shows that the semantic relevance certainly helps our method in performing better, but is not critical to its effectiveness (by comparing Table 2. (B) to, e.g., Table 1.). This is intuitive, since freeform masks have no connection with the classifier's prediction, but should still give our method sufficient capabilities to modify the given region so the model's decision changes. To verify that, we also updated the Appendix (Freeform masks subsection) with example RVCEs in sorrel $\rightarrow$ zebra task from Table 2. (B). We observe that our method mostly focuses on modifying the intersection of the random mask with parts that, intuitively, the classifier should be using (e.g., horse skin), while leaving others (e.g., background, sky) mostly unchanged. These observations explain well why the performance of our method with freeform masks is still very comparable to that of the automated approach.
• [W2.] Regarding Table 1., we included three different configurations mainly to highlight the method's versatility across different areas of the extracted regions (respectively 10%, 20% and 30% of total image area for A, B and C). For each area, we found its respective configuration manually on a held-out set of samples, but observed in general that our method is not very sensitive to hyperparameter changes (see Table 6. in the Appendix). Due to lack of space, we did not elaborate on that aspect in the manuscript and are thankful for pointing it out.

References

Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos Theodorou, Weili Nie, and Anima Anandkumar. I2SB: Image-to-Image Schr¨ odinger Bridge. In International Conference on Ma- chine Learning, pp. 22042–22062. PMLR, 2023.

Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1–10, 2022.