Diffusion Counterfactual Generation with Semantic Abduction

We thank the reviewer for their constructive feedback and are encouraged by their comments that semantic abduction is “interesting and intuitive”. Below, we address the major points raised:

“Semantic Abduction” and Novelty

Refer to our response to reviewer vbtG16 for details regarding the novelty and our contributions.

Quality of Abducted Exogenous Noise

We acknowledge the reviewer’s observation regarding Fig 3b. The denoising training procedure may influence the apparent challenges in exogenous noise abduction. However, we note that in Fig 3c and App H - Fig 16, dynamic semantic abduction and $p_\varnothing > 0.1$ improves identity preservation of hairstyle, skin colour, facial structure and backgrounds, which is especially challenging given that we choose not to center crop faces as in existing methods, such as [2,3]. Additionally, we refer to Fig 5 in [2], which shows changes in skin colour, and Fig 2e/2g in [4], which shows changes in background and image fidelity, which the dynamic abduction method largely corrects. Additionally, referring to Table 6 in [4], our effectiveness is on par with theirs.

“For MorphoMNIST, there is no reason to use composition...”

We agree that the method suggested by the reviewer is valid. Instead, we follow [5,6], which uses ground truth morphological measurement functions provided by the MorphoMNIST library to measure the intensity, thickness, and slant of the counterfactual digits directly. Both methods are valid since they use ground truth mechanisms to evaluate counterfactuals. For MorphoMNIST, we only used pseudo-oracles for attributes which don’t have a ground truth mechanism, i.e. digit class ($d$).

“Proposed method does not beat HVAE by a noticeable margin…”

Table 1 shows that for many settings, i.e. spatial and semantic mechanisms with $\omega=3, p_\varnothing=0.1$, our model exceeds HVAE effectiveness. Notably, the spatial and semantic mechanisms improve digit class accuracy by ~4%. When considering effectiveness across all parents holistically, while the target intervention may be successful in the baselines, the image's defining characteristic (digit class) is not preserved.

As requested, we also train VCI (which outperforms DiffSCM) for the scenario in Fig 2a. Again, all interventions cause large drops in digit class effectiveness:

with Comp. ($L_1$) = 1.31e-2, with full table here.

We also compare against VCI trained in a simpler setting using only $i$ and $t$ from the dataset generated with Fig 2a, and also notice large margins of improvement:

where the first three cols are for do(t) and the next three are for do(i), with the full table provided here.

Note that given our choice of metrics differs from those in VCI, the margins of improvement are not comparable across works.

Additionally, we run VCI on CelebAHQ without center cropping, such that effectiveness can be evaluated with our pseudo-oracles, and we notice a large drop in the counterfactual fidelity:

with examples here.

We will include DDCM[8] in our related work. DDCM, which uses text-based LDMs, is challenging to compare fairly with our method due to model size, dataset requirements, and prompt engineering. Their sampling method in Alg 1 in [8] complements our work and naturally extends our mechanisms alongside other sampling strategies[7,9]. This is outside the scope of this paper.

We appreciate the reviewer’s detailed feedback and believe these clarifications and additional experiments demonstrate that our work offers an improved and valuable perspective on diffusion-based counterfactual generation.

[1] https://arxiv.org/abs/2212.12570

[2] https://arxiv.org/abs/2410.12730

[3] https://www.jmlr.org/papers/volume23/21-0080/21-0080.pdf

[4] https://arxiv.org/abs/2303.01274

[5] https://arxiv.org/abs/2306.15764

[6] https://github.com/biomedia-mira/causal-gen/blob/e0e4e22f8ad972b9d3b7dd662fa77d9d7c845078/notebooks/eval_example.ipynb

[7] https://arxiv.org/abs/2206.00927

[8] https://arxiv.org/pdf/2312.04965

[9] https://arxiv.org/abs/2406.08070

We thank the reviewer for the positive comments regarding our framework’s “promising applications in medical imaging and causal reasoning tasks” with “well-designed” experiments and our integration of existing concepts in a “novel way”.

Scalability and Computational Costs

Like most diffusion-based approaches, our method incurs high computational costs at generation time. We acknowledge that techniques such as distillation [1], consistency models [2], and higher-order solvers for sampling [3] can improve efficiency; however, these optimisations are beyond the scope of our current work. We focus on exploring the trade-offs involved in using diffusion models for counterfactual generation, thereby setting the stage for future work addressing efficiency concerns.

Notably, our approach to dynamic semantic abduction via CTA leverages causally conditioned diffusion models to improve identity preservation with a single-step optimisation. This contrasts with methods that rely on less robust, image-specific test-time optimisations - such as expensive multi-step optimisations or heuristics about self and cross-attention maps - often requiring extensive fine-tuning for each image [4, 5, 6, 7]. Our single-step method simplifies and enables rapid testing on larger datasets with a simple model selection process (Fig 5). Dynamic semantic abduction via CTA improves identity preservation of backgrounds, hairstyles, and skin colours (see Fig 3c and App H - Fig 16).

Furthermore, using the latent diffusion model paradigm, our approach can be readily scaled to higher-dimensional images [8]. Our work performs comparably to existing variational methods like HVAEs [9] and VDVAEs [10], which can be challenging to scale to deeper architectures for modelling complex images due to the need to compute and match posteriors [11, 12]. In contrast, the diffusion paradigm uses a fixed posterior, offering a more computationally scalable alternative. We can provide further exposition about this in sec 3.1 for clarity.

Comparison to Causal Disentanglement Approaches in Generative Modelling

In response to reviewer kmep, we provide additional results on VCI [13], whose results improve upon DEAR [14], a popular causal disentanglement generative modelling approach. Our results show that our methods improve effectiveness in the scenario in Fig. 2c, and in a simpler causal modelling problem involving only thickness and intensity, whilst reversibility decreases as guidance scale increases due to trajectory deviation, as described in Sec 3.3. We also focus on methods where an SCM can be assumed instead of methods that infer the assumed causal graph but may struggle with scaling to larger parent sets [16]. We intend to provide further exposition about the challenges associated with defining an SCM in our limitations, and future work could explore incorporating causal discovery frameworks akin to [17].

When compared to CausalDiffAE [17], we condition on $c_{sem} = (z_{sem}, pa)$, whereas they condition solely on $z_{sem}$, which also encodes the parent SCM in the style of [16]. Unfortunately, we could not reproduce the results in CausalDiffAE for the main paper or during the rebuttal period; instead, we have included another baseline from the aforementioned work on VCI [13] in response to reviewer kmep. On inspection of Fig 2a in CausalDiffAE, we believe that their framework may perform poor abduction when all parents are unobserved, as in many real-world scenarios, given their poor preservation of digit style.

[1] https://arxiv.org/abs/2303.01469

[2] https://arxiv.org/abs/2202.00512

[3] https://arxiv.org/abs/2206.00927

[4] https://arxiv.org/abs/2211.09794

[5] https://arxiv.org/abs/2309.15664

[6] https://arxiv.org/abs/2405.01496v1

[7] https://arxiv.org/abs/2212.12570

[8] https://arxiv.org/abs/2112.10752

[9] https://arxiv.org/abs/2306.15764

[10] https://arxiv.org/abs/2303.01274

[11] https://arxiv.org/abs/2401.06281

[12] https://arxiv.org/abs/2208.11970

[13] https://arxiv.org/abs/2410.12730

[14] https://www.jmlr.org/papers/volume23/21-0080/21-0080.pdf

[15] https://arxiv.org/abs/2004.08697

[16] https://arxiv.org/abs/2004.08697

[17] https://arxiv.org/abs/2404.17735

We thank the reviewer for the positive feedback and are pleased that the paper was found to be “written exceptionally well with clear intuitions” and that the experiments “are extensive and show that the proposed abduction mechanisms are quite effective”. Below we address weaknesses and questions:

“Although the comprehensive formulation is important, the main contribution is not clear”...

Concretely, our work distinguishes itself from existing approaches by:

• Spatial Abduction: We use causally conditioned DDIM inversion rather than the unconditional forward process [1,2,3], such that spatial noise can help preserve image structure [4]. We go beyond DiffSCM, by incorporating multiple causal conditions (discrete or continuous), and DCM, by demonstrating our mechanisms on complex vision datasets. Also, we adopt a new SCM for MorphoMNIST, including a challenging relationship between slant and digit.
• Semantic Abduction: Our semantic mechanisms condition diffusion models on semantic encoding and causal conditions from the parent’s SCM, whereas CausalDiffAE [2] incorporates [5] into diffusion models, widely acknowledged to be sensitive to a large number of parents and susceptible to unstable training [6,7], diminishing the benefits of the stable training objective of diffusion models. DiffAE does not provide a method for amortised guidance. Our analysis goes significantly beyond CausalDiffAE and DiffAE, exploring how our amortised guidance procedure trades off effectiveness for identity preservation.
• Dynamic Abduction: We introduce a novel, general, dynamic semantic abduction framework, which we implement via CTA, that learns image-specific guidance tokens during the abduction step in counterfactual inference. These tokens enhance the retention of attributes that cannot be trivially measured and included as causal conditions via the SCM, such as background, hairstyle, and skin colour (Fig 3c, App H - Fig 16).

Our formulation provides a unifying perspective between image editing using pre-trained diffusion models and counterfactual image generation methods. As such, we set the stage for future work on model identifiability and incorporating techniques from LDM-based methods to improve dynamic abduction, which has been a primarily empirical field. Additionally, by carefully considering which variables should be regarded as exogenous or observed within our formalism, we can evaluate counterfactuals via metrics grounded by intuitive soundness axioms and more formally explain tradeoffs between intervention-faithfulness (effectiveness) and identity preservation (composition) when using diffusion models for counterfactual inference or image editing.

Difference between their causal interpretation of classifier-free guidance in Eq (16) and the one proposed by CausalDiffAE

We condition on $c_{sem} = (z_{sem}, pa)$, whereas they condition solely on $z_{sem}$ which also encodes the parent SCM in the style of [5].

What is the motivation behind learning the ϕ token for dynamic classifier-free guidance?

As stated in sec 3.3, we train our guided mechanisms in the standard way [8]. We only use CTA to perform dynamic semantic abduction during counterfactual inference. Given that the CTA optimisation is performed w.r.t composition, the guidance tokens can learn additional semantic information not captured by $z_{sem}$ or $pa$, thereby improving the preservation of hair colour, skin tone and backgrounds in counterfactuals (Fig 3c, App H - Fig 16).

How does this framework translate to text-to-image diffusion models?

Our general dynamic abduction procedure (eq 21-22) is analogous to many test-time optimisations for LDM-based image editing [4, 9, 10, 11], with the fundamental difference that fine-grained causal control can be challenging via text conditioning alone [12]. For an LDM, dynamic abduction using the spatial mechanism with amortised guidance (eq 19) can be formulated trivially within our framework. Natural extensions of our work involve replacing CTA with the test-time optimisation techniques cited above. Further, we demonstrate that shorter DDIM strides with single-step guidance token updates at each timestep improve identity preservation over semantic abduction. In contrast, many LDM-based methods opt for longer DDIM strides with computationally expensive multi-step guidance token optimisations.

[1] https://arxiv.org/abs/2210.11841

[2] https://arxiv.org/abs/2202.10166

[3] https://arxiv.org/abs/2404.17735

[4] https://arxiv.org/abs/2211.09794

[5] https://arxiv.org/abs/2004.08697

[6] https://ieeexplore.ieee.org/document/10021114

[7] https://arxiv.org/abs/2411.19556

[8] https://arxiv.org/abs/2207.12598

[9] https://arxiv.org/abs/2309.15664

[10] https://arxiv.org/abs/2405.01496v1

[11] https://arxiv.org/abs/2403.02981

[12] https://arxiv.org/abs/2212.12570

[13] https://arxiv.org/abs/2410.12730

We thank the reviewer for the constructive feedback and for recognising the significance of our work. We're encouraged by reviewers acknowledging our framework as "an important step towards using generative models for more controlled, interpretable, and robust counterfactual reasoning". We also appreciate the positive remarks regarding our integration of diffusion models with counterfactual inference, the novelty of semantic abduction, and the strength of our experimental design and ablation studies. Additionally, we want to highlight further the novelty, generality, and effectiveness of our dynamic abduction framework, which also offers opportunities to integrate alternative test-time optimisations from LDM-based image editing into structured counterfactual inference beyond CTA.

"It lacks human evaluation to assess the realism and plausibility of generated counterfactuals."

We agree that incorporating human evaluation to assess the plausibility and realism of generated counterfactuals would add value to our analysis. In particular, expert assessment by radiologists for EMBED and experienced annotators for CelebAHQ could provide additional insights beyond our chosen perceptual metrics. However, such user studies are time-consuming and challenging to set up and not feasible as part of the rebuttal. We acknowledge this limitation and plan to include it as a key direction for future work in our discussion section.

"However, it overstates the ease of defining causal structures, as real-world causal relationships are often ambiguous."

We recognise that our manuscript may have understated the challenges involved in specifying causal structures, especially in real-world applications. We will expand the limitations section to acknowledge the ambiguity in causal discovery, the assumptions required to define SCMs, and the fact that our results are contingent on the accuracy of our SCM. Note that if assumptions in an SCM are shown to be incorrect, the transparency of our framework means that developers can redesign their SCM, and either retrain our mechanisms under new assumptions or perform more faithful simulated interventions.

For MorphoMNIST, we leverage the known data-generating process within our DSCM directly for the parents of the image. In the CelebAHQ experiments, we adopt a simplified SCM involving attributes such as smiling and eyeglasses with mild label noise, which can be reliably predicted from observed images using pseudo-oracles, consistent with prior work (see Appendix F, [1]). For the EMBED dataset, our SCM is derived from clinical insights presented in [2] and validated by radiologists. We also note that discovering causal structure is an active and open research area [3,4], and future work could focus on jointly learning causal structure and counterfactual generation, taking inspiration from the theoretical discussion in Appendix C in [5].

"Generalization to more complex, diverse datasets is not fully explored. More testing on real-world counterfactual scenarios would strengthen its conclusions."

We appreciate the reviewer’s concern regarding generalisability. While we agree that broader testing is important, we would like to highlight that the EMBED artefact removal task addresses a clinically important and underexplored problem in a real-world medical imaging scenario. The EMBED variant proposed by [2] contains many patients with significant variations in tissue density and anatomical structure. We believe this makes EMBED a suitably complex testbed for evaluating the robustness of our counterfactual generation methods, directly addressing a real-world challenge of breast cancer detection in screening mammography. Compared to many existing works, we felt that including this real-world medical imaging application sets our work apart from the many ML papers that primarily test on readily available benchmarks.

Furthermore, we have obtained improved results on EMBED since the initial submission, achieving over 90% accuracy in triangular artefact removal. These updated findings will be included in the paper's final version to support our semantic mechanisms' efficacy further.

[1] https://arxiv.org/abs/2303.01274

[2] https://arxiv.org/abs/2410.03809

[3] https://arxiv.org/abs/2004.08697

[4] https://arxiv.org/abs/2307.05704

[5] https://arxiv.org/abs/2404.17735