Interpretable and Efficient Counterfactual Generation for Real-Time User Interaction

Transparency The black-box nature of the classifier hinders transparency. We explicitly tackle the issue leveraging human-understandable concepts. This approach allows users to understand when a machine makes a choice for a ‘right’ or ‘wrong’ reason as concepts are explicitly stated in the explanation. Evidence of this is further supported by our user study.

Flexibility Our approach is not model-agnostic. In order to apply our explanatory technique in the latent space the minimum requirement is that the classifier uses a gaussian-mixture loss. The work of [1] shows that this loss can be used to obtain equivalent performance to softmax based classification scores for a wide variety of benchmarking datasets and CNN architectures. Since one can explain any DNN that uses this loss with our method creating a counterfactual in the latent-space, in order to generate explanations in the input space a decoding model reconstruction is required. In conclusion, even though our approach does not extend to DNN with softmax classification layers, the requirement to implement our approach is very simple. Other components of our framework, such as label-relevant/label-irreleveant encoders, try to target specific desiderata of counterfactual explanations such as sparsity or validity but the overall applicability of our method is not limited and does not imply worse classification performances.

$\mathcal{P}_2$ violations We are aware of this and we added proof in Appendix B that the deviation from $\mathcal{P}_2$ of the expected counterfactual is bounded and this error is negligible.

Sparsity The role of $z_u$ is to encode generative factors that are shared across labels and therefore should be fixed when computing a counterfactual for a given user specified label. By keeping part of the encoding unchanged this improves sparsity of the explanation. The reviewer rightfully notices that this is only 25% of the whole latent encoding and does not suffice to ensure sparse explanations. For this reason we optimize the trade-off between sparsity and likeliness by the computation of the expected counterfactual in the latent space. Overall Sparsity is therefore tackled in a two-fold manner: By keeping part of the latent encoding fixed and by explicitly optimizing for it in the formulation of our counterfactual-search problem.

Versions of $S$ We apologize for typos. We updated the text to highlight that the presence of the superscript indicates that property $\mathcal{P}_1$ is also satisfied (line 273-281).

Training effects We added a new section in Appendix G to analyze this phenomenon in detail. Instead of plotting cumulative accuracies, we chose to plot cumulative errors, as this provides a clearer view of where and how frequently errors occurred. The results align with our claim, highlighting the presence of training effects among participants since most errors are made in the early stage of the user study.

Application of the gaussian mixture loss The object of the sentence is the gaussian mixture loss function that we introduced in the following line. We updated the text to make sure this is clearer (line 177-180).

Sampling labels We updated the paper to make this clearer (line 1294). More precisely we refer to a two-step sampling mechanism. First labels are sampled according to a distribution (e.g. multinomial) and then images are extracted from the sampled label conditional distribution.

Benefit of rotations It is true that $t$ is a single dimension element but the suggestion still does not compute a univariate estimate but a multivariate one as densities need to be computed according to multivariate distributions. Nonetheless one can estimate the expected position by interpolating and accumulating the densities at different values of $t$ as to approximate the integral. The difference between this approach and the one we suggest is that the reviewer’s approach requires a hyperparameter ‘step’ to evaluate the densities and that the amount of ‘steps’ needed varies at the varying of the length of the segment. Overall the accuracy of the estimate and the performance of this method will depend on the length of this segment which can vary largely according to two factors: The margin of the classifier, (large margins implementation may be used as in [1]) and the class asked by the user (If the user asks a counterfactual for a class that is distant in the latent space from the original instance). In conclusion our approach guarantees competitive running-times which are independent from the decision boundary learned by the model and the counterfactual query. The other suggested approach running times may be susceptible to significant variance.

references

[1] Luss, Ronny, et al. "Leveraging latent features for local explanations."

[2] Wan, Weitao, et al. "Rethinking feature distribution for loss functions in image classification."

Thank you for the detailed response to address my concerns. The clarification on rotations brings insight into the computational efficiency gain, which should be interpreted together with the multivariate gaussian distribution in the latent space. While the clarifications are appreciated, some of the concerns persist.

Transparency The authors claim that their approach tackles the transparency issue of the generator by showing human-understandable concepts. This claim is not entirely convincing. First, the derivation of the concepts requires additional efforts from human experts, raising questions about the scalability. Second, there seems to be a gap between the generator's actual behavior and the patterns inferred by humans through observations. Is there any mechanism to guarantee the derivation of faithful and truth-telling concepts?

Sparsity: I could not find a formal definition of sparsity in the paper, and therefore assume the common understanding in the literature, i.e. sparsity implies altering a minimal subset of features (but please let me know if the context is different in this paper). Given the definition, could the authors elaborate on how their approach ensures sparsity? The computation of the expected counterfactual balances likeliness and closeness (in terms of latent space distance), but its connection to sparsity remains unclear. If the segment $\mathbb{S}$ is not aligned with the latent space axis (which is very likely to happen in a high-dimensional space), modifications will apply to all units in $z_s$ to reach the expected counterfactual. Since each latent dimension corresponds to one concept, this suggests that deriving the final counterfactual involves altering all concepts, which contradicts the definition of sparsity.

Also, I appreciate the authors' effort in visualizing users' cumulative errors in Figure 13, which brings further questions regarding the training effect. From my perspective, the training effect should manifest as users improving their performance over time due to the additional information provided, resulting in (relatively) concentrated errors in earlier stages of a task. This would presumably lead to a curve above the red line in Figure 13, indicating decreasing cumulative errors as users adapt. However, the presented data does not align with this expectation.

While taking the split at question 13 may support the authors' claim of a training effect, it is unclear why a burst in errors occurs after a certain period of training (between questions 8 and 13). Furthermore, the changing patterns between the "Label" and "Label+Explanation" conditions appear highly similar, raising questions about the origin of the claimed training effect.

Empirical evidence We added a quantitative evaluation of our method with FID, COUT and S3 in Appendix C. We compare with the competing approach of [1] as it is the only other counterfactual generating technique which leverages concepts without supervision that we are aware of. We also compare to our method simply returning a point in the segment connecting the instance to explain and the counterfactual class mean for different model confidence values as a mean for an ablation study. Our technique has comparable performance to our competitor while being substantially more efficient.

Applicability Our approach is not model-agnostic. In order to apply our explanatory technique in the latent space the minimum requirement is that the classifier uses a gaussian-mixture loss. The work of [2] shows that this loss can be used to obtain equivalent performance to softmax based classification scores for a wide variety of benchmarking datasets and CNN architectures. Since one can explain any DNN that uses this loss with our method creating a counterfactual in the latent-space, in order to generate explanations in the input space a decoding model reconstruction is required. In conclusion, even though our approach does not extend to DNN with softmax classification layers, the requirement to implement our approach is very simple. Other components of our framework, such as label-relevant/label-irreleveant encoders, try to target specific desiderata of counterfactual explanations such as sparsity but the overall applicability of our method is not limited and does not imply worse classification performances.

Running times We added quantitative experiments to Appendix C where we also evaluate running times. We compare our method with the competing approach of [1]. Results show our approach superior performance across various architecture complexities.

Related work We extended the related work to consider more recent and relevant approaches. Our method separates from mentioned papers because it does not leverage knowledge of a causal graph (a requirement for the other works). In addition, interest for diffusion models high quality image generation led to researchers leveraging them for counterfactual explanations. Although such approaches are able to generate realistic counterfactual images, the resulting counterfactuals lack transparency which is a crucial component of our framework.

Strictly better counterfactual Our optimization process focuses exclusively on the latent space, meaning its effectiveness in the input space depends on the preservation of distances. While this reliance is a limitation, strong reconstruction and classification performance suggest that the assumption is realistic. If distinct inputs were mapped too closely in the latent space, neither reconstruction nor classification would function effectively. We updated the text to make this clearer to future readers (line 245)

Concept labelling Indeed, learned concepts require a human annotator for labeling. We updated the paper to explicitly mention this in the main text (line 354) and in the dedicated section of the Appendix: Appendix E.

Effect of concepts In our study, we directly evaluated the approach that includes both the counterfactual image and the descriptive concepts as this is the setting that is most informative for users. To keep the number of experimental conditions as low as possible we did not specifically evaluate the effect of concepts on explanations. However, we recognize the importance of this research question and plan to explore it thoroughly in a future journal version of our work.

Expectation along a segment We do not assume points in the segment $S$ are normally distributed. But $S$ is a collection of points in the space $\mathbb{R}^d$ and points in $\mathbb{R}^d$ follow a normal distribution. Finally we are interested in computing the expected value of the points that belong to $S$. We update the paper to make this clearer (line 925).

references

[1] Luss, Ronny, et al. "Leveraging latent features for local explanations."

[2] Wan, Weitao, et al. "Rethinking feature distribution for loss functions in image classification."

Thank you addressing my comments. Now, I feel better informed regarding how the paper approaches the notion of concepts and its relationship to approaches based on other generative models like diffusion models. However, the above responses still do not resolve certain issues.

Scalability. I understand the motivation behind evaluating the method on BloodMNIST. However, simply stating that the approach can scale to larger architectures and datasets based on running times for increasing architecture depths is not enough. For example, there are many reasons why VAEs are not (exclusively, they can be combined with latent diffusion models for example) the main method used in today's image generation pipeline, e.g., posterior collapse. In the above, the authors make an implicit assumption that the latent space is able to 'handle everything' and only its size must be controlled. However, there are many reasons to think that, for example, concept identification would not be possible on complex data like ImageNet. Are there any examples in related works showing that specific latent dimensions of VAEs encode easily identifiable concepts on data like ImageNet? In general, I am very skeptical regarding the statement that this approach can easily scale. While I do not state that every method must work with every level of data complexity, this issue must be clarified here. Either by proof with results on bigger and more complex data, or by explicitly saying that this approach is meant for simpler datasets.

Generalization to independent models. Once again, I would be very careful with stating that the approach is able to handle models that are not trained together with the pipeline, since no empircal proof is given. The limitation connected to the loss might not be dramatic, but it requires verification if the authors want to claim that this method is applicable to arbitrary DNN architectures trained through Gaussian mixture loss. Also, the paper's contributions remain limited if no external models are incorporated, since then the entire pipeline reduces to a solution that does not actually provide counterfactual explanations, but rather counterfactual examples only. This is fine if the goal of the method is concentrated on the human-machine interaction, but not enough to state that it provides a general method for counterfactual explanation generation.

Labeled data requirement. The authors mention that their approach is unsupervised with regard to concepts as during training our model does not have supervision with regard to which concepts to encode in the learned representations. Some datasets like CelebA provide additional supervision which can be used to guide the training process but our technique does not leverage this information and we drop this requirement because most real-world datasets do not provide such information. I think that here the relationship between concepts and labels is once again very vague. What is the additional supervision that can be used to guide the training process in, e.g., CelebA, that the authors do not leverage? If I understand correctly, the authors require labeled data for their approach to be trained. Hence, the reliance on additional supervision is there. Note that this has direct influence on the structure of the latent space (even presented pictorially, Figure 1. (b, upper part)) and the 'knowledge' gained by the autoencoder.

Identity breach in screenshot in appendix We apologize for the inconvenience, we did not realize this was part of the screenshot. We removed it from the revised version of the manuscript.

Minor points We addresed the reviewer points and made the requested changes.

Related work We extended the related to work to consider more recent and relevant approaches. Our method separates from mentioned papers because it does not leverage knowledge of a causal graph (a requirement for the other works). In addition, interest for diffusion models high quality image generation led to researchers leveraging them for counterfactual explanations. Although such approaches are able to generate realistic counterfactual images, the resulting explanations lack transparency which is a crucial component of our framework.

Running times We added quantitative experiments to Appendix C where we also evaluate running times. We compare with the competing approach of [1] as it is the only other counterfactual generating technique which leverages concepts without supervision that we are aware of. Results show our approach superior performance across various architecture complexities.

Accuracy The work of [2] shows that this loss can be used to obtain equivalent classification performance to softmax based classification scores for a variety of benchmarking datasets and CNN architectures. In addition the choice of a very simple architecture is due to the very simple input domain but much more complex architectures can be leveraged for our framework.

Evaluation We added in Appendix C a quantitative evaluation of our method with FID, COUT and S3 in the appendix. We compare with the approach of [1] and with returning an explanation consisting of the point in the segment connecting the instance to explain and the counterfactual class mean for a given model confidence value. We experiment with the confidence values of 0.6, 0.8 and 0.9 as a mean for an ablation study.

Image quality Images are blurry due to the reconstruction performance of the model that must leverage restricted latent spaces because of the concept extraction technique. We argue that even though blurry, counterfactuals were interpretable and actionable as the user study results prove.

Disentanglement Label disentanglement is what is requested for the approach to correctly generate valid counterfactuals. Latent disentanglement is also important in our framework, because it allows the method to extract clean and independent concepts that greatly improve the interpretability of our explanatory technique. We are aware of the lack of theoretical guarantees for unsupervised latent disentanglement. Our approach simply encourages it via latent regularization. We updated the manuscript to clearly distinguish between label and latent disentanglement, and we explicitly mention in the related work section the negative theoretical results about unsupervised latent disentanglement.

User study The number of 50 participants is not divided over the 3 conditions. Each condition was instead studied with 50 participants for a total of 150 participants in the study. Furthermore, the study does not require IRB approval in line with the ethical guidelines of our institution. Specifically, we assessed the risk of our study using a survey designed by our institution for this purpose. The assessment yielded a minimal risk level, which confirmed that IRB approval was not necessary.

Technical derivation We added the mentioned technical derivation in Appendix B where we show how the weights are derived. We show how to carry the expected value computation and why we implement our methodology to estimate it.

References

[1] Luss, Ronny, et al. "Leveraging latent features for local explanations."

[2] Wan, Weitao, et al. "Rethinking feature distribution for loss functions in image classification."

limited evaluation We added to Appenidx C a more thorough quantitative evaluation of our approach. We compare with the competing approach of [1] as it is the only other counterfactual generating technique which leverages concepts without supervision that we are aware of. Our technique has comparable performance to our competitor while being substantially more efficient.

100% validity We refer to validity as the explanation being classified by the model as the class asked by the user. In that regard counterfactual candidates are valid by definition as only points on the side of the decision boundary associated to the query class are considered. To further strengthen this, expectations are computed sampling from the conditional counterfactual label distribution making it impossible to obtain explanations that are not predicted as the query class. High-dimensional latent spaces are not considered due to the concept-extraction mechanism which relies on compact latent spaces.

Concept extraction The model learns latent representations which can be associated to interpretable concepts via latent traversal. In order to obtain high quality concepts latent disentanglement is needed. We ecnourage this via latent regularization. We modified related work to make this clearer and expand on the concept extraction technique in Appendix E.

Unsupervised conceps Unsupervised refers to the concepts which are not learned with supervision. Classification is supervised as the reviewer correctly notices. We rephrased to make this clearer (line 415).

Decoder generation This is because with increasing latent dimensions the densities of the points vanishes. This implies that, in order to sample, shaping data according to a distribution is not sufficient. The model needs to additionally learn ‘smooth’ latent space which is achieved with noise addiction. Please refer to [2] for more details.

Auxiliary model The noise injection mechanism is used to ‘smooth’ the latent space as it is already shaped according to a gaussian distribution in the deterministic version of the model. Since our concept extraction technique relies on compact latent spaces which causes a high loss of information after encoding, handling simultaneously noise and reconstruction can be difficult for the decoder. With the suggested approach the decoder focuses only on reconstruction in the first stage and then ‘smooth’ representations are induced with an auxiliary model helping the decoder to handle the noise improving reconstruction quality with respect to the noise injection mechanism of the VAE.

References

[1] Luss, Ronny, et al. "Leveraging latent features for local explanations."

[2] Ghosh, Partha, et al. "From variational to deterministic autoencoders."