xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology

Limited Novelty

Regarding your concerns about the novelty of applying existing XAI methods to MIL, we would like to clarify that it was not the aim of our work to develop a new explanation method for MIL. Instead, the novelty of our work extends across other dimensions:

1. Methodical novelty: Despite attempts to apply XAI to MIL models in histopathology (e.g. [1-10]), there exists no formalism guiding the interpretation of the heatmaps and defining their desired properties. Instead, the heatmaps are often considered a prediction of assumed ground-truth instance labels, which we identified as problematic (Section 2.2). xMIL is a novel framework addressing this gap. Unlike previous works, it posits that bags possess a ground truth evidence function that quantifies the instances’ impact on the bag label. Within xMIL, heatmaps estimate this ground truth, which makes interpreting the heatmaps straightforward and insightful.

2. Empirical novelty: Many high-impact medical publications use attention heatmaps or Integrated Gradients to explain histopathological MIL models [1-8]. To our knowledge, no prior research has systematically evaluated whether these methods produce sensible results. Our extensive empirical evaluation on synthetic and real-world histopathology datasets is the first of its kind. It reveals that the widely used explanation methods may yield misleading results, as they often fail to reflect the model decisions faithfully. In contrast, xMIL-LRP sets a new state-of-the-art for explainability in AttnMIL and TransMIL models in histopathology.

3. Novelty in insight generation: Previous studies [9-10] conducted qualitative assessments of heatmaps on easy-to-learn datasets like CAMELYON or TCGA NSCLC. The insights gained in these settings are limited to model debugging, i.e., “Does the model focus on the disease area?” To our knowledge, we are the first to present a method generating heatmaps that enable pathologists to extract fine-grained insights about the model in a difficult biomarker prediction task.

“This paper does not follow the MIL paradigm. It references only standard MIL assumptions and methods dedicated to that. [...]”

It seems to us that this must be a misunderstanding.

We are aware that there are various formulations and methods beyond standard MIL, and we explicitly reference and discuss many of them (lines 72-73, 86-88, 95-97, 113-115, 125-128), including additive MIL (lines 95-97, 125-128) and the formulation of threshold-based MIL from Foulds & Frank [14] (lines 72-73, 113-115).

In fact, we investigated various MIL formulations but didn’t find any that we deemed suitable to cover all the complexities possibly present in histopathology tasks like biomarker or outcome prediction (Section 2.2). Concretely, with respect to the works mentioned:

• Additive MIL is unable to learn arbitrary instance interactions (lines 37-41, 125-128).
• The MIL formulations in threshold-based and percentage MIL do not include negative evidence, e.g., evidence speaking against a class label. They also don’t account for patch interactions.
• We argue that including negative evidence and instance interactions may be crucial for capturing the complexities of histopathological datasets (lines 105-112).

This is why we formulated the xMIL framework, introducing a novel perspective on MIL that overcomes those issues.

“At some point, you are relaxing MIL assumptions so much that it is not MIL anymore [...]”

Thanks for your comment. However, we do not agree that Definitions 3.1 and 3.2 of our manuscript do not formulate a MIL problem.

In Definition 3.1, we use bags of instances with bag-level labels just as standard MIL and most other MIL formulations (lines 67-68, 135-136). The key difference between our formulation and other formulations is that we do not make assumptions about the relationship between the instances and the bag label, but instead posit the existence of an evidence function assigning context-sensitive evidence scores to the individual instances.

Note that we do not make any assumptions about the (spatial) dependence between the instances. Hence, we neither require nor not rule out that spatial dependence plays a role. Thus, our framework is a general MIL framework that can be reduced to existing MIL formulations with extra assumptions. For example, AttnMIL does not model the correlation among instances, whereas TransMIL does.

We would appreciate it if you could expand on why you think that our xMIL formulation is not MIL anymore. In particular, we would like to understand what exactly you consider requirements for a MIL formulation.

“Moreover, there are models such as CLAM and ProtoMIL that are addressing MIL in a more interpretable way [...]”

Despite extending AttnMIL, the explanations from CLAM are also based on attention scores. Therefore, the method is subject to the same conceptual limitations, namely, the lack of faithfulness and not being able to distinguish between positive and negative evidence (lines 35-37, 89-90, 118-120, we cited CLAM in reference [6]).

To our understanding, ProtoMIL is conceptually different from our approach, as it provides explainability in terms of similarity scores to prototypical instances instead of generating heatmaps over all instances of a bag. Within our xMIL framework, we would therefore argue that it does not estimate the evidence function. Hence, it is unclear to us how to quantitatively or qualitatively compare the evaluated explanation methods with ProtoMIL. We acknowledge that we did not cite ProtoMIL in our paper, but we are happy to include it.

Additional comments

As outlined above, we believe some of the comments and questions may have been based on misunderstandings. We have outlined our key contributions and clarified the raised questions. We are happy to continue the discussion.

References

See Author Rebuttal.

Details on LRP

We acknowledge that some descriptions are too condensed. Therefore, we will add a section in the appendix with text and figures to properly describe LRP.

Faithfulness experiments

We will clarify the faithfulness experiments and their motivation in the manuscript, with algorithmic details already provided in Appendix A.4. The primary goal of the faithfulness experiments is to evaluate the ordering of relevance scores (property 3 of evidence function in Definition 3.2). However, the results also reflect whether the explanation scores contain meaningful positive/negative evidence for the target class: if such information is present, we expect the model’s prediction to flip when all patches supporting the target class are excluded. In Figure 4, the model decision always flips when patches are excluded based on xMIL-LRP scores, whereas other methods show inconsistent results.

Formalization of evidence function

The evidence function is crucial to xMIL. It serves as the ground truth, while the explanation method estimates it. We posit that each instance contains evidence for or against a bag label. Sometimes, the ground truth is (partly) known, such as through pathologist annotations in disease detection tasks. However, many biological associations remain unknown even to experts. Thus, relevance scores provide inexpensive annotation for each patch, as they estimate the ground truth. We will add more details about this to Sections 3.1 and 4.2.

Applying LRP to AttnMIL and TransMIL

We will allocate a section in the appendix for LRP details. In a general attention mechanism, let $x_k$ be the embedding vector of the k-th token and $p_{ij}$ the attention score between tokens i and j. The output vector is $y_j = \sum_{i} p_{ij} x_i$. The AH rule of LRP treats attention scores as a constant weighting matrix during the backpropagation pass of LRP, not affecting the forward predictions. If $R(y_{j,d})$ is the relevance of the d-th dimension of $y_j$, the AH rule computes the relevance of the d-th feature of $x_k$ as: $R(x_{k,d})=\sum_{j} \frac{x_{k,d} p_{kj}}{\sum_{i} x_{i,d}p_{ij}} R(y_{j,d})$.This formulation can be directly applied to AttnMIL, and also adapted to a QKV attention block in a transformer, where $x_k$ is the embedding associated with the value representation. A figure will be added to illustrate how AH and gamma LRP propagation rules combine to explain multi-head attention.

Additive MIL and integrated gradients (IG)

Additive MIL (AddMIL) provides positive and negative class-wise evidence. However, AddMIL cannot model arbitrary instance interactions, which we consider crucial in histopathological prediction tasks (lines 109-112). Comparing post-hoc explanations and AddMIL is challenging due to the lack of ground truth for the heatmaps. The faithfulness experiments do not apply to AddMIL, as its explanations are faithful by design [9]. Furthermore, our manuscript focuses on explaining non-additive models, as they are widely used in the computational histopathology community (see e.g. [1-8]).

We ran additional experiments with AddMIL on the toy datasets, where ground truth evidence is available. We used the Additive Attention MIL implementation (with Adam optimizer) provided by Javed et al. [9], and the same training protocol in Sections 4.1 and A.2 of our paper. The results presented in Table B.1 indicate that AddMIL performed worse in all settings, making its explanations not competitive with the post-hoc explanation methods on AttnMIL and TransMIL. This supports our point that AddMIL may not be competitive in difficult prediction tasks (lines 126-128).

IG has been used to interpret MIL models in histopathology. We implemented IG, and our faithfulness evaluations (Table B.2) show that IG explanation scores are less faithful to model decisions compared to xMIL-LRP, especially for TransMIL. This underscores the importance of our work in rigorously assessing explanation methods for MIL models in histopathology. We will add these additional results to the manuscript.

Multi-class classification

xMIL-LRP is applicable to multi-class classification. All evaluated explanation methods in the paper, except attention, yield one heatmap per class. For example, the 4-Bags dataset in the toy experiments is a 4-class classification problem (Table 1 of the paper).

Regression and survival prediction

We do not explicitly evaluate regression tasks, but prior work shows how to apply LRP in regression [11], which can be adapted to histopathological MIL tasks. In survival prediction, outcomes are typically not predicted via regression due to data censorship; instead, survival probabilities at event times are computed, resulting in Kaplan-Meier analyses. While attention heatmaps have been used to interpret survival models (e.g. [12]), explaining these models remains challenging since they are neither purely regression nor classification tasks. We are extending our work to survival analysis in an ongoing project and will address this in a future work section of the paper.

Feature encoders

A major advantage of our method is that propagation through the feature encoder is not required: the patch scores can be computed at the input level of the MIL aggregation. Thus, the feature encoder’s architecture does not affect the explanation quality. Possible differences in model predictions and heatmaps are due to the quality of the feature embedding. To address this, we repeated the histopathology experiments with the more recent UNI foundation model [13], with the same training parameters as for the best models with CTransPath (CTP) encoder. In our results (Table B.3), xMIL-LRP shows a robust performance in faithfulness experiments in comparison to the results with CTP backbone. Interestingly, while “Single” shows high faithfulness for the LUAD TP53, its performance for NSCLC is much worse in comparison to that with CTP.

References

See Author Rebuttal.

Limited Novelty

Regarding your concerns about the novelty of applying existing XAI methods to MIL, we would like to clarify that it was not the aim of our work to develop a new explanation method for MIL. Instead, the novelty of our work extends across other dimensions:

1. Methodical novelty: Despite attempts to apply XAI to MIL models in histopathology (e.g. [1-10]), there exists no formalism guiding the interpretation of the heatmaps and defining their desired properties. Instead, the heatmaps are often considered a prediction of assumed ground-truth instance labels, which we identified as problematic (Section 2.2). xMIL is a novel framework addressing this gap. Unlike previous works, it posits that bags possess a ground truth evidence function that quantifies the instances’ impact on the bag label. Within xMIL, heatmaps estimate this ground truth, which makes interpreting the heatmaps straightforward and insightful.

2. Empirical novelty: Many high-impact medical publications use attention heatmaps or Integrated Gradients to explain histopathological MIL models [1-8]. To our knowledge, no prior research has systematically evaluated whether these methods produce sensible results. Our extensive empirical evaluation on synthetic and real-world histopathology datasets is the first of its kind. It reveals that the widely used explanation methods may yield misleading results, as they often fail to reflect the model decisions faithfully. In contrast, xMIL-LRP sets a new state-of-the-art for explainability in AttnMIL and TransMIL models in histopathology.

3. Novelty in insight generation: Previous studies [9-10] conducted qualitative assessments of heatmaps on easy-to-learn datasets like CAMELYON or TCGA NSCLC. The insights gained in these settings are limited to model debugging, i.e., “Does the model focus on the disease area?” To our knowledge, we are the first to present a method generating heatmaps that enable pathologists to extract fine-grained insights about the model in a difficult biomarker prediction task.

Unclear Method Description

We apologize for any potential misunderstanding regarding the description of the attention mechanism used in the considered MIL models. We will clarify our description of the AH rule in lines 173-174. In a general attention mechanism, let $x_k$ be the embedding vector of the k-th token and $p_{ij}$ the attention score between tokens i and j. The output vector is $y_j = \sum_{i} p_{ij} x_i$. The AH rule of LRP treats attention scores as a constant weighting matrix during the backpropagation pass of LRP, not affecting the forward predictions. If $R(y_{j,d})$ is the relevance of the d-th dimension of $y_j$, the AH rule computes the relevance of the d-th feature of $x_k$ as: $R(x_{k,d})=\sum_{j} \frac{x_{k,d} p_{kj}}{\sum_{i} x_{i,d}p_{ij}} R(y_{j,d})$.This formulation can be directly applied to AttnMIL, and also adapted to a QKV attention block in a transformer, where $x_k$ is the embedding associated with the value representation. A figure will be added to illustrate how AH and gamma LRP propagation rules combine to explain multi-head attention.

Absence of Fair Comparison with Additive MIL

We agree that just mentioning the performance from the original paper without testing under the same conditions is unfair. We will remove the respective lines from the manuscript (lines 277-278).

A comparison between post-hoc explanations and Additive MIL (AddMIL) is inherently difficult, as we cannot compare heatmaps from different models without having a ground truth. The faithfulness experiments from Section 4.2 are not applicable to Additive MIL, as its explanations are faithful by design [9]. Furthermore, we would like to highlight that the focus of our manuscript is explaining non-additive models, as they are widely used in the computational histopathology community.

Regardless, we ran additional experiments with AddMIL on the toy datasets, where ground truth evidence is available. We used the Additive Attention MIL implementation (with Adam optimizer) provided by Javed et al. [9], and applied the same training protocol as described in Sections 4.1 and A.2 of our manuscript. The results are presented in Table B.1. The test performance of AddMIL was worse in all settings, resulting in explanations that are not competitive with the post-hoc explanation methods on AttnMIL and TransMIL. This supports our point that AddMIL may not achieve competitive performances in difficult prediction tasks (lines 126-128). We will add these results to the manuscript.

Effect of xMIL-LRP on prediction performance

​​​​xMIL-LRP and all the benchmarked methods in this manuscript are post-hoc explanation methods. Therefore, they do not impact the training process or prediction performance of the original model, as they are applied after the model is fully trained. Thus, balancing between model explainability and prediction accuracy is not a concern in this context. We did not investigate utilizing xMIL-LRP scores to further improve prediction performance, but may consider this in future works.

Additional comments

Many thanks for the thoughtful and constructive comments! We hope our responses have effectively addressed your concerns and showcased the novelty of our research. Given your acknowledgment of the relevance and soundness of our work, we kindly hope you might reconsider your rating. If you have any further questions or suggestions, we are happy to continue the discussion.

References

See Author Rebuttal.