DSEG-LIME: Improving Image Explanation by Hierarchical Data-Driven Segmentation

A revised version of the manuscript, incorporating the feedback marked with (*) in the text, will be uploaded next week. Any additional changes will be included in the final camera-ready version.

We are grateful for your recognition that our paper is well-motivated and addresses an important issue in XAI. Below, we present a detailed response to each of the points raised, addressing the questions directly where applicable.

Point-by-Point Response

[...] not allow to fully understand if the method is correctly assigning large feature attribution to the right regions.[...]

We acknowledge that we had not previously considered this issue. Your point is well-taken (Q1), and we fully agree with your observation. We will integrate the suggested examples into our paper and explicitly highlight this issue in the main text (*), as it is indeed relevant and valuable. Additionally, we will include a quantitative assessment in the appendix (final version) to further strengthen our analysis. Thank you for bringing this to our attention!

The hierarchical technique is described briefly, but does not come with a clear formulation or pseudo-code. [...]

To address your concerns, we will include additional details about the specific alterations made to LIME. As you suggested, we will integrate pseudocode (Q2) and modify the main paper to make these changes more transparent in the revised version. We believe this will resolve the issue and improve the clarity of our work(*).

[...] arbitrary number of iterations d of the algorithm. However, identifying how many and how large the relevant region(s) are is a very relevant problem, which does not seem to be conceptualized and approached very solidly [...]

We appreciate your insightful comment on the multi-scale analysis of DSEG-LIME. Our goal was to conduct a basic evaluation across all methods and evaluation techniques using consistent configurations to ensure fairness and objectivity. While we recognize that fixing a parameter (d) may not fully capture the multi-scale nature of the problem, our intention was to establish a uniform evaluation framework rather than optimize for specific methods. To address your feedback, we will include an additional experiment with (d = 2) in Appendix A3. Furthermore, in response to concerns about attribution maps (mentioned in 1), we will incorporate this feedback while adhering to the standard procedures in LIME-related studies.

In Eq. (1) you use a distance function D(), which actually count the number of masked segments. Why is this a fundamental component (181)? [...]

We confirm that we use LIME's original distance function without any modifications. However, we understand the potential confusion (Q4) caused by the way this was described. To clarify, we reference the effect of superpixel generation on this function in lines 212-214 but do not explicitly modify it. We will revise the paper to eliminate this ambiguity (*).

A critical component of LIME relates the replacement values that are used to mask the segments. [...]

We take the mean pixel value of each segment in our approach. We will explicitly integrate this information into the revised version of the paper for clarity (*).

You report that a weakness of SAM is that it may generate overlapping segments, which makes it somewhat troublesome to deal with. [...]

We apologize for any confusion regarding overlapping superpixels. In DSEG-LIME, we only consider non-overlapping superpixels. Overlap information is solely used to identify the parent segment of smaller segments. At no point does LIME assign relevance scores to the same pixel belonging to two different superpixels. To address unsegmented areas, we apply Equation (7) to fill these gaps with the relevance score of the nearest superpixel. This explanation also addresses the question raised in Q6. To clarify further and avoid future confusion, we will include pseudocode (Q2) and expand this explanation in the main text (*).

(270) What is a grid overlay parameterized by points? Please clarify and/or add references.

The grid overlay is directly tied to SAM, which determines the number of grids used for automatic mask generation [1]. This parameter influences the number of identified segments, and we will ensure this relationship is clearly articulated in the revised paper (*).

(1384) Why are you mentioning a "NeurIPS code of Ethics" [...]?

We adhere to the “NeurIPS Code of Ethics” to ensure compliance with ethical standards. We understand the confusion surrounding this reference and will revise the terminology to prevent further misinterpretation (*).

A revised version of the manuscript, incorporating the feedback marked with (*) in the text, will be uploaded next week. Any additional changes will be included in the final camera-ready version.

We thank the reviewer for their thoughtful feedback and for highlighting our contribution to making LIME’s interpretations more intuitive for end-users and emphasizing its flexible approach.

Below, we address the points raised in detail.

Point-by-Point Response

[...] novelty of this manuscript [...] combines elements from SAM and LIME [...].

The integration of SAM is a key motivation for using segmentation models in post-hoc explanations. DSEG-LIME acts as a framework where SAM is one possible segmentation backbone that can be utilized. In the paper, we also report statistics with DETR and SAM2. Additionally, we not only integrate SAM into the LIME framework but also construct a hierarchical tree that guides LIME in its feature importance calculations. This transforms LIME’s pipeline from a static to a dynamic approach, enabling users to explore model behavior from coarse to fine-grained concepts while offering control over the level of granularity. That said, we will revise the formulation of SAM within DSEG to shift the emphasis toward the framework itself and the overarching hierarchical concept (*).

[...] A more relevant comparison with other XAI methods is needed.

We primarily compare SAM within DSEG against other mentioned segmentation techniques to evaluate how the hierarchical conceptualization built internally in DSEG compares to other approaches. Additionally, we examine how this internal representation performs at deeper levels of the hierarchy in Appendix A3. Since DSEG does not rely solely on SAM (e.g., it also utilizes SAM2 or DETR), our key contribution lies in the DSEG framework itself. This framework transforms LIME into a dynamic process by constructing a hierarchical superpixel tree, allowing to leverage various segmentation techniques as its backbone. In addition, incorporated an evaluation against EAC in the supplementary material, utilizing SAM as the segmentation backbone for SHAP [1]. Our main comparisons focused on DSEG against other LIME-based techniques, as highlighted in related works [2,3,4].

[...] threshold 𝜃=500 to filter out small masks, [...] strategy should be reconsidered or justified more robustly.

The influence of the threshold of 500 is further investigated in Appendix B1, where we present an ablation study. While testing different values might improve DSEG-LIME’s results, we aimed to avoid extensive hyperparameter tuning in the initial quantitative study. In our view, the end user has the flexibility to decide whether smaller superpixels should be considered, much like setting hyperparameters for other segmentation techniques.

Additional visual explanations comparing DSEG with current state-of-the-art XAI methods would strengthen the manuscript[...]

We will integrate more experiments with advanced techniques in the paper for camera-ready version.

---

Answer to Question: We acknowledge that the current results may not fully demonstrate whether DSEG improves LIME’s stability. To address this, we plan to build a dataset consisting of images with more complex and diverse concepts, which are designed to challenge the stability of LIME. We will evaluate DSEG on this dataset and include these additional results in the camera-ready version. Furthermore, we will consider incorporating metrics such as Local Fidelity from GLIME to provide a more comprehensive evaluation of DSEG’s impact on LIME’s performance and stability.

In the meantime, we integrated your feedback by applying our DSEG framework to the SLICE [1] approach and conducted a comparative evaluation against Quickshift (QS) using hyperparameters from the SLICE repository to test stability.

Experimental Setup

• Dataset: 50 images from the main paper.
• Iterations: 200 steps per image, repeated 8 times.
• Metrics:
  • Average Rank Similarity (p = 0.3): Focused on positive/negative segments. (Different segmentation numbers were taken into account to consider the same number of segments for computation, by min length of both)
  • Sign Flip Analysis: Number of segments flipping signs during computation (Considering all segments of each technique).

Results

Key Observations

• Similarity: DSEG and QS exhibit comparable performance in rank similarity.
• Sign Flip: DSEG significantly outperforms QS, primarily due to QS generating many segments, while DSEG inherently focuses on meaningful segments aligned with the underlying foundation model.

We hope that we have clarified your question further and would be very pleased to receive a revised evaluation.

[1] Bora, Revoti Prasad, et al. "SLICE: Stabilized LIME for Consistent Explanations for Image Classification." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

A revised version of the manuscript, incorporating the feedback marked with (*) in the text, will be uploaded next week. Any additional changes will be included in the final camera-ready version.

We thank the reviewer for acknowledging the importance of addressing LIME’s stability concerns and evaluating it rigorously through quantitative and qualitative methods. Additionally, integrating our proposed method, DSEG, with LIME and its variants demonstrates the versatility of our approach across a variety of models.

Point-by-Point Response

[...] revised as DLIME [...] also discusses using hierarchical clustering with LIME.[...]

DLIME is indeed a relevant work that we will integrate into the manuscript accordingly (*). However, in contrast to DLIME, DSEG-LIME operates on the image modality. In addition, our hierarchy directly relates to the composition of image segments, providing LIME with a dynamic and adaptable approach. In contrast, DLIME employs agglomerative hierarchical clustering to group training data, and focusing on tabular datasets. The rational by focusing on segmentation for stability is finding the model's decision boundry closer than other segmentation methods would do. Compared to e.g., Slice handling stability with entropy calculation.

[...] seems to stabilize LIME explanations without investigating into the segmentation aspect. It would benefit readers [...] complements or differs from stabilization methods [...]

We developed DSEG-LIME because many LIME-based approaches, including methods like SLICE, do not specifically investigate the stability of explanations through the segmentation process itself. This is a critical aspect that we believe requires closer examination. Therefore, we created DSEG-LIME, building on the principles outlined in [1], to provide a more detailed and systematic analysis of how segmentation impacts the stability of LIME explanations. However, we understand this point and will include a comparison of the methods (*).

[...] mean score of 8 runs? To improve clarity, the authors should provide more details on how they calculated metrics across multiple runs, including whether they used mean scores and how they handled any inconsistencies.

For other metrics except of stability, we present results based on one run, thus not taking the mean, due to runtime of all results for all models and LIME combinations.

[...] why BayLIME with non-informative prior was chosen. [...]

We fully agree that BayLIME with non-informative prior is nearly indistinguishable. Our rationale behind this evaluation was to stick to a setting, where we do not build any priors into the process. Since we also integrate DSEG within BayLIME, there was no disadvantage regarding this method and ensured an equal comparison against other LIME backbones.

[...] dataset consists of 50 images, which may not be sufficient. Also, more than the number of runs (i.e., 8) may be needed for the Rep. Stability test. [...]

We use 50 images, like related approaches [2]. After consideration and discussion with other authors, we acknolwedge that evaluating more than 50 images, and increasing it with other works to 100 images makes our approach more robust. Regarding qualitative evaluation, we stick to random 20 images to have a greater audience of evaluating the explanations than having only a few participant evaluating the whole 50 (or in the future 100) images. For the user study, we used t-statistics and p-values to evaluate user performance and preferences, as this setting involved comparing groups and drawing conclusions about their significance. However, these metrics were not applied in the quantitative evaluation because the focus there was on measuring performance metrics rather than testing differences between groups, but we will add this information with the increased dataset. To clarify, our alternative hypothesis is as follows: DSEG performs significantly better than other segmentation methods within the LIME framework in terms of preference by the participants, when applied to the dataset with the selection of five explanations. (*)

1) Author's comments: "We developed DSEG-LIME because many LIME-based approaches, including methods like SLICE, do not specifically investigate the stability of explanations through the segmentation process itself. This is a critical aspect that we believe requires closer examination. Therefore, we created DSEG-LIME, building on the principles outlined in [1], to provide a more detailed and systematic analysis of how segmentation impacts the stability of LIME explanations. However, we understand this point and will include a comparison of the methods (*)."

Further Question: SLICE stabilizes LIME explanations without investigating the segmentation aspect, but I did not find any reason why it cannot be compared with the proposed method. I think the authors need not run additional experiments for this but can clarify their reason for not comparing with SLICE or other such methods.

2) Author's comments: "For the user study, we used t-statistics and p-values to evaluate user performance and preferences, as this setting involved comparing groups and drawing conclusions about their significance."

Further Question: What is the rationale behind the choice of a parametric test like the t-test? How did the authors verify that the data follows the assumptions made by the test?

3) Author's comments: "However, these metrics were not applied in the quantitative evaluation because the focus there was on measuring performance metrics rather than testing differences between groups."

Further Question: I request the authors to explain this statement.

4) Author's comments: "We use 50 images, like related approaches [2]. After consideration and discussion with other authors, we acknowledge that evaluating more than 50 images, and increasing it with other works to 100 images makes our approach more robust."

Further Question: My concern was not just the number of images but also the number of runs. I request the authors to explain how running the proposed method only 8 times can help in robust estimation of Rep. Stability.

A revised version of the manuscript, incorporating the feedback marked with (*) in the text, will be uploaded next week. Any additional changes will be included in the final camera-ready version.

We appreciate the reviewer’s feedback and for recognizing the strengths of our approach, including its ability to tackle real-world interpretability challenges and enhance explainability through SAM’s hierarchical segmentation. The user study and empirical results further highlight the method’s effectiveness in generating interpretable and faithful explanations.

Point-by-Point Response

Does SAM consider an image as an input or extracted features from a classifier as an input, figure 2 points at extracted features, while equation 3 points at input image?

We agree with your observation regarding the input image processing in Figure 2. We will revise the figure to better reflect the workflow and input structure to ensure clarity and accuracy (*).

Given the above statement, the conducted human evaluation states SAM’s capabilities to capture human-recognisable features and not the classifiers perception of these concepts, I would like to know authors thoughts on this?

We integrate SAM and other segmentation models in a manner analogous to techniques like SLIC. Specifically, the final masks used by LIME (after removing empty spaces) correspond to superpixels formatted similarly to those generated by SLIC. This ensures consistency in the pipeline and comparability across methods. We understand the confusion here and will clarify it in Section (5.3) (*).

In our qualitative evaluation, we compare explanations generated using different backbones and assess human subjects' preferences for specific classifications. The results demonstrate that SAM’s segmentation tendencies—which align with human-recognizable objects and classes—produce explanations that are preferred by users. This outcome underscores SAM’s ability to generate features that resonate with human intuition, enhancing the interpretability of the explanations.

Additionally, DSEG offsets computational cost of extracting explanations as compared to LIME

Regarding computation time, we acknowledge the trade-off but emphasize that the improved quality of the results, both qualitatively and quantitatively, justifies and offsets the associated costs. The higher interpretability and fidelity of explanations lead to tangible benefits, including better alignment with human understanding, enhanced usability, and greater trust in the model outputs. We believe, these benefits validate the additional computational investment.

I would like to know authors view-point on explaining one deep-learning model using another much deeper more complex model?

We recognize the concern regarding the interpretability of deep models when utilizing even larger foundation models for their explanation. While this shifts the problem to a more complex model, our work, along with other studies, demonstrates that large foundation models offer unique advantages in interpreting smaller models [1, 2, 3,4]. These examples show that leveraging the hierarchical and semantic understanding of large models can meaningfully address interpretability challenges. Furthermore, since our goal is to explain the local behavior of models like EfficientNet around a specific instance, the trustworthiness of the explanation remains intact

The output from the foundation model isn’t always human-interpretable, highlighting the need for more quantitative studies to evaluate interpretability in greater depth.

We acknowledge that the outputs of foundation models can sometimes lack direct human interpretability. This reinforces the importance of robust evaluation methodologies that go beyond subjective human assessments. In response, we conducted a comprehensive quantitative evaluation to complement qualitative assessments. This ensures that our findings are not only based on subjective human preferences but are also supported by objective, data-driven insights.

However, we agree that evaluating segmentation interpretability requires further exploration. Future work should investigate the qualitative segmentation performance of such models to deepen our understanding of their interpretability capabilities.