Benchmarking XAI Explanations with Human-Aligned Evaluations

We appreciate the reviewer’s comments and will address them in the following seven points:

1. Process of Generating Computed Explanations and Annotating Concepts (W1):

We thank the reviewer for raising this point and have revised the manuscript to provide a clearer explanation of how computed explanations are generated and concepts are annotated. Specifically, we have included details about the pipeline for generating explanations, the criteria for selecting annotators, and the methodology for annotating concepts. Additionally, we have added a visual illustration in the main paper to improve clarity and facilitate understanding.

2. Applicability of PASTA to Unseen Datasets or XAI Methods (W1):

We acknowledge the importance of validating PASTA's generalization to other contexts and modalities. As shown in Table 3, PASTA has been applied to unseen images from the datasets, demonstrating its adaptability. However, we emphasize that the primary goal of this work is to provide a benchmark that enables fair and consistent comparisons of XAI methods. While the diversity of datasets and methods in the current version is limited, PASTA represents a significant advancement over prior works by introducing a larger number of methods and a greater total number of samples. To offer a more comprehensive perspective, we have added a comparison with existing benchmarks in the appendix (Section: Comparison with Existing Benchmarks)

The table below provides an overview of datasets and human evaluation frameworks for XAI methods. Name specifies the reference of the dataset utilized in each study. Annotations describe the type of labels used in the dataset. N_{Samples} indicates the total number of samples that make up the dataset, while N_{Part} represents the number of participants involved in the labeling process. The column Modality identifies the types of data the dataset includes: I refers to images, C to concepts, and T to text. N_{Q} denotes the number of distinct questions posed to the annotators during the study. N_{XAI} refers to the number of XAI methods tested within the experiments, with No indicating cases where the dataset was used to label ground truth explanations without direct comparison to XAI methods. Finally, N_{Data} represents the number of unique data samples (e.g., images) shown to annotators during the experiments. Regarding the types of human annotations used in existing studies (as shown in the "Annotations" column of the table), Likert refers to the use of a Likert scale for scoring, Saliency refers to pseudo-saliency map evaluations, 2AFC indicates two-alternative forced choices, Clicktionary corresponds to the click-based annotation game defined in [4], MCQ represents multiple-choice questions, and Binary refers to binary decision tasks. Our novelty lies in the larger volume of annotations compared to previous approaches, the large number of XAI techniques, and the use of transparent, post-hoc methods.

3. Clarity on Metric Calculations and Visual Aids (W2):

To address the reviewer’s concerns, we have added new figures and equations in Section B.2 (Comparison with Existing Metrics), including formulas and visual aids to illustrate the process of computing our scores. These additions aim to improve the clarity and accessibility of the metric explanations.

We sincerely thank the reviewer for their comments and questions, which will help us strengthen the paper.

1. Methodological Concerns and Scope + model development (W1) + Q2

We appreciate the reviewer’s insightful feedback and agree that the ultimate goal of our work is to enhance the quality of future XAI systems. To clarify, the primary aim of our work is to propose a perceptual criterion for XAI evaluation. This criterion allows researchers to assess the quality of their XAI techniques within our benchmark, making it easier to determine if an algorithm is effective. Our approach is inspired by advancements in the image quality assessment community. For many years, this field relied solely on human annotators to evaluate image quality until methods like LPIPS [1] introduced a standardized way to perform such assessments.
While we agree with the reviewer that our criterion could also be used to improve XAI techniques, we chose to leave this application for future work due to the extensive scope of this paper. There are potential challenges in using the same criterion for both improvement and evaluation—if we refine a technique based on this criterion, how do we objectively assess it afterward?

The PASTA benchmark and dataset currently deals with saliency-based, counterfactual-based, and concept-based explanation methods. Since the human evaluation protocol is normalized and has been developed with the help of a psychologist, it is reproducible and can be extended to new modalities with new annotators. This is why we intend to open source the data and the protocol is carefully described.

We acknowledge the reviewer’s suggestion to elaborate further on the experiments of PASTA. In the revised version of the paper, we plan to expand this section and include the following table (currently found in 3.) to provide additional context.

2. Depth Analysis of the results and automated rating part (W2)

2.1: It is true that the results of the benchmark of Section 3.5 are relatively short. We have added some more comments in this Section. However, due to space limitations, extensive comments on the results of the benchmark are provided in Appendices B.2 and B.3. Concretely we discussed more the interpretations of why saliency-based methods seem to perform better, and a note about the higher performances of class independent explanations like EigenCAM.

2.2: Concerning the close-to-zero correlations between human ratings and existing XAI metrics, our interpretation, although quite short, is that human scores cover an aspect of explanation quality unrelated to that of perceptual quality, as previously noted by Biessmann & Refiano (2021). Indeed, on the one hand, human evaluations are most likely to accurately measure the usefulness of an explanation, which is ultimately targeted at a human audience. On the other hand, other computational metrics, such as faithfulness, are more likely to measure the adequation between the explanation and the real functioning of the model. This aspect is inherently inaccessible to a human judgment, as the human cannot access the internal functioning of the model. We have added a clarification on this in Section 3.6.

We would like to thank reviewer cj8g for their thoughtful comments and suggestions which helped us improve the clarity and thoroughness of the paper. Below, we give an answer to each indicated weakness and question.

1. Annotator Demographics and Diversity (W1)

The creation of the benchmark has involved 15 human subjects, each of whom has annotated a part of the whole dataset (roughly one third). For each sample of the dataset, 5 different annotators among the 15 have made an annotation. The precise demographics of the annotators are described in Appendix A.3. In particular, the genders of the annotators were willingly chosen to be balanced.

2. Sample Size of the Benchmark (W2)

We would like to clarify an important point. While traditional datasets are often evaluated based solely on the number of images, our benchmark incorporates two additional dimensions: the number of annotators and the number of XAI techniques. Moreover, we account for the number of questions, which directly impacts the total number of annotations per data point. Consequently, although our benchmark includes only 100 images sourced from 4 different datasets, it features 5 independent answers to 6 distinct questions across 22 XAI techniques. This results in a total of 66,000 human annotations. This annotation scale is comparable to that of MMbench [1], which involves approximately 3,000 images.

Rather than increasing the number of images, we prioritized a comprehensive evaluation of XAI techniques and coverage of 6 questions addressing different XAI axioms—an original contribution to the literature. To provide a clearer comparison with existing benchmarks, we have included a detailed analysis in the appendix (Section: Comparison with Existing Benchmarks). This analysis lists various XAI benchmarks [4-11] present in the literature. The following table demonstrates that our benchmark is comparable to these existing benchmarks:

Regarding the types of human annotations used in existing studies (as shown in the "Annotations" column of the table), Likert refers to the use of a Likert scale for scoring, Saliency refers to pseudo-saliency map evaluations, 2AFC indicates two-alternative forced choices, Clicktionary corresponds to the click-based annotation game defined in [4], MCQ represents multiple-choice questions, and Binary refers to binary decision tasks. Our novelty lies in the larger volume of annotations compared to previous approaches, the large number of XAI techniques, and the use of transparent, post-hoc methods.

3. Data Splitting and Generalization (W3)

As explained in Sec. 4.4, in the experiment, if no split restriction is given, it is possible that the same images with different XAI techniques, or the same XAI techniques applied to different images could be found in the training and test set. As can be seen in Table 4, adding a split restriction to avoid this has a small, but non-zero effect on the accuracy of the metric. To simplify this, we have chosen in the updated version to remove the ‘’no restriction” and only consider the restriction about images in all our experiments. As such, all the values in Section 4.3 and Appendix C have been replaced. The observations remain the same.

4. Choice of Grounding DINO Technique (W4)

To generate bounding boxes, among the different available solutions, Grounding DINO is one of the best methods [2]. In addition, since the Cats Dogs Cars dataset is of moderate size, we have manually checked the correctness of the bounding boxes produced by Grounding DINO and found no significant errors.

Dear reviewer cj8g,

Thank you for serving as a reviewer for our paper. We wanted to kindly remind you that it has been two weeks since we submitted our response and we have not yet received any feedback.

We value your insights and suggestions. If there are any additional points you would like us to clarify or discuss, please do not hesitate to let us know.

Best regards,

We would like to thank reviewer cta3 for his or her questions on the adequation between the deep learning model and the XAI methods, and on the dependence of our new metric on CLIP.

1. Methodology and Architecture Dependency (W1)

It is true that many XAI methods were originally designed for specific deep convolutional neural network architectures and may produce suboptimal results when applied to different models. However, we believe it is essential to evaluate how various XAI techniques perform across multiple backbones to ensure a fairer and more comprehensive assessment. This is precisely why we tested saliency-based methods (such as GradCAM and its variants) on different models, referred to as “backbones” in the paper. To this end, we collected human evaluations of 47 XAI methods across specific backbones, though we focused on 21 distinct XAI techniques in our analysis. For instance, GradCAM was tested on ResNet50, ViT-B, and CLIP zero-shot (which is based on ViT but uses a different training process). Similarly, SHAP-CBM was evaluated with both CLIP-QDA and the original Concept Bottleneck architecture proposed by Koh et al. [1]. We have added a paragraph clarifying this point in Section 3.2. Moreover, the results in Section 3.5 highlight a potential bias in the design of certain XAI methods, particularly favoring ResNet50.

2. Limitations of Using CLIP (W2)

First, we would like to insist on the fact that CLIP is only used as a tool to build the PASTA metric, which extends the results of the benchmark to potential new XAI methods. The PASTA benchmark itself is independent of CLIP. For the PASTA metric, it indeed relies on CLIP. To mitigate the dependence, we have added two new experiments:

• A PASTA metric similar to the CLIP-based one, where CLIP is replaced by LLaVa as a text-image encoder. The results are shown in Section 4.3.
• An alternative PASTA metric that does not depend on a neural network to align image and text embeddings. It is a low-dimensional embedding based on several scores that can be computed both on saliency maps and concept importances. The details of these metrics have been added as Appendix D, and the results are shown in Section 4.3.

However, it should be noted that this alternative metric is more computationally expensive, as some scores (e.g., Classification Criterion or Variance Criterion) require a lot of computations.

This table presents Mean Square Error (MSE), Quadratic Weighted Kappa (QWK), and Spearman Correlation Coefficient (SCC) metrics for several strategies to compute scores obtained using CLIP and LLaVa [2] as multimodal encoders, referred to as $\text{PASTA-metric}^{\text{CLIP}}$ and $\text{PASTA-metric}^{\text{LLaVa}}$, respectively. We also tested an alternative approach involving handcrafted feature extraction, as described in Appendix D. Human refers to inter-annotator agreement metrics.

3. Suitability for Conference Venue (W3)

We respectfully disagree with the reviewer’s comment. Our manuscript directly addresses topics explicitly listed in the ICLR 2025 Call for Papers, namely “datasets and benchmarks” (as we propose a benchmark and release a dataset) and “societal considerations” (since explainability methods and their human evaluation contribute to enhancing the safety and fairness of machine learning systems). For these reasons, we believe our paper is well-suited for the conference. Additionally, this is not a user study for two key reasons: (1) annotators are not users, as they are not interacting with or using a system, and (2) the objective is not to interpret human annotations but rather to assess the performance of XAI techniques.

References

[1] Koh, Pang Wei, et al. "Concept bottleneck models." International conference on machine learning. PMLR, 2020.

[2] Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2024). Visual instruction tuning. Advances in neural information processing systems, 36.

First and foremost, I would like to thank the authors for the reply on the reviews. One of my major concerns for this method is the dependency on the CLIP model (W2). The authors have replaced it with LLaVa, the problem is it will still be dependent on the internal architecture of the model, in this case LLaVa. The response to W1 is not satisfactory. But I appreciate the authors for the additional results they put in the paper. I would tend to increase my score from 3 to 5.

Dear AC and Reviewers,

Following the plan outlined in the previous post, we have implemented the following changes:

• Added examples of both high- and low-rated explanations, showcasing human annotations alongside PASTA-metric ratings (Appendix E).
• Included more detailed examples in the main text to provide better context before referencing supplementary sections.
• Corrected typos throughout the manuscript.
• The typo in Figure 8 will be corrected tomorrow.

Thank you for your feedback and guidance.

Dear Reviewers,

We hope this message finds you well.

We have revised our manuscript to address the concerns and suggestions you raised. We greatly value your expertise and would highly appreciate it if you could review the updated manuscript and share your thoughts on whether these revisions satisfactorily address your concerns. We sincerely thank the reviewers for their efforts and would greatly appreciate it if they could begin engaging in a discussion with us. Please feel free to reach out if you have any further questions or require additional clarification.

Best regards,

The corresponding authors

We sincerely thank reviewers cj8g, 1rMP, cta3, and 2wcf for their insightful comments and suggestions. In addition to the revisions detailed in our previous responses (and based on your feedback), we have conducted new experiments to investigate potential applications of the PASTA-score, inspired by the reviewers' suggestions. Specifically, we added the following experiments in Section E.2:

• Experiment 1: Using the PASTA-score to automate the selection of the optimal kernel width for the exponential kernel used in blurring images during the LIME perturbation set optimization process.
• Experiment 2: Identifying among GradCAM explanations on different layers of ResNet-50, the one with the highest PASTA-score and evaluating whether the resulting explanations align with expectations.

Additionally, we have expanded the conclusion to outline further avenues for future research.

Once again, we sincerely thank you for your constructive feedback and would be happy to continue the discussion with you.

best regards

As the rebuttal period comes to a close, we would like to provide a comprehensive summary of the changes made to our manuscript based on the insightful feedback from the reviewers. We remain open to further discussions should there be any additional issues.

Changes in the revised manuscript are marked in purple for clarity.

Precision About Contributions to the State of the Art

One key concern was clarifying the contribution of the PASTA-dataset to the field of perceptual evaluation of XAI methods. To address this:
We added in Section B.1 a detailed comparison between the PASTA-dataset and existing human-aligned evaluation benchmarks. The PASTA-dataset is, to our knowledge, the first dataset dedicated to perceptual evaluation of XAI methods that integrates both saliency and concept-based explanations. It combines 2,200 samples from 21 XAI methods and incorporates 6 questions carefully crafted with input from a psychologist to assess perceptual quality. We also emphasize the importance of collecting 5 annotations per sample, resulting in 66,000 Likert annotations, far exceeding the scale of existing works. The choice of multiple annotations per sample, despite being costly, seems relevant considering the high variance inherent to perceptual assessment, as underlined in Section B.3.

Clarity Issues

Several points in the original manuscript were unclear, and we made the following improvements:

• Section 3.6: We clarified our interpretation of the lack of correlation between PASTA-dataset annotations and existing computational metrics, we have placed more emphasis on the fact that we consider that human evaluations and computational metrics should be viewed as complementary aspects of the evaluation process.

• Sections 3.5 and 3.6: We expanded the analysis of results to address the reviewers’ request for a deeper interpretation.

• Section 3.2: Clarified why and how we tested XAI methods across multiple backbones.

• Corrected typos.

• Added more detailed examples in the main text for better context before referencing supplementary sections.

• Section B.2: Additional formulas and figures were included to illustrate the process of computing the PASTA-scores more clearly.

Details About the PASTA-Dataset Process

Thanks to reviewers suggestions, we provided more detailed information about the dataset creation:

• Section A.1.1: Clarified the classes and concepts used for training inference models and included a table of labels. Note that the same images and labels are used across all models.
• Section A.4 In response to a reviewer’s observation about the importance of perturbations in Q5 and Q6, we added a detailed list of perturbations and their usage.

Testing Variants of the PASTA-Metric

To address concerns about the robustness of the PASTA-metric, we conducted additional experiments:

1. Bias From CLIP Encoding: In section Section 4.3, we added experiments to investigate the potential bias introduced by using CLIP as a multimodal encoder.

   • Study the replacement of CLIP with another multimodal encoder, LLaVa; results were similar.
   • Proposed an alternative using handcrafted features in Section D, addressing concerns about potential biases of DNN-based encoders.

2. Scoring Network Alternatives: Also in Section D, we tested Ridge Regression, Lasso Regression, Support Vector Machines, and a Multi-Layer Perceptron. Then, we demonstrated that our proposed network, with ranking and cosine similarity losses, performed better.

3. Impact of Label Information: In Section C.3: We tested the effect of adding label information to the PASTA-metric network. Results showed no improvement, likely due to the large number of classes.

4. Bias in Data Splits: For all of our experiments, we Re-ran computations to ensure no overlap between samples from the same image but different XAI methods in training, validation, and test splits. Updated values are consistent with previous observations.