EXAGREE: Towards Explanation Agreement in Explainable Machine Learning

We thank reviewer for recognition of the work as interesting. We clarify several points that may have led to misunderstandings below. Our goal with EXAGREE is not to present a comprehensive solution to explanation disagreement. Rather, we aim to provide a framework that begins to reduce explanation disagreement. We answer your major concerns first.

Major

[Major] In Section 2 (Preliminaries), there are several mistakes in the formulation: a) M* is not defined in Eq. 1. I assume what you mean is the optimal model by M* (the model with the best performance on the data). b) a_1, a_2,...,a_p are not defined as features. c) In Equation 1, I think the sign should be >= instead of <= (the loss of the other models will be higher than M* (assuming M* refers to optimal model) d) Why is ϵ multiplied by L(M*), usually similar performing models are supposed to be an absolute threshold, not relative to the loss of the original model.

We respectfully disagree with some of the points raised and appreciate the opportunity to clarify:

• a) We apologize for this oversight and It indeed represents the optimal model.
• b) The variables $a_1, a_2, ...a_p$​ represent feature attributions, not the features themselves.
• c) The sign in Equation 1, ≤, is correct. It indicates that all models in the Rashomon set perform within a specified threshold relative to $M^{*}$.
• d) The threshold $\epsilon$ can indeed be defined as either a relative or absolute value. Both approaches are valid, and more details can be found in cited works.

3. [Most Major] In Section 2.1, you mention 4 axes of explanation disagreement. I am with you on the model and method disagreement, but what does stakeholder and ground truth disagreement mean?

a) In Stakeholder disagreement you mention "Different stakeholders in S might prefer different rankings". Why would that ever happen? Do these stakeholders want something that is not real and want fake explanations that aligns with their mental expectations of what a model should care about? What is an example of such stakeholders wanting something different from reality? Your current example of a data scientist wanting statistical significance and domain expert valuing different features does not answer this. Does wanting statistical significance and valuing different features amount to wanting different ranking of features (this also assumes that these rankings won't be same in the first place)? And even if they want different rankings, should a technique be optimized to suit their demands and provide fake explanations?

Response: Thank you for raising this point. The stakeholder disagreement emerges not because stakeholders want "fake" explanations, but because different roles require different aspects for their specific tasks, while respecting medical constraints.

Consider a more concrete breast cancer diagnostic model in a healthcare setting, with real-world diagnostic features, such as mammogram findings (e.g., microcalcifications, mass shape/margins), ultrasound results (e.g., solid vs. cystic masses), biopsy cell characteristics (e.g., nuclear grade, mitotic count), tumor size, age at diagnosis, family history, and breast density. The model developer considers all features to optimize performance, while other stakeholders need explanations that emphasize features relevant to their decision-making stage:

1. Radiologists primarily use imaging features during initial screenings. If the model's explanation emphasizes biopsy results (which aren't available during screening), this creates an operational mismatch - not because the biopsy features aren't important, but because radiologists need to understand the model's reasoning based on imaging data available at their decision point.

2. Pathologists focus on cellular characteristics from biopsies, such as nuclear grade and mitotic count, which are essential for assessing malignancy at the cellular level. Conflicts arise if the model emphasizes imaging features over biopsy data, as imaging cannot provide the cellular detail pathologists need.

3. Primary Care Physicians make initial referral decisions using family history, age, and previous records. They need explanations focused on these early-stage risk indicators rather than specialized imaging or biopsy findings.

These stakeholders aren't seeking incorrect explanations - they need explanations that reflect how the model uses features accessible for their roles, making the model more actionable and useful. This stakeholder disagreement represents just one of the four fundamental sources and the other three sources all contribute to the broader challenge in practice. We will explain the ground truth disagreement below.

[Major] The above were the problems with the problem setup, now coming to the experiment section.
a) In table 1 you mention k= 0.25, what does k stand for? You have used k and l in Section 2.1 without ever defining them. For the same reason, I do not comprehend what does the k in caption of Figure 3 stand for. b) Why is SAEM technique bolded in Tables 1 and 2. The standard practice in ML papers to the bold the best performing method, but your technique, and this is misleading. c) If SAEM selects the models with the highest agreement with the stakeholder, why is it not the best for LR in Table 1 and 2? (especially in Table 2, it is pretty behind techniques that are not optimized for agreement, which alludes to the case that explanation disagreement might not even show up in actual experiments?) d) how is FIS_LR computed? Is this not just the feature importance and since you are using ground-truth explanations from pre-trained LR, this should have 100% agreement and should be the best 8 number of times (currently in Table 1 it is best 0 times)?

a) We apologize for this oversight. $k$ represents the top percentage of features considered. For example, with $k=0.25$ in a list of 12 features, we consider the top 3 most important features.

b) and c): As the first framework specifically targeting explanation agreement improvement, our experimental setup focuses on demonstrating relative improvements. This is why we bolded the improved metrics. This does not mean our method is always the best performing method compared to any explanation method in explaining any model.

We acknowledge that our framework cannot satisfy all metrics for all stakeholders due to performance constraints. As formally proven in Section D and Appendix "Constrained Sorting and Ranking in a Rashomon Set". Regarding Table 2, gradient-based methods achieve perfect agreement for LR models because they directly compute derivatives that match ground truth coefficients. However, these same methods show significantly lower agreement when explaining ANN models. This highlights a key limitation - while existing methods may perform well for specific model types, we provide more consistent agreement improvements across different model architectures and stakeholder needs.

d) FIS_LR is computed based on the loss change, described in cited works. The definition is referred as $\varphi_{i}(**X**, {y})$ = $L_{emp}$($f(\{X\}\_{\setminus i}, **y**)$ - $L_{emp}(f(**X**), **y**)$ for feature importance of feature $i$, where $**X**_{\setminus i}$ is the input when the feature of interest is replaced by an independent variable.

As acknowledged in Section 3.1, this is a model-agnostic post-hoc explanation method that doesn't guarantee perfect agreement with ground truth. This is similar to other established methods like SHAP, which shows 50% agreement in our experiments but the method remains widely used and trusted in practice.

Minor Questions

Starting with Figure 1 on Page 1 -- totally unclear. How does the right side of the figure indicate agreement while the left does not? (this figure is not crucial to understand the paper, but I am just making a point of why the paper is so unclearly written and you do not need to address this point in the rebuttal, there are many more important problems. )

We will either improve or remove this figure.

Section 2.3, I think the title of that subsection should be "Evaluation Metrics" and not "Evaluation Matrices". Anyway, in that section you mention two metrics: faithfulness assessment and fairness assessment. In faithful assessment you give a further breakdown in several metrics -- however, most of these metrics seems highly correlated, for e.g., the attribution values directly affect the ranking, so FA, RA, RC, SA, SRA, PRA should have extreme correlation (when you consider the attribution values with its sign). Can you measure this please? If they are highly correlated, then that is effectively one metric and not six. If they are not correlated, can you explain why that is not the case (except the case where you consider the absolute values, its obvious then).

Thank you for your suggestion and the title will be corrected.

We agree that FA, RA, RC, SA, SRA, PRA are correlated. It's important to note that we did not define these faithfulness metrics but followed the work (OpenXAI https://arxiv.org/pdf/2206.11104).

Thank you for all your feedback.

I appreciate the time taken by the authors to read and give a response to my review. I apologize for the lateness in my response. I have follow-up questions that I am listing:

1. In Equation 1, I think the sign should be >= instead of <= (the loss of the other models will be higher than M* (assuming M* refers to optimal model) Authors said: The sign in Equation 1, ≤, is correct. It indicates that all models in the Rashomon set perform within a specified threshold relative to M*.

If M* is the optimal model, then how can the loss of any other model be lower than that and therefore I thought the sign should be reversed. Please correct me if I am wrong.

2. Authors said: "These stakeholders aren't seeking incorrect explanations - they need explanations that reflect how the model uses features accessible for their roles, making the model more actionable and useful."

The example you give is still not convincing to me. There are radiologists, pathologists, and PCPs in this setup and they want explanations from the model which are accessible for their roles, I can understand that. There are a few options they can use: a) build separate models using the features that are accessible to each of the people. Now I know, you might think this will give a low-performing model than a model trained on all features, in which case: b) just get the explanations from the model trained on features, rank them based on importance and show the relevant groups of features accessible to each person.

Is this not an easier and effective solution? If not, then please let me know why.

3. Authors said: Consider the healthcare scenario again, where the decision tree's interpretation could be problematic despite high accuracy: a decision tree flags small cell size (<4mm) more important than larger cell sizes (>4mm), contradicting established medical knowledge where larger cell normally indicates positive prediction.

If such a situation arises, we don't use such a model, because we know it is wrong and dangerous to use such a model. Since it is an interpretable model, we know exactly how the model is making the decision. And if it is totally opposite of how biology works, we should not use such a model. (If you disagree, please give me an argument about why we should use continue using such a model, just because its accurate on a dataset the model was trained on or a related test dataset?) Therefore I don't see any argument for ground-truth disagreement.

4. We hope the examples above help clarify the role of stakeholder-specific expectations in explanation disagreements. Thank you for the opportunity to clarify.

Not yet unfortunately.

5. The "mask" (line 256) refers to the characterization of models in the Rashomon set

What does this line mean? Please use the terms that are defined in the paper. After reading your rebuttal it was much more clear what the job of DMAN is, but because of missing definitions when a reviewer is reading your paper it completely derails them. The same thing happened in the experimental section with missing "k". Also, you only answered my question about DMAN, please also explain the role of SAEM, Multi-heads architecture in Section 3.1.

6. As the first framework specifically targeting explanation agreement improvement, our experimental setup focuses on demonstrating relative improvements. This is why we bolded the improved metrics.

No, you bolded the row for your technique, not the "improved metrics" -- in which case you would have bolded the best performing technique per column.

The paper’s formalization of four types of disagreements lacks sufficient motivation and clarity, especially in comparison to the foundational work it draws from. For example, the classification of "model disagreement" as a type of "explanation disagreement" is unclear - different models that produce the same predictions can indeed have different internal mechanisms of doing so - in this case explanations should disagree and illuminate this fact rather than obscure it... There is no "model disagreement" problem.

Response: We appreciate the reviewer’s feedback regarding the motivation behind the different types of disagreement in our paper. We would like to clarify the following points:

"Model disagreement" is not a problem; rather, it represents a source of potentially conflicting explanations, consistent with the Rashomon set concept, which in turn leads to explanation disagreement. Similarly, this perspective applies across all four types we present in the paper. Explanation disagreement becomes problematic when these differences are unrecognized and explanations are trusted blindly, like in the criminal justice case.

We are with the reviewer on this perspective and indeed, we take advantage of model disagreement as a means to reduce explanation disagreement, where our aim is to identify a model from a Rashomon set that provides personalized explanations based on their specific needs and preferences.

The paper's contributions then are better studied and understood in the context of a related line of inquiry about adversarial attacks and explanation fairwashing, such as Slack et al. The primary problem addressed appears to involve modifying the explanations produced by a Rashomon set of models to align with a predefined set of explanations from an external oracle.

Response: We sincerely thank the reviewer for recognizing the experimental design and for your insightful understanding of the use of LR coefficients as ground truth for the ANN model. This was indeed the approach we intended, and we appreciate the clarity of your observation.

We agree that our work shares conceptual space with research on adversarial attacks and explanation fairwashing. The intersection between adversarial attacks and the Rashomon set concept is an interesting direction for further exploration.

Finally, the paper’s entire framing around "explanation agreement" could be made clearer. Rather than resolving "model disagreement," the proposed SAEM approach seems to modify model explanations without altering predictive accuracy, which could be viewed as a form of adversarial attack on explanations. I encourage the authors to address how this approach contrasts with adversarial manipulations of explanations (if at all), discuss potential connections to explanation fairwashing, and consider any ethical implications that arise from intentionally adjusting explanations while maintaining predictive outputs. What is presented in this paper as an SAEM to resolve the apparent "model disagreement" class of explanation disagreement problems is essentially a means to make the FIS score for the ANN model to match the coefficients from the LR model trained on the same data - this is an adversarial attack.

Response: We agree that there are conceptual similarities between our approach and adversarial attacks, particularly in the idea of modifying model explanations. However, the objectives, methods, and context of our work are distinct.

Unlike adversarial attacks designed to exploit vulnerabilities in explanation methods, our aim is to systematically explore and improve model explanations to better align with stakeholder needs. Certainly, we believe that similar ideas from adversarial attack research can be applied in this context to address explanation disagreement.

While adversarial attacks typically operate by perturbing inputs to manipulate explanations while maintaining potentially biased behavior, our method is built on the Rashomon set framework. We're not modifying/fooling a single model's explanations, but rather exploring the model space and identifying models that provide desired explanations. It is noted that a Rashomon set can contain models of different architectures, e.g., a tree or a network, not just wrapper modifications. Our mask-based sampling method represents one approach to exploring the space. Interestingly, scaffolded models produced by adversarial attacks could theoretically exist within the Rashomon set if they meet the performance condition. Further exploration of this intersection could be a valuable direction, though it is outside the scope of our current work.

Finally, while we acknowledge that any technique modifying explanations could potentially be misused, EXAGREE's optimization within the Rashomon set represents a different perspective which helps users understand the range of feature attributions (attribution sets) instead of a single ranking.

My biggest concern is that the paper uses local explanatory models (i.e. LIME, Integrated Gradients) to generate global (model-level) explanations. In Section 2.3, the authors mention that they have adapted feature attribution methods for local explanations "by averaging feature attributions across all instances to obtain global attributions." These methods were not designed to be used this way. Although SHAP does have functionality to provide model-level feature attribution, it takes an average of the absolute attribution across instances.

Reponse: We appreciate the reviewer's attention to methodological rigor. However, we must respectfully disagree with the concern about aggregating local explanations to obtain global feature importance. This approach has solid theoretical and empirical foundations in the XAI, as demonstrated by following studies:

• Section 4 in Why Should I Trust You?” Explaining the Predictions of Any Classifier
• Section 4 in A Unified Approach to Interpreting Model Predictions
• Section B.2 in Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
• Section 2 in Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Global Explanations of Neural Network Mapping the Landscape of Predictions
• Chapter 3 in book Interpretable Machine Learning - A Guide for Making Black Box Models Explainable
• COMPAS experiment in Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
• Section 3 and 4 in Fairness via explanation quality: Evaluating disparities in the quality of post hoc explanations

Moreimportantly, LIME, SHAP, and other local explanation methods are used ONLY as baseline comparisons, following the exact evaluation protocol from the OpenXAI benchmark paper. They are not components of EXAGREE and our contributions stand entirely independent of these comparison methods.

My impression of the paper's proposed solution, EXAGREE, is that it attempts to address "explanation disagreement" by finding a model that aligns with stakeholder expectations (i.e., based on domain knowledge) through examining its post-hoc explanation... Besides, I don't think the solution addresses the problem of "explanation disagreement", but is rather a model-selection tool using post-hoc explanations. I see that there are two cases of "explanation disagreement" (both of which is mentioned in the paper): 1. Models with similar performance give different explanations (explanation method fixed) 2. Explanation methods provide different explanations for one model (model fixed) EXAGREE addresses 1 to an extent but not 2 --- the paper does not make this clear. The authors seem to suggest that the "stakeholder centric" approach can address complex disagreements (Section 2.2). But I don't see how it addresses case 2.

Reponse: Our intention is not to fully resolve the complex issue of explanation disagreement, but rather to contribute a step toward addressing this challenge. We identify four potential sources of disagreement: model disagreement, method disagreement, stakeholder disagreement, and ground truth disagreement. Given the inherent complexity of these sources, we view the problem as an ongoing research question that warrants further exploration.

We acknowledge that EXAGREE primarily addresses model and stakeholder disagreements (Case 1). In our experiments, we focused on fixing the explanation method to ensure a fair comparison, which means method-based disagreement (Case 2) was not fully explored in this work. We recognize that this is a complex open problem that cannot be resolved with a one-size-fits-all approach.

Moreover, I am not convinced that "higher agreement between rankings implies greater faithfulness". Bad actors might want explanations that hide the discriminatory nature of their models, hence want features to be ranked a certain way. In fact, several works have highlighted that explainability methods are prone to manipulation: Slack et al. (2020), Aivodji et al. (2019), Goethals et al. (2023). Explainability methods are tools to gain insight into a model (to potentially build trust) not project our desired belief upon the model.

Reponse: We followed OpenXAI metrics that measures faithfulness through ranking agreement. While we acknowledge that any technique modifying explanations could potentially be misused, as demonstrated in the cited works, the potential for misuse shouldn't prevent responsible research and development, similar to other technological advances in AI. EXAGREE's optimization within the Rashomon set offers a distinct perspective: unlike methods that produce single rankings, it helps users understand the range of possible feature attributions (attribution sets), enabling more informed decisions with confidence.

We thank the reviewer for the feedback and appreciate the recognition of reducing explanation disagreement through our framework. While the reviewer raises valid concerns regarding technical details, we prioritized problem formalization and overall framework development within the page constraints. We address the main points raised:

The loss functions for $L_{sparsity}$ and $L_{diversity}$ are not defined clearly in the paper or the appendix.

Response: $L_{sparsity}$ controls the distribution of values across different masks and $L_{diversity}$ maximizes the variance within each mask, ensuring they focus on distinct feature subsets. We will refine their definitions mathematically.

Precise mathematical definitions of what a mask is and how it is derived are essential for readers to understand the methodology in depth, as this concept is central to the approach.

Response: Given space constraints, we focused more on novel contributions over technical details available in prior work. We recognize the need for clarity on the mask concept and will include precise mathematical definitions in the Appendix.

Simply put, a mask characterizes/represents a model’s unique architecture within the Rashomon set by capturing distinct feature attribution patterns. This abstraction layer allows EXAGREE to compare and optimize models in the Rashomon set against stakeholder rankings and expectations. Such a mask can be obtained by adding additional layers to the optimal (reference) model, while ensuring that the resulting model still meets the loss condition within the Rashomon set. Equation (1) $L(M(X), y)$ can be adjusted to $L(M^{*} \circ m(X), y)$ for understand, where $m$ represents masks.

Consider adding a sentence or footnote to define core terms in your algorithm, ... as these abstract concepts can vary in meaning.

Response: We agree with the reviewer's suggestion and include a brief discussion here.

• Attribution Set: In our framework, each model generates a ranked list of feature attributions. The combined feature attributions from all models within the Rashomon set form what we refer to as an attribution set.
• Model Representations &Model Characterizations: In EXAGREE, masks serve as representations and characterizations for models within the Rashomon set. Each mask corresponds to a specific model, producing associated feature attributions. This structure allows for consistent model comparisons and end-to-end optimization.
• End-to-End Optimization: A training approach where all components of the framework are optimized simultaneously to minimize a specified loss function.
• In Equation 4, the surrogate model is designed to learn the non-linear relationship between feature attributions and their associated masks in the Rashomon set, enabling further optimization.

It would be helpful to include insights or references for the result in row 196. Why is faithfulness proportional to agreement? Is this a theoretical result, an empirical finding from the paper, or something else?

Response: The relationship follows OpenXAI's faithfulness metric definition, where faithfulness is measured by the agreement between rankings. Therefore, higher agreement with ground truth rankings naturally leads to higher faithfulness scores. We will add more discussion in the revised paper.

Figure 1.... Additionally, why are stakeholders grouped together in the first half but not in the second half?

Response: We will revise the figure and captions. We aim to illustrate the disagreement problem through grouped stakeholders with conflicting interests, while in the second half, individual needs are highlighted to represent our objective of satisfying diverse stakeholder requirements.

Figure 2 is clear, but it seems to appear too early in the paper...

Response: We will relocate this figure to Section 3.

Is the fairness improvement an explicit objective of the EXAGREE model?...

Response: This is an empirical outcome rather than an explicit objective for the current framework. We will clarify this and discuss its implications.

In row 276, in the explanation of the "Ranking Supervision and Correlation Metric," it would be beneficial to provide more context and motivation for this metric.

Response: Our choice of a ranking correlation metric is motivated by its connection to OpenXAI's faithfulness metrics, where faithfulness is measured through ranking agreement. We specifically selected Spearman's rank correlation because it provides a differentiable measurement and is well-established in the literature. We will revise Section 3.2 to include more motivation and discussion.

Specific suggestions

• Rashomon is corrected.
• Clarified the use of two pre-trained models plus one interpretable model.
• Refined the notation for $f_{\text{diffsort}}$.

We hope these clarifications are helpful and welcome any further discussion to strengthen the paper.

I would be grateful if the authors could answer at least some of the specific concerns raised in the reviews. The citation of [1] continues to be mis-placed, and as per my understanding strengthens the critique of the paper, rather than supporting the current way of framing the contributions.