Explanations that reveal all through the definition of encoding

Thank you for the generous comments and detailed feedback. We fixed the writing issues in the paper. If our response below addresses your primary concerns, would you kindly consider raising your score?

[rank explanations, compare with ROAR]

We thank the reviewer for raising this point. Unlike ROAR, EVAL-X and DET-X provably at least weakly detect encoding on any distribution, not just on the toy examples. We provide a comparison on the image recognition experiment; see the PDF in the general response [LINK.]. ROAR incorrectly ranks multiple encoding explanations as high as the optimal explanation.

[Encoding is like spurious correlations?]

One could think of encoding as a spurious correlation between the selection and the label. This correlation is not a fact about the DGP that produces $(X,Y)$. It is a consequence of the explanation depending on features that are informative of the label but only selecting a subset of them.

[Def 1 and 2 sound like propositions, domains of $e$]

We have updated the writing to make definition 1 start with "An explanation is called encoding if ... ". In definition 2, we define the notion of weak detection for a distribution $q(y,x)$ as a property an evaluation method can have. The domain of $e,^*$ is that of all functions that takes and input $x$ and return a binary mask over input. For example, considers explanations for images-label pairs; the domain is the space of images. We have clarified this in the draft.

[definition 3, depend on the direction of the metric]

The reviewer is correct. We assume higher scores mean better. If the direction flips, Definition 3 would say $\leq$ instead of $\geq$.

[the argument in line 234-239]

Definition 1 and 2 are about weak and strong detection, which are properties of evaluation methods. Lines 234-239 only define EVAL-X. Theorem 1 proves that EVAL-X is a weak detector. In other words, EVAL-X is a weak detector. The sentence 239 leads to theorem 1. We have updated the draft to separate the definition of EVAL-X from the sentence that says it is a weak detector.

[lines 123-124, "... to predict the label have not been selected ..."]

The updated sentence reads "An encoding explanation should not score optimally under a good evaluation because the requisite input values needed to predict the label have not been selected by the explanation on a subset of the input values."

[lines 154-161 unclear]

This is an example of a "bad" encoding explanation. We flesh out the example here. We elaborate here. Consider reviews that can either be of type "My day was terrible but the movie was [ADJ1]." and "The movie was [ADJ2], but the day was not great." where ADJ1 can be "good" or "not great" and ADJ2 can "not great" or "terrible".

Due to english parlance, "terrible" indicates bad sentiment more often than "not great". Then, in the example setup above, only seeing that the fourth word is "terrible" yields bad sentiment with higher probability than when only seeing that the phrase is "not great". However, the fourth word but does not always describe the movie. An explanation can look at "not great" describing the movie as bad but then select terrible to encode the bad sentiment. Such an explanation is encoding because it selects a word that does not describe the movie but is informative of the sentiment.

[in Eq. 3, "Different" vs. "Additional"?]

The word additional denotes that knowing the explanation provide extra information about the label not in the selected values.

[lines 196-198 informal.]

The lines say that the these explanations are encoding due to lemma 1. By "the explanation e(X) varies with inputs other than the selected ones" we mean that $e(X)$ depends on inputs that weren't in the binary mask. By "... inputs provide information about the label .. " we mean that the inputs that are not selected in $e(X)$, are informative of the label which implies the second condition in lemma 1.

[fig1c unclear. Posenc selects the top-half? $Y \perp$ color?]

We construct PosEnc to select the top-left-patch if label=cat. Intuitively, the label is independent of the color regardless of which patch it selects, the label is only determine the animal in the image. We added math to prove this in the draft.

["may seem okay" informal][

The selected inputs (third panel, second row, left square for each input) show exactly the animal of the label. Selecting only the animal of the label seems desirable at first; this is what we meant by "may seem okay". We then explain why such an explanation is encoding.

[how to show control flow?]

In the figure 1 and figure 3 examples, the "control flow feature" is the color patch, because it branches the DGP where the top right or the bottom right patches produces the label.

[209-210 optimization? constraints?]

Optimization here refers to finding an explanation that maximize the evaluation score $\max_e \alpha(q, e)$. Various constraints can be put on explanations, such the explanation cannot select more than $K$ inputs.

[220-221 "x3 predicts the label"? why?]

The reviewer is correct. However, conditional on the selected inputs, does predict the label. See the PDF in the general response for the math.

[Discuss limitations]

See the general response LINK.

[Based on assumptions that may not be true on real data like Imagenet.]

Evaluations like ROAR and EVAL-X have been used on explanations on imagenet and on chest x-rays. We prove such methods highly rank undesirable explanations that encode. In turn, conclusions about important inputs drawn from such high scoring explanations may not hold for the model being explained. Further, we experiment with a real sentiment analysis task where DET-X uncovers evidence that LLM-generated explanations encode.

We thank the reviewer for their careful feedback. If our response below addresses your primary concerns, would you kindly consider raising your score?

[paper is very very dense and this makes it at times intelligible]

We thank the reviewer for this feedback. We modified the draft to use the standard paragraph spacing with the following changes to the writing

• Made section 2.1 more concise, where each encoding example now only spans a single paragraph covering both intuitive example formal construction.
• Moved lemma 1 and associated text to the appendix.
• Made 5.1 and 5.2 concise by only listing results and conclusions, and moved experimental details about training models for the encode-meter to the appendix.

If the reviewer has any other concerns about the clarity/density, we'll be happy to address them.

[Weak contextualization with regard to prior work]

There are limited studies in evaluating explanations and we discuss them. Due to space constraints, we had moved the related discussion about faithfulness and label-leakage to appendix B. We have moved it back and give it here:

Other investigations into evaluating explanations focused on label-leakage [14, 34] and faithfulness [11, 16, 35, 36, 37]. Akin to encoding, label-leakage is an issue that occurs with explanations that depend on both the inputs and the observed label; such explanations, when built naively, can yield attributed inputs that are more predictive of the label than the full input set. In this paper, we do not consider explanations that have access to the observed label. Faithfulness, intuitively, asks that the explanation reflect the process of how a label is predicted from the inputs. Typical faithfulness evaluations rely on the quality of prediction from the selected inputs [12, 16]. Jacovi and Goldberg [11], Jethani et al. [13] discuss how such evaluation methods are insensitive to encoding by checking how they score encoding constructions.

- [11] https://arxiv.org/abs/2004.03685
- [12] https://arxiv.org/abs/1806.10758
- [13] https://proceedings.mlr.press/v130/jethani21a.html
- [14] https://proceedings.mlr.press/v206/jethani23a.html
- [16] https://arxiv.org/abs/2005.00115
- [34] https://aclanthology.org/2020.findings-emnlp.390/
- [35] https://arxiv.org/pdf/2111.07367
- [36] https://ojs.aaai.org/index.php/AAAI/article/view/21196
- [37] https://arxiv.org/abs/2109.05463

If the reviewer sees a work we missed we would be happy to add it.

[Fig.1C, "the label is produced from the bottom image if the color is red" then why is the explanation about the top image]

There may be a little misunderstanding here. In this example, the explanation selects the top left quarter patch only, which only contains color. This explanations encodes information about the label in the selection. The reviewer may also be asking why such a construction can be called an explanation. As the overarching goal in the paper is evaluating explanations, we consider any functions of inputs that outputs binary masks as candidates.

[l.156 "For example, How would we differentiate a model that makes that mistake from an erroneous explanation]

We respond here assuming that the reviewer used the word "erroneous" to mean that the explanation selects inputs other than the ones that the model uses to predict. If the reviewer meant something else and clarify their comment, we would be happy to engage in further discussion.

To understand whether explanation selects the input the model depends on, one would have to look at how predictive the selected inputs are of the labels produced by the model . Let's look at evaluating a candidate explanation of a models predictions without assuming anything about how the explanation or the model were produced.

• If the model makes mistakenly relies on the word "terrible" and the explanation correctly selects the word "terrible", then the selected features would have maximum information about the model-predicted labels.
• If the model correctly depends on value that ADJ takes but the explanation selects the word "terrible", then the selected features would can only be informative of the model-predicted labels by encoding.

Due to these differences, an encoding explanation of a correct model and a non-encoding explanation of an incorrect model have different signatures that EVAL-X or DET-X would detect.

[l.165 "control flow" is not define]

Thank you for pointing this out. Control flow is not common parlance and comes from the software engineering literature. We explain it below. We use the phrase "control flow" to indicate that the DGP looks at the value of one input to determine which other inputs to produce the label function; this process reflects an if-else statement where the condition being checks determines the flow of computation. We have added this paragraph to the paper.

[l.366 "Trained the same way as EVAL-X but, instead of predicting the label, it predicts the identity of subset selected by the explanation?]

EVAL-X trains a model for $q(y | x_v)$ that predicts the label $y$ from the selected inputs $x_v$, by randomly choosing $v$ for every sample. Similarly, ENCODE-METER relies on the conditional distribution $q(E_v | x_v)$. To model $q(E_v | x_v)$, we replace $y$ with $E_v$ in the EVAL-X training procedure. Here, $E_v$ is the indicator of whether the explanation for the input $x$ is the subset $v$.

[I do not see a limitation section in the paper.]

We discuss limitations in the Discussion section 6, paragraph "Mis-estimated models, ... ". We elaborated on the limitation in the general response LINK. If the reviewer can point to other questions or limitations that we should discuss, we are happy to add them.

We thank the reviewer for their careful feedback. If our response below addresses your primary concerns, would you kindly consider raising your score?

[The proposed DET-X score may require complex implementation and computational resources, which might limit its adoption in practical scenarios.The DET-X score's implementation complexity might be a barrier for widespread adoption, especially in resource-constrained environments.]

While the reviewer is correct that one needs to estimate the ENCODE-METER in addition to EVAL-X, the training process is standard supervised learning. The only difference is that the inputs are randomly masked like in EVAL-X. As training and evaluating predictive models via supervised learning is well-studied, even at scale, we do not foresee that training DET-X as a difficult task.

The reviewer is correct that DET-X requires twice as much computation as EVAL-X, which itself can take more computation than training a single model that predicts the label from the full inputs. This extra computation comes from have to learn to predict from different subsets. However, one cannot escape training to predict from different subsets when evaluating explanations based on how informative the selected inputs are. Using large pre-trained models does speed up this process. The estimation of encode-meter with GPT-2 in the LLM experiment, including hyperparameter tuning, took under a single day on a single GPU. Alternatively, given a conditional generative model for the full inputs given the masked ones, both components of DET-X reduce to averaging a single model's predictions over generated samples.

Code for training models from masked images for EVAL-X already exists and we will release our implementation for DET-X with the camera-ready version, if accepted.

[The paper acknowledges that misestimation of models used in evaluation could lead to incorrect conclusions, suggesting a need for robust estimation techniques. The paper highlights the risk of misestimation in model-based evaluations, which could impact the reliability of DET-X in certain scenarios.]

Thank you for highlighting this point. The problem with misestimated models goes beyond our work and applies to all evaluation methods that build a model to score explanations, including ROAR, FRESH, and recursive ROAR. Avoiding misestimation is unavoidable, but post-hoc methods for capturing the uncertainty of predictions, via conformal inference or calibration, can help mitigate the errors in evaluation due to misestimation.

[While the experiments are thorough, they are limited to specific datasets and types of tasks (image recognition and sentiment analysis). More diverse applications would strengthen the claims. How does the DET-X score perform in different domains outside of image recognition and sentiment analysis?]

The goal of the paper is to establish the mathematical definition of encoding and strong detection of encoding. So we focused on two popular tasks for which explanations are produced.

We have since added a tabular data experiment and report the results here. We ran an experiment on tabular CDS Diabetes dataset from the UCI repository and show the effectiveness of EVAL-X and DET-X on weakly and strongly detecting encoding respectively. See the PDF in the general response LINK.

The results show that

1. EVAL-X scores the Optimal explanation above all the encoding ones, showcasing weak detection.
2. However, EVAL-X scores the last non-encoding explanation, which selects features informative of the label, below the encoding ones PosEnc and MargEnc, showing its not a strong detector.
3. DET-X correctly scores all the non-encoding explanations above all the encoding ones, demonstrating strong detection

[Are there specific conditions or types of models where DET-X might not perform as expected?]

DET-X is model-agnostic and would extend any type of model as long one can obtain input-output pairs from the model. DET-X depends on models that predict the label from subsets of the inputs. Learning to predict from subsets may take much larger models than predicting from the whole input set when the distribution of the label changes dramatically between conditioning two similar input subsets. As we discuss in section 6, DET-X only works for explanations the select subsets of features. Future work can extend the notion of encoding to free-text (natural language) rationales.

[How does the computational cost of DET-X compare to existing evaluation methods?]

DET-X has twice the computational time of EVAL-X because it runs the EVAL-X training process twice, but the two models are independent and can be trained in parallel. Compared to methods like ROAR, the relative increase in computational context might depend on the problem at hand as the training procedures are different which mean optimization may converge at different rates.