A Principled Evaluation Framework for Neuron Explanations

Thank you for your feedback! We have greatly strengthened our paper with additional experimental and theoretical results, please see General response for an overview.

Below we respond to your concerns:

Weakness #1/Q#1: Benefits of unified framework

• We think a framework does not need to be complex to be useful, in fact to the contrary, we think the best frameworks feel simple and intuitive after you read them (but not before you read it).
• We believe that the main contribution of our framework is that it provides conceptual clarity to the topic of neuron descriptions which has been lacking in existing work, and makes it clear how standard statistical metrics apply in this setting.
• This framework allows us to analyze separate parts of each study, instead of considering them as a whole. For example, looking at a study in terms of metric, granularity, concept source etc. allows us to analyze existing studies with more detail (which was not done clearly in the literature before our paper; this is why we provide Table 1 in the main text Section 4, and the expanded version in Appendix E, Table 10), and combine good parts of different studies into new study designs. For example, evaluating F1-score by measuring recall in a human study and precision with a generative model is a promising study that would not be thought of without our framework.
• Our framework also gives us several new insights such as:
  • The fact that we can unify so many very different studies under one framework is surprising in itself. Before, they were thought to be more different than this, as can be seen in Table 1 (Section 4) and Table 10 (Appendix E) in the updated draft
  • Existing papers sometimes use a different framing in terms of whether concept $c_t$(simulation) or neuron activation $a_k$(classification) is the predicted variable, as we discuss in Appendix A.2. This causes some confusion in terms, because the simulation framing of the Recall metric is equivalent to the classification framing of Precision.
  • Methods that use generative models as the source for concept labels $c_t$ typically use metrics that fail the missing labels test as discussed in Appendix A.4. This actually might be necessary as labels from generative models typically are missing labels. This leads us to recommend combining these analyses with a different kind of evaluation such as a human study of recall.
  • Separating metric from other details of the study allows us to focus on analyzing metric choice alone, and inspired by the framework we develop sanity checks to find bad metrics.

Weakness #2: Why trust sanity checks over metric outputs themselves

Thank you for the response. We believe you would find our new theoretical sanity check in Appendix B interesting. In that section, we find that metrics failing the sanity checks is mostly caused by poor performance on imbalanced data. We cannot just trust the guarantee offered by the metric at face value every time. For example, if someone claims their model has 99.9% accuracy on a binary classification task, does that mean they have a good model? Not necessarily, they could just have a dataset where only 0.1% of the inputs are positive and the model simply predicts no every time. This is precisely why accuracy is not a good metric for evaluating neuron description accuracy, as the concepts of interest are often highly unbalanced, and in response we can see accuracy fails our sanity checks in Table 2 and Appendix B. The point of our sanity checks is to identify metrics whose output scores are not always indicative of good predictions, in particular on imbalanced data.

Weakness #3: Limitations of correlation

Thank you for the suggestion. Our proposed sanity check tests can be regarded as providing a necessary condition for good evaluation metrics. Thus, a good metric should pass the tests and for those metrics that do not pass the tests, they should not be used. As a result, we have toned down our language regarding correlation as the best metric, and updated the conclusions in Sec 6 to say that we focus on identifying a set of good metrics to be more in line with the strength of our experimental results. We have not identified any specific issues with correlation but some may exist.

Question #3: Limitations:

Thank you for the suggestion. We have added a comprehensive discussion of the limitations of our framework and experimental results in Appendix A.1.

We believe we have addressed all your concerns, please let us know if you have any remaining questions or concerns, and we would be happy to discuss further!

Dear Reviewer ibwF,

Thanks again for reviewing our paper! With the discussion period ending today, we wanted to follow up as we still haven’t heard from you. We have conducted extensive additional experiments and believe we have resolved your concerns but please let us know if that is not the case. We would appreciate your feedback or an acknowledgement that you have read through rebuttal response.

(continued for Reviewer 1n3c)

Weakness 5: Passing sanity checks does not guarantee the metric is faithful

Our approach is inspired by the sanity checks of Adebayo et al [R1], which have been very influential in the field of local feature importance/saliency maps. To clarify, passing the sanity checks proposed by Adebayo et al. [R1] does not guarantee that a method is good, but not passing them means that a method certainly is bad. In other words, one can think of this kind of sanity check as a necessary condition for a good metric. Our sanity checks are intended to be used the same way, to identify bad metrics and are not designed to guarantee a method passing them is good. We do not believe this reduces the value of having these tests and similar to the sanity checks of Adebayo et al. [R1], our checks can help guide the field in the right direction. We have clarified this point and included discussion of this limitation in Appendix A.1.

Weakness 6: Lacking fine-grained concept experiments

Thank you for the suggestion. Following your recommendation, we have conducted all our experiments additionally on the fine-grained concepts of CUB200 dataset, using two separate models, a standard Concept Bottleneck Model trained on this data, as well as a linear probe trained on CLIP embeddings. Please see Tables 18, 19, 21 and 23 in Appendix G for detailed results. In terms of final layer metrics our method can very clearly differentiate between different metrics, see Table 21 for example. For missing/extra labels test there is less difference between metrics, which we identify is caused by the CUB concepts being more balanced (positive on at least 5% of the inputs) than for example ImageNet classes (positive on 0.1% of the inputs). See Tables 4 and 5 in Appendix B for detailed study on the effect of concept balance on our sanity test results.

Reference:

[R1] Adebayo, Julius, et al. "Sanity checks for saliency maps." Advances in neural information processing systems 31 (2018).

Summary

In short, there is one comment (Weakness 2b) that we hope the reviewer could help to clarify, and we would be happy to address the reviewer's question or concerns. We believe we have addressed the reviewer's other concerns in weakness. Please let us know if you have any additional concerns and we would be happy to discuss further!

Dear Reviewer 1n3c,

Thank you for the review! In response to feedback from you and other reviewers, we have greatly expanded our experimental results and restructured some of the paper, please see General response for description of all the changes and the updated manuscript.

Below we clarify and respond to specific concerns you have:

Weakness 1: Limited to explanation based on simple concepts:

Our framework can evaluate any textual explanation, and is not limited to class/superclass labels at all. To showcase this, we have added experiments using the fine-grained concepts in the CUB200 dataset, such as black wing-color. In addition, our framework can be easily extended to non-textual concepts such as a prototype-based explanation as long as we have a way to determine the concept vector $c_t$ for that concept as we discuss in Appendix A.1.

One thing that is unclear to us is about your comments on the other methods such as a saliency map or the receptive fields. To clarify, in this work, we are focused on global neuron explanation, while a saliency map is a local input importance method which does not belong to neuron explanation methods. If you are aware of a method that produces these types of global explanations for a single neuron, please provide a reference and we would be happy to look into this. In short, our aim is not to evaluate every explainable AI method, but specifically global neuron descriptions, which provide global explanations to characterize neuron functionalities.

Weakness 2a: Why do you only adjust the concept label for testing? Why not try to adjust neural activations?:

May we ask if this question refers to the missing/extra labels test? To clarify, we focus on adjusting the concept as typically in neuron explanation the neuron is considered fixed and the concept is the thing we can change. In addition, the current formulation of the test relies on $c_t$ being binary, and neuron/unit activations are not binary in practice, so it’s not clear how to extend this. However, in our theoretical setting in Appendix B where we consider an ideal neuron with $a_k = c_{t_i}$, we can see that these two formulations give the same results.

In particular, a missing labels test on neuron $a_k$ that activates on fraction $\gamma$ of the neurons is equivalent to an extra activations test on neuron $a_s$ that activates on fraction $\gamma{}/2$ of the inputs. That is, $M(a_k, c_{t_k}^-) = M(a_s^+, c_{t_s})$ for any metric.

Weakness 2b:The theoretical guarantee for the completeness of the proposed method is needed.

Can you please clarify what you mean by this? What does completeness mean in this context of global neuron explanations?

Weakness 3: What is $t_k$

$t_k$ is a text description. We have clarified this in Section 2.2 (Line 101) in the revised draft.

Weakness 4: Missing linear combinations of neurons.

Computing the tests on a linear combination of neurons is mathematically no different from calculating the tests on a single neuron. In fact, every neuron in a feedforward network is a linear combination of previous layer neurons. In mathematical terms, let A be a d x n matrix of the activations of all d neurons in a layer on all inputs. A regular neuron’s activations $a_k = A_k$, i.e. the kth row of A. For a linear combination, we can represent its activation pattern as a virtual neuron $\hat{k}$ $a_{\hat{k}} = wA$, where $w$ represents the coefficients of the linear combination. To showcase this ability in practice, we trained a Linear probe for CUB concepts on top of CLIP image encoder embeddings, where each concept is represented by a linear combination of neurons and included the results in our evaluation. Please see Tables 19, 21 and 23 in Appendix G for detailed results.

Dear Reviewer 1n3c,

Thanks again for reviewing our paper! With the discussion period ending today, we wanted to follow up as we still haven’t heard from you. We have conducted extensive additional experiments and believe we have resolved your concerns but please let us know if that is not the case. We would appreciate your feedback or an acknowledgement that you have read through rebuttal response.

Dear Reviewer y8T5,

Thank you for the detailed review and positive feedback! We have conducted extensive additional experiments, based on all reviewers feedback, please see the General response for an overview of all new results.

Below we would like to address specific concerns you have:

Weakness 1a: Parameter selection:

• As described in Line 511 in the original manuscript (Line 482 in the updated draft): For all experiments we split a random 5% of the neurons into a validation set. For metrics that require hyperparameters such as $\alpha$, we use the hyperparameters that performed the best on the validation split for each setting. We then report performance on the remaining 95% of neurons. Note that this is how we chose $\alpha$ for results in Table 3.

• For Missing/extra labels test on final layer neurons, we used the $\alpha$ that gave the best performance in terms of AUC evaluation. For hidden layer neurons we used $\alpha = 0.005$ for all metrics following Network Dissection [1]. We have clarified this with additional explanation in Section 3.1 and Appendix G.

• The threshold parameter $b_{\alpha}$ in Eq. 5 is determined by the value of $\alpha$ and is not an independent parameter.

• For Eq. 6, we used 0.5 as this is the conventional cutoff when measuring accuracy in binary classification and it seems hard to justify using a different cutoff for a probability.

Weakness 1b: Ratio selection in Missing/Extra labels test

Thank you for the suggestion, we have conducted an ablation using different ratios in Appendix F.1, please see Tables 11 and 12. We find that our results are not sensitive to this choice of ratio, and instead we get very consistent results in terms of which metrics pass/don't pass the tests.

Weakness 2: Missing Function 2

Thank you for the feedback. We have added the following discussion under Limitations in Appendix A.1 to make it clear that this work we focus on evaluating function 1 only.

“Function 2: Activation → Output: Probably the most significant limitation of our framework is that it is focused on evaluating function 1 only (Input → Unit Activation) as discussed in Section 2.3, and we believe measure Function 2 is equally important. While currently our framework is meant for function 1 only, we believe many of the ideas and metrics we discussed could be useful in evaluating function 2. For example, in a generative model we could use the same metrics to measure similarity between unit activation and the presence of a specific concept in the output. However in function 2 there are additional considerations and things like measuring difference in outputs when changing the unit activation are likely more important. We believe extending this framework or creating a similar one for function 2 evaluations is an important direction for future work.”

Weakness 3: All methods have low accuracy in Setup 4 of Table 7

This low performance is caused by limitations in our pseudo-labels. Specifically, the pseudo-labels from SigLIP are quite accurate for original imagenet classes, but become much less accurate for superclasses, making it impossible to reach great accuracy on Setup 4 (while Setup 2 many methods perform well). Note old table 7 is now in Table 20 of the revised manuscript

We hope we have addressed all your concerns, please let us know if you have any additional questions and we would be happy to discuss further!

Dear Reviewer y8T5,

Thanks again for reviewing our paper! With the discussion period ending today, we wanted to follow up as we still haven’t heard from you. We have conducted extensive additional experiments and believe we have resolved your concerns but please let us know if that is not the case. We would appreciate your feedback or an acknowledgement that you have read through rebuttal response.

Dear Reviewer DS3i,

Thank you for the detailed feedback and thoughtful comments. We have added extensive additional experiments in response to reviewer feedback, please see the General response for a summary. In particular, we think our theoretical missing/extra labels test in Appendix B (in the revised draft) will be of interest to you and potentially addresses many of your concerns, so we hope you could take the time to read through our results and please let us know if you have any additional feedback.

In response to your specific concerns:

Weakness #1: Framing missing/extra labels test as label noise

You raise an interesting point, and we agree that one way to view these tests is in terms of label noise. However, we disagree with your conclusions, in particular that a good metric should be robust to label noise. This is based on the following reasons:

• To clarify, we introduce a very large amount of label noise, i.e. in extra labels tests, 50% of the positive labels are now incorrect. While robustness to low amounts of label noise could be desirable, if a metrics output does not change in response to a very large amount of noise, it is insensitive/not measuring the right thing. This can be seen in other settings, for example in the context of adversarial learning, robustness to small perturbations is desirable, but the only way to be robust to very large perturbations is a constant classifier that predicts the same class regardless of input, which is clearly not desirable. Another example of this same phenomenon can be observed in the Sanity Checks for Saliency Maps [R1], which inspired our sanity checks. In their Figure 2 they perform a randomization test on network parameters. While you might initially think that robustness to small changes in NN parameters is desirable for a saliency map method, if we completely randomize the network weights and the results still don’t change it means the saliency map is insensitive to network parameters in problematic ways instead of being robust.
• As an extreme case, consider adding uniform random noise with 50% chance of flipping each label. Since there is little to no information left in the labels, clearly the evaluation metric should output a different score.
• We do not think a good evaluation metric should be robust to label noise. A good learning algorithm should be, but an evaluation metric should not. For example, if you measure the accuracy of a good classifier, but 10% of your labels are incorrect, you should get a lower accuracy.
• Our setting is a little different from standard label noise settings, where typically every label has a uniform probability of being incorrect. Instead, our label noise completely depends on the ground truth label, i.e. in missing labels test the labels where concept is present have 50% chance of being incorrect, while labels where concept is not present are always correct.

Overall we think that because we introduce a very large amount of label noise, metric not changing its value in response is not a case of positive robustness, but instead reveals a problematic insensitivity to certain features of the labels.

Additionally, in Appendix B, we represent an alternative way to view this test, which we think can provide more theoretical insights to our proposed tests. In particular, in an ideal setting where a neuron’s activations perfectly align with a concept, i.e. $a_k = c_{t_k}$, our Missing and Extra Labels tests correspond to being able to differentiate between the following 3 concepts:

• $c_{t_k}$: The perfect predictor for neuron $a_k$, with precision=1 and recall=1.
• $c_{t_k}^{-}$: (Missing labels) This concept has Precision=1 since whenever a concept is present, the neuron is also active, and Recall of 0.5 since only half to inputs where the neuron activates now have the concept.
• $c_{t_k}^{+}$: (Extra labels) Inverse from above, this concept has Precision=0.5 and Recall=1.

Clearly, the concept with perfect recall and precision should achieve a higher score on the evaluation. However, this is not the case for most metrics as we show in Appendix B, as they fail one or both of these tests, especially if data is imbalanced, i.e. $c_{t_k}$ is only rarely positive. This motivates our test, a good metric should be able to differentiate these concepts regardless of concept imbalance, and the passing metrics such as F1-score/IoU and Correlation do.

I would like to thank the authors for their detailed response. While I plan to take additional time to analyze the full response, I am prioritizing speed over precision in this reply to allow the authors as much time as possible to address the remaining concerns.

One of my earlier concerns has not yet been fully resolved. I agree with the authors that, at a practical level, a metric should be sensitive to label noise. However, as highlighted in my original review, it remains unclear why a "suggested/better" metric should be more sensitive to label noise than the F1-Score, for instance. The F1-Score is widely recognized as the standard metric for binary classification in the presence of label noise (in the general case, without specific constraints).

Could the authors elaborate further on this point?

Specifically, should the objective of this research area be to identify the most sensitive metric to label noise? In this context, could the authors provide examples demonstrating cases where the F1-Score is a better metric for binary classification but not for neural explanations, and conversely, where correlation is superior for neural explanations but less effective for binary classification? Alternatively, it would be helpful if the authors could provide a reference supporting the claim that correlation is a better metric than the F1-Score for binary classification problems, or offer a theoretical proof or justification.

Thank you in advance for your clarification.

(continued for Reviewer hRsh)

Q1: Sanity checks in other domains

Yes, applying the sanity checks on other domains is something we wish to do in the future but were unable to do in the rebuttal timeframe. However we do provide results on a linear probe trained on CLIP embeddings which is technically multimodal. More importantly, our theoretical sanity check results in Appendix B are completely independent of the domain used and closely match our empirical results on the vision domain, making us confident we would observe the same things in the language domain.

Q2: Incorporating feature visualization

This is an interesting extension and could be measured in our framework by including some feature visualizations in the probing data and would be interesting to evaluate the effects of in the future. We will add this into future work discussion.

Q3: Extend analysis to new architectures and neuron types:

As discussed in General Response, we have greatly extended our results to 2 additional datasets, 3 new models and new types of neurons, including hidden layer neurons, CBM concept neurons and linear probe outputs. Overall we find the results are highly robust across scenarios.

Q4: Incorporate qualitative analyses or practical case studies or concept-based dataset:

We have extended our evaluations to CUB200 which is a concept-based dataset.

In short, we hope we have addressed your concerns and clarified your questions. Please let us know if you have any additional concerns and we would be happy to discuss further!

Dear Reviewer hRsh,

Thank you for the detailed feedback. In response to feedback from you and other reviewers, we have greatly expanded our experimental results and restructured some of the paper, please see General response for description of the changes and the updated manuscript.

Below we respond to your concerns:

M.1. More related works:

Thank you for the feedback, following your request, we added an extended discussion on related works in Appendix E, including evaluation of individual neuron explanations, sanity checks, concept extraction, concept importance estimation and human-centered evaluation of concept-based models. Specifically, following your request, we added more reference on concept extraction, concept importance estimation and human-centered evaluation of concept and neurons, discussing their difference from our work. We hope this should address your concern about lacking references.

M.3. Limited discussion on Metric Limitations:

Thank you for the suggestion. We have added an extended discussion of limitations in Appendix A.1. In particular, the section on experimental result limitations should address this somewhat. More specifically, we view our proposed evaluations and sanity check tests as necessary conditions for a good metric (i.e. a good metric should pass the sanity check, and for those metrics that do not pass the tests, they should not be used). Thus, although we have not really identified a failure case/issue with using correlation so far, we have toned down our language throughout the paper regarding correlation as the best metric, and instead focus more on identifying (multiple) good metrics that pass the sanity checks.

M.5. Focus on Final layer neurons:

We agree our analysis on Section 5 and Table 3 is focused on final layer neurons, even though we have included results on CBM concept neurons in the rebuttal which are an intermediate layer. However there is no way to avoid this as these metrics are only possible if you know the ground truth function of the neuron. To address this we have included discussion of this in Appendix A.1 (experimental result limitations).

On the other hand, our Missing/Extra label tests are not focused on final layers, and we run the evaluations on both last layer and hidden layer neurons with very similar findings. In particular, our new theoretical Missing/Extra labels test results described in Appendix B show these results are not dependent on the specific neuron/evaluation setup but a fundamental feature of the evaluation metric.

minor1. Impact of $\alpha$:

As described in Line 511 in the original manuscript (Line 482 in the updated draft): For all experiments we split a random 5% of the neurons into a validation set. For metrics that require hyperparameters such as $\alpha$, we use the hyperparameters that performed the best on the validation split for each setting. We then report performance on the remaining 95% of neurons.

That means for each (metric, experimental setting) combination, we find the best value of $\alpha$ using grid search on a small split of validation neurons, and then report the performance on unseen neurons. We believe this is a fair way of choosing the $\alpha$ parameter that lets us report performance with a good choice of $\alpha$ for each task. We have added this description to Section 3.1 as well to make this more clear.

minor2. Lengthy recap of Basic Definitions:

Thank you for the response, we have moved most of the metric definitions from Section 3.1 to Appendix C, as well as added a short description of the remaining ones. We believe this helps the clarity and readability of the paper.

minor3. Redundant Explanations of Failure cases:

Thanks for the feedback. We have chosen to keep these sections for now, while they are standard statistics concepts we think it is important for some readers to highlight how they apply in the context of a neuron explanation, as this topic has been largely lacking statistical rigor so far.

minor4. No aggregate analysis:

We think this may be a misunderstanding. Both of our main results represented in main text (Table 2 and Table 3) are aggregations across models/settings(aggregated across many more settings after rebuttal), and individual model results are only represented in the Appendix G.

Dear Reviewer hRsh,

Thanks again for reviewing our paper! With the discussion period ending today, we wanted to follow up as we still haven’t heard from you. We have conducted extensive additional experiments and believe we have resolved your concerns but please let us know if that is not the case. We would appreciate your feedback or an acknowledgement that you have read through rebuttal response.

Dear Reviewer b3hf,

Thank you for the valuable feedback! We would like to address your questions below:

#1 Form of $\mathcal{G}$ in Equation 1:

To clarify, we do not specify the form of $\mathcal{G}$ on purpose, as it does not matter for our paper. As we specify, on Line 92, the focus of our paper is evaluation, and we can evaluate description the same way regardless of the process that generated it. As an example for G, e.g. in Network Dissection, we can write $\mathcal{G}(\mathcal{D}, f, k, l) = argmax_{t \in C} M_{IoU}(f_k^{0:l}(X), CC(X, t))$ where $X$ is a tensor of all inputs in $\mathcal{D}$ concatenated, $C$ is the set of concepts they have labels for and $CC$ is a function that maps an (input, concept) pair into $P(t|x)$, in their case a lookup table on the labeled data. However sometimes G cannot be written explicitly, e.g. G could also be a human worker that writes a description for the neuron, as discussed in Line 98, and clearly we can’t write that in a closed form equation.

To reiterate, the explanation generation process is not part of our framework or this work. We are trying to answer the following question: Given an explanation, how do you evaluate how good that explanation is?

#2 How are the results in Figure 2 computed:

To walk through the calculations in Figure 2, let us write down the relevant vectors, with the images in the top row corresponding to the first 4 elements and bottom row corresponding to the last 4 elements. We then get:

• $B(a_k) = [1, 1, 1, 1, 0, 0, 0, 0]$
• $c_{\text{dog}} = B(c_{\text{dog}}) = [1, 0, 1, 0, 0, 0, 0, 0]$
• $c_{\text{cat}} = B(c_{\text{cat}}) = [0, 1, 0, 1, 0, 0, 0, 0]$
• $c_{\text{pet}} = B(c_{\text{pet}}) = [1, 1, 1, 1, 0, 0, 0, 0]$
• $c_{\text{animal}} = B(c_{\text{animal}}) = [1, 1, 1, 1, 1, 1, 1, 1]$

Equation 10 (Eq. 13 in updated manuscript) refers to a special case where $c_t = e_i$. However for figure 2 we are not in this special case as each concept is positive on multiple inputs. Instead, to calculate precision in general we should use Equation 8 (in the new manuscript): $M(a_k, c_t) = \frac{B(a_k) \cdot B(c_t)}{||B(c_t)||_1}$. Plugging in the values we get:

• $M(a_k, c_{\text{dog}}) = 2/2 = 1$
• $M(a_k, c_{\text{cat}}) = 2/2 = 1$
• $M(a_k, c_{\text{pet}}) = 4/4 = 1$
• $M(a_k, c_{\text{animal}}) = 4/8 = 0.5$

Please let us know if you have any additional concerns regarding this calculation

#3 Equation (1). Why consider layer l after specifying target neuron k?

To fully specify a neuron we need to specify both which layer it is (using $l$), as well as its index within that layer (using $k$)

#4 Equation (1). Is $t_k$ a textual description of neuron function or a one-hot vector based on concepts?

$t_k$ is a text description, we have clarified this in the updated manuscript

#5 Considering Precision, Recall and F1-score together:

In general it can be beneficial to consider multiple metrics together. A major concern we have in this paper is that many previous evaluation methods focus only on evaluating Recall, which is insufficient. Since F1-score is the harmonic mean of Precision and Recall, it can be useful by itself, but for a more complete picture it can be helpful to evaluate it together with precision and recall.

We hope we have clarified your questions, please let us know if you have any additional concerns. We also conducted many additional experiments and restructuring of our text, please see General response for summary of these changes.

Dear Reviewer b3hf,

Thanks again for reviewing our paper! With the discussion period ending today, we wanted to follow up as we still haven’t heard from you. We have conducted extensive additional experiments and believe we have resolved your concerns but please let us know if that is not the case. We would appreciate your feedback or an acknowledgement that you have read through rebuttal response.

6. Writing:

• Added a large section discussing limitations in Appendix A.1
• Following reviewer suggestions we moved many of the metric definitions from section 3.1 to Appendix C
• Added additional discussion on related work on Appendix E
• Toned down language regarding best metric, and focus more on identifying a set of good metrics that pass the tests.
• Fixed equations for score diff and decrease acc to have correct sign in Sec 4.2

Overall we also have made many small changes in writing and paper structure. We highlight most new text in blue, for sections that are entirely new or mostly rewritten we only have titles highlighted in blue. We will post responses to individual reviewers shortly.