Thank you for the review and positive feedback! Your summary is very well written and highlights a strong understanding of our work. We would like to address your concerns below:

Weakness 1 - Labeling Cost

This is a great point. It is true that a significant reason for the use of recall is that it is cheaper to evaluate as we only need to only evaluate $c_{ti}$ on highly activating inputs, i.e. where $a_{ki} = 1$. In contrast, evaluating most other metrics requires knowledge of the full $c_t$. One way to address this issue is to use a combination of metrics, for example evaluating F1-score by combining a crowdsourced evaluation of Recall with a generative model based evaluation of Precision as we suggest in Line 897 of the Appendix.

As you mentioned, crowdsourced evaluation of the entire $c_t$ will likely give noisy results as most inputs do not contain the concept. To avoid this, another approach is to oversample highly activating inputs similarly to Top-and-random sampling. This can be done without failing the extra labels test if the proper importance sampling correction is applied to correct for the bias introduced by the sampling. However, the effective sampling/user study design for non-recall metrics is a rather large and complex topic that does not fit within the scope of the current paper, but it is something we are actively looking into and we are confident that there are ways to evaluate the other metrics with a cost not much higher than recall evaluation. We will include discussion of this under limitations/future work.

Weakness 2 - Missing citation

Thank you for pointing this out. We will cite [1] as well as other papers[2, 3] conducting similar studies based on neurons with known concepts as suggested by Reviewer 7Dju.

[1] Soltani Moakhar, A., Iofinova, E., Frantar, & Alistarh, D. SPADE: Sparsity-Guided Debugging for Deep Neural Networks. ICML 2024.

[2] Schwettmann et al, "FIND: A Function Description Benchmark for Evaluating Interpretability Methods", 2023.

[3] Shaham et al, "A Multimodal Automated Interpretability Agent", 2025.

Weakness 3 - Limited novelty i.e. not introducing new metrics

Similar to how you acknowledge this is as “not necessarily a weakness”, we do not think this is a weakness but a conscious decision as we thought it would be a more valuable contribution to the field to rigorously analyze existing and standard statistical metrics instead of focusing on creating new ad-hoc metrics. This way our contributions are more clearly defined, and designing metrics particularly to do well on our tests risks overfitting them to these specific tasks, or worse tuning the tests themselves in a way that our new metrics would do well.

Other comments or suggestions 1- Examples

We have included an example with dataset inputs and concepts values Appendix Figure B.1., to improve clarity similar to your suggestion. Please let us know if you would find additional examples useful or have any additional suggestions.

Other comments or suggestions 2 - Typo

Thank you for pointing this out, we have corrected the typo in Line 1636 to: “… we defined the “correct” concept $t_k$ as the …”

Additional Experiments:

In response to other reviewer's comments, we have conducted additional experiments such as evaluating novel combinations of metrics, showing the robustness of our results to the choice of $\epsilon$, and showcasing that our results using a random subset as $c_t^-$ are similar to using a real semantic subset. We have also included a new figure showcasing the Missing and Extra Labels tests. These results are available at https://drive.google.com/file/d/1OHMxyMW1KVIzxd2Rd_Hx34qIjVecUVmo/view.

Summary

We are happy to hear that you find our work is addressing a crucial problem and produces logical and valuable contributions to help solve it. We believe we have addressed you main remaining concerns regarding labeling cost and missing citations and will include a discussion of these in the revised manuscript. Please let us know if you have any remaining questions and we would be happy to discuss them further.

Thank you for the detailed review and positive feedback!

Re: Additional Related Work:

Thank you for pointing out these references. While these references focus on a very different type of XAI, in particular local input-importance evaluations and as such cannot be applied to our setting, they contain important ideas about reliable evaluation of explanations and we will cite and discuss them in the related work(Appendix B.4) as follows:

The field of input-importance explanations has seen an evolution in the metrics used, with initial focus on finding the features that humans think are important. Later metrics such as deletion and insertion proposed by [1] allow for more principled evaluation of the explanation fidelity, i.e. whether it actually matches what the model in vision models, and [2] extends the deletion metric to natural language settings. [3] proposes the Remove And Retrain (ROAR) framework as an alternative method for evaluating the quality of input-importance explanations by evaluating whether a model retrained on data without the most important pixels can still solve the task. [4] propose additional checks, such as measuring whether an explanation method has high sensitivity, with the intuition that similar inputs should have similar explanations.

Overall, we think that this line of research [1]-[4] does not reduce the novelty of our contribution, as they are focused on a different problem and produce no actionable insight for our setting of global neuron-level explanations. In addition, these papers are mostly focused on introducing new evaluation metrics, instead of conducting meta-evaluation between existing metrics which is the focus of our work.

Additional Experiments:

In response to other reviewer's comments, we have conducted additional experiments such as evaluating novel combinations of metrics, showing the robustness of our results to the choice of $\epsilon$, and showcasing that our results using a random subset as $c_t^-$ are similar to using a real semantic subset. We have also included a new figure showcasing the Missing and Extra Labels tests. These results are available at https://drive.google.com/file/d/1OHMxyMW1KVIzxd2Rd_Hx34qIjVecUVmo/view.

Please let us know if you have additional questions and we would be happy to discuss them further! If not, we hope you consider updating your score as you have noted our work is comprehensive, clarifying/unifying and a worthwhile contribution to interpretability research with little weaknesses.

References:

[1] Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: randomized input sampling for explanation of black-box models. In BMVC, pp. 151. BMVA Press, 2018. URL http://bmvc2018.org/contents/papers/1064.pdf.

[2] Thomas Fel, Rémi Cadène, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. Advances in neural information processing systems, 34:26005–26014, 2021.

[3] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. NeurIPS, 32, 2019.

[4] Umang Bhatt, Adrian Weller, and José MF Moura. Evaluating and aggregating feature-based model explanations. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3016–3022, 2021.

Thanks for the detailed review! We have conducted extensive additional experiments available at: https://drive.google.com/file/d/1OHMxyMW1KVIzxd2Rd_Hx34qIjVecUVmo/view, see in particular Tab G5-G7 as these experiments were conducted to address your questions.

1.Realistic failure modes

We argue that most real world explanation failures are closely connected to one of two failure modes, and that these failure modes are directly captured by our tests:

• FM-A) Explanation too generic: This means the explanation concept is a superset of the “true” neuron concept, e.g. describing a neuron as “animals” when it only activates on dogs. Our Extra Labels Test captures whether a metric can detect this failure mode.
• FM-B) Explanation too specific: Explanation concept is a subset of real neuron activations, i.e. describing a neuron as “black cat” when it activates on all cats. Our Missing Labels Test captures whether a metric can detect this failure mode.

Also, our idea of a “concept” is very general and includes any text-based description. This means a single concept could be highly specific (e.g. “flying bird”), or a composition of simpler concepts (e.g. “water OR river”). The realistic failure modes you discussed then fit into the above two failure modes:

• Polysemanticity: A popular model of polysemanticity is to model neuron activations as an OR of different concepts. If the explanation only captures one of these concepts, this means the explanation is too specific (FM-B).
• Context specific activations: A context specific neuron activation means the neuron’s “true” concept is a subset of the non-context specific concept. If the explanation is not-context specific, the explanation is too generic(FM-A).
• Non activating inputs still contain the concept: This means that the explanation concept is too generic i.e. a superset of the “true” concept(FM-A).

2. Random vs semantic sub/supersets

Thanks for the suggestion! Our missing/extra labels tests are a mathematical model of semantic sub/superclasses that can be run without knowledge of the semantics or relationships between concepts. To test this model, we conducted a version of extra labels test on the ImageNet dataset where instead of randomly sampling $c^+$ we used the smallest superclass of the concept according to WordNet hierarchy, and a version of missing labels test where we used the largest subclass of the concept as $c^-$. As shown in new Table G.5, the results are essentially identical to using random sub/supersets, showcasing our random sub/superset is a good model of real semantic relationships for our purposes.

3. Importance of Epsilon Choice:

To clarify, as mentioned on lines 307-309, we normalize each metric so that their values lie in [0,1] before running the tests to ensure fair comparison. To measure the sensitivity of our results to the choice of eps, we ran our tests with different epsilons as shown in Tab G.7. We can see that changing eps by an order of magnitude does not significantly change which metrics pass.

Our theoretical results also suggest our tests are robust to choice of $\epsilon$. As we show in Tab. C.4 & D.4, the score difference of metrics that fail our tests approaches 0 as the concept/neuron activation frequency $\gamma$ approaches 0, while passing metrics retain a nonzero score difference regardless of $\gamma$. This means that for any $\epsilon > 0$ there exists $\gamma > 0$ s.t. metrics like accuracy will fail the tests.

4. Lack of expectation over $c^{-}_{t}$:

Great point. We will add an expectation over $c_t^{-}$ in Eq. 10 as this is a more principled definition. While our experimental results only used one sample of $c_t^-$, we conducted a study on how the results change with additional samples as shown in new Tab G.6. We can see averaging over multiple samples has little impact on the results, likely because we already average over a large number of neurons and settings.

5. Number of neurons

Thanks for pointing this out. For the theoretical missing/extra labels test, we simulate 1k neurons as mentioned in line 1129. For empirical Missing/Extra labels test, we evaluated a total of 5549 neurons, and for meta-AUPRC we evaluated a total of 2989 unique neurons.

6. Source of $c_t$

We would like to clarify we have specified the source of $c_t$ in the manuscript. For missing/extra labels tests, we use ground truth $c_t$, i.e. from a labeled dataset(line 301-302). For meta-AUPRC, we test with both ground truth $c_t$ from a labeled dataset as well as pseudo-labels from SigLIP(App. F.2, lines 1764-1782).

7. Additional references

Thanks, we’ll cite and discuss these references.

8. Performance on different modalities

As our main contributions are mathematical features of the metric itself and not tied to any particular model or modality, we did not observe significant differences between the modalities.

Please let us know if you have any remaining questions!

Thanks for the review!

Please see https://drive.google.com/file/d/1OHMxyMW1KVIzxd2Rd_Hx34qIjVecUVmo/view for our new experimental results, in particular Tables G1-G4 as those experiments were conducted to address your questions. Below we address your concerns and questions in detail.

Weakness 1 - Lacks analysis of some metrics that failed the tests

We would like to clarify that in Appendix C of the manuscript, we have conducted extensive theoretical analysis on why metrics fail the tests; and in our motivating example (Sec 4.1, table 2, Figure B.1) we showed why recall and precision fail the tests.

In particular, failing the extra labels test is caused by the metric not being able to distinguish between the correct concept (pet) and a superclass (animal), while failing the missing labels test is caused by the metric not being able to differentiate the correct concept (pet) from a subclass (dog/cat). Overall, our tests are a mathematically formalized measurement of whether a certain metric can differentiate between a correct concept vs sub/superclass of it. For many metrics, this only happens on imbalanced neurons/concepts that do not activate on most inputs as we show in tables C1-C4, which indicates failure is tied to poor metric performance on imbalanced data. Please let us know if you had a specific kind of analysis in mind that you would like to see.

Weakness 2 - Combinations of Metrics

Thank you for the suggestion! We would like to point out that our current result (Table 3, 4) already contains a combination of metrics, since F1-score is the harmonic mean of Recall and Precision: $F1 = 2/(recall^{-1} + precision^{-1})$. In the original manuscript, we also briefly discuss the idea of combining different metrics on lines 900-901 in the Appendix.

Following your comment, we have conducted additional experiments evaluating more extensively whether combinations of metrics can work as a good evaluation metric. Specifically, inspired by F1-score, we used the harmonic mean of other existing metrics to see how they perform. Full results are shown in Tables G1-G4 in our new results. We can see that many combinations of metrics achieve quite a good performance, and now pass the theoretical missing/extra labels tests. For example the harmonic mean of Balanced Acc and Inverse Balanced Acc performed well. In general, our initial results indicate that combining a metric that passes the extra labels test with a metric that passes the missing labels test will pass both tests. Conversely, combining two metrics that fail the same test, such as Recall and AUC will still fail that test.

Weakness 3 - Section 3: Evaluation Metrics can be classified

We are not sure which specific part the reviewer is referring to. Do you refer to framing neuron explanation as a binary classification problem in line 114? If so, we have discussed the details in Appendix A.2. We would be happy to discuss further if you have specific questions.

Weakness 4 - Intermediate layer explanations

We believe there might be some misunderstanding, our results and contributions are not limited to final layer neurons. Our proposed NeuronEval framework in Sec 3 and the theoretical missing/extra labels test (Tab. 3, Fig. 2, App. C) cover explanations of all neurons regardless of where they are, including hidden layers neurons, final layer neurons and even non-neuron units like directions in activation space. In addition, we did show intermediate or non final-layer results in our experiments, including:

• intermediate layer neurons (settings 2 & 4, Tab F1-F2),
• concept neurons in an intermediate layer of a concept bottleneck model (setting 5 in Tab. F3 & setting 7 in Tab. F.6)
• linear probes trained on hidden layer representations (setting 6 in Tab. F3 and setting 8 in Tab. F6).

Our main text results (Table 3, Table 4) are averaged across settings and contain these results.

Weakness 5 - “Input-based explanations” are vague

Thank you for pointing this out. We will change the term to Input-based neuron explanations to avoid confusion.

Theoretical claims - the theoretical part has some shortcomings.

Could the reviewer expand on what specific shortcomings in the theoretical part are? We would be happy to discuss and clarify further if you have any specific comments on shortcomings.

Summary

We believe that we have addressed your major concerns by clarifying our theoretical analysis and experiment results in the intermediate layer neurons (Weakness 1, 4), running additional experiments on combined metrics (Weakness 2) and clarifying our terminology (Weakness 5). We had a hard time understanding a few concerns (Weakness 3, Theoretical Claims), and we hope the reviewer can clarify these if concerns still remain. Otherwise, we hope you consider adjusting the rating if our response has addressed your concerns.