Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization

Dear Reviewer,

Thank you for dedicating your time and effort to reviewing our paper. We are glad that you find our approach holistic and evaluation meticulous. We carefully considered your suggestions and conducted additional experiments to improve our paper. Please find our point-to-point responses below.

---

Q1: The introduction contains some issues: The methods like ASH, React do not negatively impact the ID accuracy. The model can employ an unchanged classifier for the ID task, which only introduces a negligible computational cost.

A1: We apologize for the confusion. In practice, our claim that "ASH, ReAct decrease the model classification ability" is referred from the ASH paper [R1], where they clearly state that ReAct and their ASH methods lead to InD accuracy drops due to pruning. We would like to invite the reviewer to check Figure 2 in ASH paper [R1] for the details.

[R1] Extremely Simple Activation Shaping for Out-of-Distribution Detection. ICLR, 2023. https://openreview.net/pdf?id=ndYXTEL6cZz

Despite the above clarification, we do agree with the reviewer that using an unchanged classifier can be an efficient solution. However, it is also important to recognize that the success of this solution hinges on the strict assumption that only later layers are utilized by neuron pruning. This restricted perspective may largely limit the potential of neuron-based methods, since shallow layers also offer valuable information, as demonstrated in Table 4 of our paper. Therefore, it is imperative to approach this solution with caution and careful consideration.

To further address your concern, we have added a footnote in our Introduction to clarify this point: "While it may be argued that maintaining neuron outputs for double-propagation preserves InD accuracy with low computational cost, it relies on the assumption that only later layers are utilized in neuron pruning, thus undermining the potential of these neuron-based methods."

Lastly, we would also like to remind the reviewer that our NAC-UE significantly outperforms ReAct and ASH on three benchmarks. For your convenience, we provide their averaged AUROC results below. The scores are sourced from our Table 1 and 2, where ReAct and ASH are officially implemented by OpenOOD.

[To be continued]

Dear Reviewer,

Thank you for dedicating your time and effort to reviewing our paper. We sincerely appreciate your positive remarks regarding the writing, novelty of the method, motivation, and robustness of our evaluations. We carefully considered your suggestions and conducted additional experiments to improve our paper. Please find our point-to-point responses below.

---

Q1: Given the time-consuming process of using KL gradient, computational cost of proposed method with other SOTA should be reported.

A1: Thanks for your valuable comment! Following your suggestion, we have conducted a series of experiments to analyze the efficiency of our method. To provide a comprehensive comparison, we selected the top-3 performed methods from Table 2 as baselines, and compared them with our NAC-UE in terms of preprocessing and inference time on ImageNet. For consistency, we used the same software and hardware configurations as in our previous experiments, which are detailed in Appendix F.

From the above results, the following observations can be made:

1. While NAC-UE requires more inference time as more layers are considered, it significantly reduces preprocessing time compared to ViM and SHE, e.g., 7.75s (NAC-UE) vs. 1019.34s (ViM). This finding coincides with our previous experiments (Appendix G.1), where we show that NAC-UE achieves favorable performance despite utilizing only 1% of the training data for NAC modeling.

2. Remarkably, even when utilizing just a single layer (layer4), NAC-UE outperforms SoTA methods in terms of both inference time and detection performance. For example, NAC-UE achieves an AUROC of 94.57 (39.63s inference time), whereas GEN achieves only 89.76 on AUROC with 43.33s inference time. This highlights the efficiency of our approach.

Additionally, it is also worth noting that there are numerous ongoing research efforts dedicated to facilitating gradient calculation [R1, R2], which could potentially complement our proposed method. We will include the above experiments in Appendix G.

[R1] Low Complexity Gradient Computation Techniques to Accelerate Deep Neural Network Training, TNNLS, 2021.
[R2] Acceleration of DNN Backward Propagation by Selective Computation of Gradients, DAC, 2019.

---

Q2: The proposed metric is extended for uncertainty estimation, while no quantification or evaluation (calibration of uncertainty) on this dimension.

A2: We are so sorry for the confusion. In line with many prevalent OOD uncertainty estimators (e.g., Gram [R3], ReAct [R4], MOS [R5]), our uncertainty estimation mainly serves for OOD detection problems: predicting whether a test sample is from OOD or not. Thus, our experiments primarily emphasize OOD detection performance rather than calibration errors, similar to the aforementioned studies.

In response to your suggestions, we have conducted additional experiments to further explore the potential of our NAC-UE. Our experiments carefully align with [R6], where two calibration error measures (RMS and MAD) are employed. For the calibration evaluation, we utilized a pretrained model on the CIFAR-10 dataset as the foundation, and assessed the calibration errors on both InD and OOD test data. We mainly compare NAC-UE with two simple baselines: MSP (Hendrycks & Gimpel, 2017) and Temperature (Guo et al., 2017).

From the above results, we can see that our NAC-UE significantly outperforms two baseline approaches, which demonstrates its potential in prediction calibration. We will include the above experiments in Appendix G.

[R3] On the Importance of Gradients for Detecting Distributional Shifts in the Wild. NeurIPS, 2021
[R4] ReAct: Out-of-distribution Detection With Rectified Activations. NeurIPS, 2021
[R5] MOS: Towards Scaling Out-of-Distribution Detection for Large Semantic Space. CVPR, 2021
[R6] Deep Anomaly Detection with Outlier Exposure. ICLR, 2019

[To be continued]

---

Q3: How does the choice of rank correlation instead Pearson correlation might affect the results?

A3: Great question! The choice between rank correlation (RC) and Pearson correlation (PC) can have an impact on the results. More specifically, as supported by [R7], RC is more suited to our situations.

RC measures the monotonic relationship between two variables, while PC assesses the linear relationship. In our situations where the target is to select the best model based on InD evaluation criteria, the monotonic relationship becomes crucial [R7]. To gain a clearer understanding, let's consider the following example:

A (Test Accuracy): [0, 0.7, 0.6, 0.5];
B ( Val. Accuracy): [0, 0.8, 0.8, 0.9],

Here, sets A and B consist of test and validation accuracy at each epoch, respectively. PC between A and B is 0.93, while RC is 0.3. If we rely on PC, we might conclude that validation accuracy is highly correlated with test performance, and safely choose the best model with the highest validation accuracy (0.9). However, such a model only achieves a test accuracy of 0.5, contradicting our expectations. This example highlights the limitation of PC in our scenario, where the linear relationship could be misleading.

In addition to the above toy example, we have also conducted an analysis by measuring PC on Vit-b16 model. As exhibited below, results mostly show surprisingly high PC values, i.e., 0.98/0.99 on three datasets. This contrasts sharply with our RC results in Table 8, where RC is hardly larger than 0.5. As the performance gap between the oracle-selected and validation-selected/NAC-selected model consistently exists, this again leads to the deduction that Pearson correlation is not as suitable as rank correlation in our situations.

[R7] Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization. NeurIPS 2022.

---

Q4: How does this metric generalized to regression problems?

A4: Thank you for this insightful question. Our proposed metric NAC, which focuses on neuron distributions, can indeed be generalized to regression problems without any changes to its definition. The only update required is about the formulation of neuron states. Since the current formulation involves KL divergence which is not suitable for regression problems, we need to simplify the neuron states from $z \odot g'(z) (p-u)$ to $z \odot g'(z)$. This simplified version corresponds to the Input $\times$ Gradient explanation, which is also meaningful in explaining neuron behaviors and provides insights into the model status.

Dear Reviewer,

Thank you for dedicating your time and effort to reviewing our paper. We are glad that you think our method is well-motivated and presents strong improvements. We took your suggestions very seriously, and have conducted additional experiments to update our paper. Please see below for our point-to-point responses.

---

Q1: For NAC-UE uncertainty estimation, the paper averages coverage scores across layers when applying to multiple layers. Is it possible to weight the contribution of different layers especially given that earlier layers tend to be more general?

A1: Nice suggestion! Inspired by your question, we have conducted a series of experiments on the CIFAR-10 benchmark, where we randomly searched the weight for each layer within the same space: [0.2, 0.4, 0.6, 0.8, 1.0]. We found that our NAC-UE can be further improved in this weighted version (e.g., 2.47% gain on average FPR):

What's interesting is that we noticed assigning larger weights to the deeper layers often results in better performance. We conjecture this is due to that deeper layers often encode richer semantic information than shallow layers, making them crucial in detection problems. We will include such experiments in Appendix G.

---

Q2: Missing a related work that shares a similar motivation of leveraging neuron behaviors. [1] uses neural mean discrepancy of neuron activations for OOD detection. Discussing relations to such works could provide additional insights.

A2: Thank you for pointing out the missed related work. We will cite the paper and add a discussion section in Appendix H. Here are the main differences outlined:

1. [1] primarily investigates the raw neuron output, while our paper centers on a new formulation of neuron states. This formulation can be decoupled as the neuron gradients, neuron output, and model prediction deviations, thus offering a fresh interpretation of neurons in OOD scenarios.
2. Our NAC specifically focuses on the distribution of neuron states, while [1] mainly examines the mean of neuron output. This distinctive perspective makes our NAC more comprehensive and superior in understanding neuron behaviors.
3. While Neural Mean Discrepancy [1] may effectively detect OOD samples, it requires an additional classifier during the inference phase. Instead, NAC directly calculates the coverage scores in a parameter-free manner, serving as an efficient measure for both OOD detection and generalization.

[1] Dong, Xin, et al. "Neural mean discrepancy for efficient out-of-distribution detection." CVPR, 2022

---

Q3: Is it possible to directly train or regularize models based on NAC to improve robustness. For example, one could minimize the entropy of NAC distributions to encourage balanced coverage. Evaluating such NAC-driven training schemes could further validate its usefulness.

A3: Yes, it is possible! Based on your suggestion, we have tried to regularize the model using our NAC entropy loss: $H(\mathbf{z})= -\sum_{i=1}^{N}p_i(z_i) \log{p_i(z_i)}$, where $p_i$ associates with the NAC distribution of $i$-th neuron. We conducted experiments on the PACS dataset, and the results are shown below:

Interestingly, we find that maximizing NAC entropy leads to improved performance. This finding also aligns with the intuitive understanding presented in [2]. By maximizing NAC entropy, we encourage the activation of neurons in unexplored regions over NAC distribution, thus diversifying the neuron activities and improving the model robustness. Conversely, minimizing entropy may result in collapsed neuron behavior. We will include the above experiments in Appendix G.

[2] Dubey, Abhimanyu, et al. "Maximum-Entropy Fine-Grained Classification". NeurIPS 2018.

Dear Reviewer,

Thank you for dedicating your time and effort to reviewing our paper. We are glad that you think our method is novel, effective, and practically applicable. We took your suggestions very carefully, and have conducted additional experiments to update our paper.

Clarification: Before diving into detailed responses, we hope to firstly clarify one fundamental aspect of our NAC, which seems misunderstood by the reviewer. That is, our NAC-based methods all operate in a post-hoc fashion, which do not involve any training process. Specifically, for OOD generalization, our NAC-ME is designed to assess model robustness for the purpose of best model selection, rather than directly improving OOD performance through optimization.

---

Q1: The definitions of OOD are different for OOD Detection and OOD Generalisation...... so why it is also applicable for OOD generalization.

A1: Thank you for this insightful comment. The response to your valuable question can be three-fold:

1. The overlap between OOD detection and generalization exists: While we agree with the reviewer that OOD detection/generalization mainly centers on semantic shift/covariate shift, the overlap between these two problems indeed exists. Previous studies have also explored covariate shift for OOD detection [R1, R2, R6], as well as semantic shift for OOD generalization [R3, R4]. Given such situations, it seems less suitable if we consider these two problems as entirely separate. We will include a discussion section in Appendix H.

2. NAC benefits from data-centric modeling (Reason 1): Our NAC method is rooted in a data-centric approach, leveraging the neuron distributions within InD training data to characterize model status. This data-centric modeling enables NAC to effectively capture the intrinsic patterns and characteristics of the model (i.e., from a neuron level), thus serving as an effective tool for model robustness evaluation (i.e., applicable to OOD generalization). This also aligns with the principles of network quality assessment in system testing [R5].

3. Shallow to deep layers account for covariate and semantic shifts (Reason 2): As noted by [R6], shallow layers in models often closely correlate with the image style information (covariate level), while deep layers capture semantic information. Since our NAC often works by leveraging multiple layers spanning from shallow to deep, it naturally accounts for both covariate and semantic shifts.

[R1] Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data. NeurIPS workshop, 2021.
[R2] Unified Out-Of-Distribution Detection: A Model-Specific Perspective. ICCV, 2023
[R3] Overcoming Concept Shift in Domain-Aware Settings through Consolidated Internal Distributions. AAAI, 2023
[R4] NICO++: Towards Better Benchmarking for Domain Generalization. CVPR 2023
[R5] NPC: Neuron Path Coverage via Characterizing Decision Logic of Deep Neural Networks. ACM Trans. Softw. Eng. Methodol. 2022.
[R6] Full-Spectrum Out-of-Distribution Detection. IJCV, 2023.

---

Q2: There are many dimensions that are useless, and even worse......there may exist misleading information and redundant message when using NAC to compute the activated rates, of which the performance cannot be guaranteed.....

A2: We appreciate this valuable comment, and concur with the reviewer regarding the possible redundancy and noise in multiple dimensions/neurons. Nonetheless, it is still worth noting that there are also numerous neurons that are meaningful and can encode significant information [R7, R8]. This observation underscores the viability of neuron-based methods (which is also supported by our Figure 4 and 5).

On the other hand, our NAC method actually goes beyond considering individual neurons alone. It models the neuron distributions on InD data, and leverages averaged statistics of neuron behavior to reflect model status. This also aligns with the widely adopted statistical method in many network testing methods (e.g., [R9, R10]), which allows for the modeling of more abundant information.

Despite the above clarification, we do acknowledge that filtering noisy neurons in NAC-based methods can be a viable strategy to further improve the performance. But we still want to remind the reviewer that our NAC method has already demonstrated strong results on three benchmarks for OOD detection (i.e., outperforming 21 baselines), and four datasets for OOD generalization. This has showcased the effectiveness and robustness of our approach.

[R7] Understanding the role of individual units in a deep neural network. PNAS, 2020
[R8] Interpreting Deep Visual Representations via Network Dissection. TPAMI, 2018
[R9] DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems. ASE, 2018
[R10] DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP, 2017

[To be continued]

---

Q3: When it comes to OOD generalization, the authors suggest use NAC-based learning objective to improve OOD performance. It seems that it just makes the embedding features to be sparse, which has been discussed in previous works in OOD generalization, such as [2]. Moreover, the gradient-based learning objective can be computational hard, since the second order gradients are needed to be computed in each optimisation step.

A3: We are so sorry for the confusion. As we clarified earlier, our NAC-based method specifically targets the evaluation of model robustness for OOD generalization, which actually does not involve optimization processes. Thus, we do not need to calculate 2nd-order gradients, which saves computational cost.

On the other hand, NAC also differs from [2] in the following aspects:

1. As noted, unlike the approach in [2] which concentrates on refining model training, our NAC focuses on the evaluation of existing models, thus providing a different perspective.

2. Drawing parallels with system testing coverage criteria, NAC tracks neuron behaviors in the whole network. In contrast, feature sparsity, as addressed in [2], is primarily concerned with feature representation, specifically identifying areas where most features are zero or irrelevant. Hence, these two methods differ in their measurement and targets.

We will cite [2] and add a discussion in Appendix H.

[2] Sparse Invariant Risk Minimisation. ICML, 2022

---

Q4: Detailed discussion about the hyper parameter setups should be given, especially for the hyper parameter tuning. More ablation studies should be given to test the respective power of z \times g'(z) and p-u.

A4: Thank you for this valuable comment. In practice, our Appendix D.4 and E.4 have already provided the details for all employed hyperparameters.

To further address your concern, we have added more ablation studies to analyze the respective power of z, g'(z), and p-u. We provided the results on ImageNet below. Firstly, it can be drawn that z*g'(z)*(p-u) does perform best among all variants of formulations, showing its superiority. Moreover, arbitrary combinations of z, g'(z), and p-u can lead to improvements compared to using a single component alone. For instance, utilizing z*(p-u) yields better performance than using either z or p-u in isolation. These findings suggest that all three components encode meaningful information in OOD scenarios, further supporting the rationale behind our proposed neuron states. We will include these experiments in Appendix G.

Dear reviewer,

It'd be helpful if you could provide feedback and engage with the authors further. Let me know if you need any help.

AC