Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement

We express our appreciation to reviewer 5qNg for your insightful review and comprehensive suggestions.

• Incorrect Motivation: ID-OOD accuracy trade-off:

Thanks for highlighting this confusion. Indeed the ID accuracy drop can be avoided through a separate classifiers with minimal computational cost. We will emphasize this point more in the introduction. We note that ID-OOD accuracies trade-offs are also reported in ReAct Table 2 and DICE Table 5, and further explored in ASH where they argue that an ideal OOD detection pipeline "should not deteriorate ID task performance, nor should it require a cumbersome parallel setup that handles the ID task and OOD detection separately". We will clarify this point. We note that our motivation is not only achieving a better tradeoff between OOD detection and ID classification but also achieving higher OOD detection performance than others. We do this in a principled way through theoretical analysis, giving explanations and insights into the working mechanisms of existing works before making our proposal.

• Limited Novelty of SCALE and ISH:

Thank you for your comments. The main contribution is providing a theoretical understanding of the current state-of-the-art method ASH. Our development of SCALE is directly born out of this analysis and verifies that it is only the scaling operation which benefits OOD detection (as pointed out by reviewer AeGx). Furthermore, we propose to leverage scores as ID-ness in extra training, which further improves OOD detection without the need for extra OOD validation data. For experimental results. Compared to ASH-S, we achieved 1.73+ on near-OOD, +0.06 on far-OOD, and +0.67 on ID accuracy for SCALE, and achieved 4.83+ on near-OOD, 0.32+ on far-OOD, and 0.56+ on ID accuracy for ISH+SCALE. We note that current methods are already good for easier far-OOD detection so the improvement room is smaller.

• Unsubstantiated Claim on ReAct:

Thanks for your suggestions. Our statement in the introduction is that ”the lower-part pruning approach, in contrast to ReAct, hinders the OOD detection process”. ReAct is a higher-part pruning. In our original paper, we provide an analysis that ASH pruning impedes the OOD process because lower-part pruning brings ID and OOD data closer in the logits space. For ReAct, higher-part pruning enlarges the distance between ID and OOD and this provides an explanation for its effectiveness.

• Minimal Performance Gain and combination of ISH with other OOD detector:

Thanks for your suggestions. For the experiments results, ISH + SCALE improves an additional notable 1.34 on SCALE, which has 21% improvement than AugMix with less training efforts. We provide results for ISH + MSP/ODIN/Energy/ReAct/ASH in the following table and show that ISH increases all results.

• Ablation Study for the placement of SCALE :

We appreciate your valuable suggestions and provide the ablation study on the following table:

As the stage becomes shallower, the performance decreases. This is because the early stages tend to learn low-level features that are common to both the ID and OOD datasets.

We extend our gratitude to RAx2 for the insightful review and suggestions.

• LSUN-Crop and LSUN-Resize results: Thanks for pointing this out. In our revision, we will remove LSUN-Crop and LSUN-Resize from the main text and place the original table in the Supplementary for consistent comparison to ODIN; and provide discussion to give awareness to the reader.

• Typos and mistakes: Thank you for pointing out these. We will fix them in the revision and proofread them in the future carefully.

• Explanation of ReAct: Our analysis is relevant to ReAct, a higher-part pruning approach. In our original paper, we provide an analysis that ASH pruning impedes the OOD process because lower-part pruning brings ID and OOD data closer in the logits space. For ReAct, higher-part pruning enlarges the distance between ID and OOD and this provides an explanation for its effectiveness.

We thank reviewer DQbC for your insightful review.

• The method is tested only in Imagenet: Table 3 shows experiments on CIFAR-10 and CIFAR-100.
• Other Data Domains:
  Our method, like prevailing ad-hoc methods for OOD (EBO, ReAct, ASH) are intended for already-trained networks by design. The readily available off-the-shelf image backbones is why we and others work primarily in the image domain, extending it to other domains like time series and tabular data which do not have analogous backbones will likely need fundamental changes in the assumptions and approach beyond our current scope; we leave this for future work. We strongly support the point that OOD for other domains is a worthwhile focus and we leave it for future research.

EBO: Liu, Weitang, Xiaoyun Wang, John Owens, and Yixuan Li. "Energy-based out-of-distribution detection." Advances in neural information processing systems 33 (2020): 21464-21475.

ReAct: Sun, Yiyou, Chuan Guo, and Yixuan Li. "React: Out-of-distribution detection with rectified activations." Advances in Neural Information Processing Systems 34 (2021): 144-157.

ASH: Djurisic, Andrija, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. "Extremely simple activation shaping for out-of-distribution detection." arXiv preprint arXiv:2209.09858 (2022).

We extend our gratitude to RAx2 for the insightful review.

• Algorithm for ISH pipeline: Thanks for your suggestion. We provide pseudo-code of the algorithm here and will add it to Sec. 4.3 in the revision.

function ISH(pretrained_model M, data D, learning_rate L, epochs I)
	for input, label in D: 
		# forward pass get predicted results and intermediate activations.
		activation_1, ...,  activations_p, predicted = forward_pass(M, input)
		gradient_z = loss(predicted-input)
		
		# get scaling s for the activations of for penultimate layer:
		s = SCALE(activations_p) # as defined in equation (7)
		
		# for penultimate layer:
		# scale activations_p by s for gradient computation
		gradient_weights = dot_product(s  * activations_p^T, gradient_z)
		gradient_biases = gradient_z
		
		update_parameters(gradient_weights, gradient_biases, learning_rate) 
		
		update other parameters
		
		return weights, biases

• Applicability to Transformers: Our current assumption and analysis is made in light of the ReLU because ResNet 50 is still the prevailing architecture of choice for other works in OOD [EBO, ReAct, ASH]. However, similar ideas could be applied to transformers, and we are looking into this for future work.

References

EBO: Liu, Weitang, Xiaoyun Wang, John Owens, and Yixuan Li. "Energy-based out-of-distribution detection." Advances in neural information processing systems 33 (2020): 21464-21475.

ReAct: Sun, Yiyou, Chuan Guo, and Yixuan Li. "React: Out-of-distribution detection with rectified activations." Advances in Neural Information Processing Systems 34 (2021): 144-157.

ASH: Djurisic, Andrija, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. "Extremely simple activation shaping for out-of-distribution detection." arXiv preprint arXiv:2209.09858 (2022).