Towards Optimal Feature-Shaping Methods for Out-of-Distribution Detection

We sincerely thank you for the time and effort you invested in reviewing our paper. We are grateful for your acknowledgment of its well-crafted composition, the significance of the problem it addresses, and its demonstration of superior results compared to current state-of-the-art methods.

We compare our proposed method and others, including training a model with outlier exposure and using contrastive learning objectives. These training methods are not designed for a large-scaled dataset like ImageNet-1k. Thus, our experiments are conducted on the ImageNet-200 benchmark with ResNet18, following OpenOOD V1.5 [1].

We emphasize that this is not a fair comparison, because our method is designed for post-hoc OOD detection, assuming no access to OOD data and no extra model training process. But as shown in the table, there are no significant differences between different methods. Training methods do not show a significant advantage over the post-hoc methods, as also observed by OpenOOD V1.5 [1].

Due to the time constraints associated with the rebuttal, we are currently unable to carry out very extensive experiments to compare post-hoc OOD methods with training methods. However, exploring the performance of training methods across various benchmarks and model architectures is a promising direction that we look forward to investigating.

Bold numbers are superior results whereas Italic numbers denote the second and third best results.

[1] Zhang, Jingyang, et al. "OpenOOD v1. 5: Enhanced Benchmark for Out-of-Distribution Detection." arXiv preprint arXiv:2306.09301 (2023).

We greatly appreciate your time and effort in reviewing our paper. We are pleased that you acknowledged the effectiveness of our method across diverse model architectures.

Q1: What is the motivation to introduce ISFI?

We aim to optimize the multiplicative function $\theta(z)$ in Eq. 4, which models elementwise feature-shaping functions. The primary challenge arises from the continuous nature of the feature value $z$, complicating the optimization problem. To address this, we introduce ISFI, a strategic discretization facilitating the assessment of feature importance for OOD detection. Specifically, we partition the domain of features into a disjoint set of intervals, which allows us to approximate the original function $\theta(z)$ with a piece-wise constant function, thus simplifying the optimization problem.

Q2: What is the meaning of Figure 2?

Fig. 2 compares our optimal piecewise constant shaping functions with previous methods, showing that these prior works are empirically approximating our function optimized with access to OOD data.

The methods shown in Fig. 2(a-c), including ReAct, BFAct, and VRA-P, are element-wise shaping functions. These functions identically treat each element of the feature vector, simplifying the comparison.

For example, given any element $z$ of feature representation $\boldsymbol{z}\in \mathbb{R}^M$, our shaping function is $\bar{z}=\theta(z)\cdot z$, where the function $\theta$ is discretized with $K$ parameters. ReAct applies an element-wise shaping function $\bar{z}=\min(z,t)=\frac{\min(z,t)}{z}\cdot z$ to each element. We do not need to compare the two methods on each of the $M$ features, but instead, we compare the two functions $\theta(z)$ and $\frac{\min(z,t)}{z},z\in\mathbb{R}$.

Besides, ASH families in Fig. 2(e-h) incorporate the entire feature vector and are not reducible to element-wise functions. As an approximation, we estimate their average shaping effects across all samples.

Q3: Other feature shaping methods such as ReAct and ASH do not have other parameters to be optimized.

We respectfully argue that this point actually leads to a disadvantage of ReAct and ASH compared with our method. The shaping functions of ReAct and ASH are manually designed for specific model architectures and OOD datasets. This eliminates the need for additional parameters but consequently limit their generalization ability. In contrast, while our shaping function introduces more parameters, it can be efficiently optimized in closed form based on different models and ID data. Additional parameters do not increase user burden; instead, they bring greater flexibility and generalization ability to our method (Tables 1&2).

Q4: Can authors discuss the comparable performance claimed by ReAct and ASH?

There might be some typos in your comment. Here we discuss the generalization ability claimed by ReAct and ASH over different distributions. Please engage with us in discussion if your comment is asking something different.

The experiments conducted in the ReAct and ASH papers involve a narrower range of models and OOD datasets compared to our study. This limitation restricts the evaluation on the generalization ability of their methods.

For instance, when using ImageNet-1k as the ID data, both ReAct and ASH restrict their experiments to two model architectures and four OOD datasets. In contrast, our research encompasses a more extensive evaluation, employing eight different model architectures and eight OOD datasets. Note that the models and OOD datasets used in the ReAct and ASH studies are also included in our experimental setup.

This broader scope in our study allows for a more robust assessment of the generalization ability of our method, significantly contributing to the reliability and applicability of our findings.

I thank the authors for their response.

Q3: Other feature shaping methods such as ReAct and ASH do not have other parameters to be optimized.

I respect the authors' comments. I agree with the advantage of the proposed method as the authors mentioned. I think that introducing more parameters is a clever approach.

Q2: What is the meaning of Figure 2?

I might have misunderstood the figure at first. I appreciate the authors' comments. I have a question about the interpretation of this figure. The authors said that the proposed method flips the sign at low-value features. Why do the low-value features are flipped? I think that it is an interesting phenomenon although these parameters are determined by the optimization problem.

Thank you for your response!

Visualization to show "low-value features often align with negative weights"

We apologize for any confusion caused by the phrase "our experiments revealed" in the last reply. The referred experiments are not currently included in our paper. They were designed to empirically explan the specific form of the optimized function, based on a specific model architecture and ID dataset. Note that our optimized function may vary across different model architectures and datasets, so the findings, such as the tendency for low-value features to align with negative weights, might not be universally applicable.

However, in light of your recommendation, we recognize the value of including an additional discussion in the supplementary materials. This will cover the characteristics of the optimal function and provide corresponding empirical explanations. We plan to update the materials accordingly soon.

Whether K-dim ISFI improves the OOD performance

Previous feature-shaping methods predominantly focus on manipulating features based on their values, instead of feature indexes within representations. For instance, ReAct employs truncation for features exceeding a certain threshold, while ASH families disregard features with low values.

Aligned with prior research, our paper seeks to refine the shaping function $\theta$ based on feature values. The introduction of ISFI emerges from the challenges posed by the continuous nature of feature values. By dividing the feature values into separate intervals, ISFI enables us to approximate $\theta(z)$ with a piecewise constant function, thereby simplifying the optimization process. Thus, the primary goal of proposing ISFI is not to enhance OOD performance but to serve as a crucial step in our modeling.

Dear Reviewer c5SY,

This is a kind reminder that we have updated the supplimentary materials (attached at the end of our manuscript). We added an additional section to discuss the specific form of our optimized shaping function as you suggested. Please refer to Section C where new content have been highlighed in red.

Warm regards,

The Authors

Improve the guide of Section 3

Thank you for your suggestion to improve the readability of this part. We revise the guide of Section 3. The revised version is as follows:

In this section, we propose a specific instance of the general formulation described in Section 2, following previous element-wise feature-shaping methods (Sun et al., 2021; Xu & Lian, 2023; Kong & Li, 2023). To optimize the shaping function, we initially examine which feature value ranges are effective for OOD detection. However, a significant challenge arises from the continuous nature of feature values. To address this, we partition the domain of features into a disjoint set of intervals, which allows us to approximate the optimal shaping function with a piece-wise constant function. With this approximation, we can formulate an optimization problem to maximize the expected difference between the maximum logit values of ID and OOD data.

Subsequently, we show that the optimized shaping function using OOD samples bears a striking resemblance to prior methods, such as ReAct (Sun et al., 2021), BFAct (Kong & Li, 2023), and VRA-P (Xu & Lian, 2023). This similarity sheds light on the mechanics of these methods.

Finally, we propose a formulation that does not require any OOD data to optimize and admits a simple, closed-form solution. Experiments will show its robustness across various datasets and models.

In your comment, steps 2-5 are basically correct. We did not add step (1) in the guide. As previously mentioned, this step is actually an ad-hoc observation after we have solved our optimization problem, and is specific to a certain model and dataset.

A kind reminder that these editorial changes you are asking for are not related to the technical contribution of our paper. We can improve the readability of the paper further as you, other reviewers and AC suggest in camera-ready version if it is accepted.

Show the advantage of the proposed method across different tasks beyond classification

Different from these earlier studies [R1-4], we extensively expand the range of model architectures and datasets in our experiments, and conduct a thorough comparative analysis between our proposed method and previous ones. The results have shown the improved generalization ability of our method across a large variety of datasets and model architectures.

Previous research [R1-4] and our own work, primarily focus on classification, which is a key and fundamental task in CV. Regarding your suggestion to apply these techniques to other tasks such as object detection and semantic segmentation, it indeed presents an intriguing avenue for exploration. Given the limited time remaining in the discussion period (less than 7 hours), we propose to leave this as a potential future research direction.

Again, thank you for your taking interest and thinking about how to improve the paper.

We greatly appreciate your time and effort in reviewing our paper. We are pleased that you acknowledged our work's contribution in addressing a significant task, proposing a generalizable method without OOD data, providing strong theoretical and experimental support, and clear writing.

Q1: Is there any discussion or analysis about the robustness of the proposed method to thresholding.

There are two thresholds in the paper. One is the threshold for classifying ID and OOD samples based on OOD scores. Note that AUC already reflects performance robustness across all thresholds. The other pertains to setting feature value limits for optimizing our shaping function, as detailed in Section 3.1. Here we discuss the robustness of our method to the latter.

Given that feature distributions typically exhibit long-tailed characteristics, we select the 0.1 and 99.9 percentiles of all features in a training set as the lower and upper limits of feature values, respectively, to optimize our shaping function. Here we validate different choices of percentiles on the performance of our method, on the ImageNet-1k benchmark with ResNet50. The results are shown as follows:

Our method exhibits a insensitivity to the chooses of lower and upper limits. Although in the main experiments (Tables 1 & 2) we consistently use the 0.1 and 99.9 percentiles, exploring alternate settings reveals potential for slight improvements.

These experiments will be incorporated into the revised paper if space allows or updated in the Supplementary Material.

Q2: Further explanation is needed as to why $E_{\boldsymbol{x}^{\text{OOD}}\sim D_{\cal X}^{\text{OOD}}}\left[{\boldsymbol{I}(\boldsymbol{z}^{\text{OOD}})}\right]$ can be ignored.

In the general setting where we have no prior knowledge about the OOD data encountered during deployment, the distribution of $\boldsymbol{I}(\boldsymbol{z}^{\text{OOD}})$ is unknown, rendering the problem in Eq. 12 intractable. However, if we ignore the unknown term $E_{\boldsymbol{x}^{\text{OOD}}\sim D_{\cal X}^{\text{OOD}}}\left[{\boldsymbol{I}(\boldsymbol{z}^{\text{OOD}})}\right]$, Eq. 12 can be easily solved in closed form and our method shows improved performance for OOD detection. We have some possible working theories on why we can just ignore it (Fig. 3), which are supported by the experiments (Section 4.2).

Our working explanation is based on the similarities or differences in feature representations between these two types of samples.

1. In the Case of Hard OOD (High Similarity to ID): When OOD samples closely resemble ID samples, their feature representations tend to align. This similarity implies that the expectations we calculate for both OOD and ID samples point in similar directions.
2. In the Case of Easy OOD (Divergence from ID): Conversely, when OOD samples are markedly different from ID samples, the maximum logits (a measure of model confidence) for these OOD samples are typically lower. This lower confidence translates to a reduced magnitude of the expectation value for OOD samples compared to ID samples.

Thus, the aspect of $E_{\boldsymbol{x}^{\text{OOD}}\sim D_{\cal X}^{\text{OOD}}}\left[{\boldsymbol{I}(\boldsymbol{z}^{\text{OOD}})}\right]$ that is orthogonal to the ID expectation is minimal, and the term can be ignored with a small affect on the optimized results.

If there are specific concerns or areas that seem unclear, please highlight them and engage with us in the discussion for a more targeted explanation.

Q3: Is there a more intuitive explanation for this method that would make it more understandable?

Thank you for your question. We write a new explanation, trying to make it easy to understand and planning to revise the paper accordingly. If certain aspects or points are not entirely clear or if you have particular concerns, please point them out for a more focused clarification.

Feature-shaping methods for OOD detection aims to reshape feature representations to better differentiate between ID and OOD samples. Existing methods usually employ shaping functions manually designed for specific model architectures and OOD datasets. In this paper, we want to explore "what feature-shaping function is optimal for OOD detection", focusing on element-wise feature-shaping functions like ReAct. We model this as an optimization problem - optimize the shaping function to maximize an objective function designed for OOD detection. To make the optimization problem tractable, we do two simplifications. The first is to make the shaping function piecewise constant. The second is to use a optimization objective function without OOD terms based on some working theories. The two simplifications lead to a close form solution and both are validated by experiments. Our method improves the generalization ability of OOD detection across a large variety of datasets and model architectures.

Q4: Will the code and data be open source?

Yes. We have included our code as supplementary material alongside this submission. We are also committed to releasing the code on GitHub upon the official acceptance of our paper. Additionally, all the data utilized in our experiments is already open-sourced. To further assist, we will be adding a tutorial in our code repository, detailing the preparation process for the data.