Learning to Shape In-distribution Feature Space for Out-of-distribution Detection

We deeply appreciate the reviewer's dedicated time and effort in evaluating our work. In response to the insightful comments provided, we have provided detailed responses to each point raised, hoping that our responses can adequately resolve your concerns. Please find our responses below.

Q.1. How quickly does this method converge?

A.1. Thanks for your insightful idea. As can be seen from the loss trajectory shown in the uploaded PDF file, training with the same epochs as prior works [a,b] is sufficient for our method to converge. We will highlight the nature of our method following your valuable comments.

[a] Learning with Mixture of Prototypes for Out-of-Distribution Detection, ICLR 2024

[b] How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection, ICLR 2023

We sincerely appreciate your dedicated time and effort in reviewing our work. In response to the valuable comments provided, we have provided detailed responses to each point raised, hoping that our responses can adequately address your concerns. Please find our responses below.

Q.1. More clarification regarding the inconsistency issues

A.1. We kindly note that the inconsistency stems from that, under the batch-based training scheme, the model used in the E-step of the $m$-th iteration is trained without the data used in the M-step of the same iteration. In particular, the prediction $p_{\boldsymbol{\theta}}(k|\mathbf{x})$ is only updated once per training epoch while the discriminative model $f_{\boldsymbol{\theta}}$ keeps updated throughout mini-batches (iterations) in the training.

We sincerely appreciate your time and effort in reviewing our work. Below are detailed responses to your valuable comments.

Q.1. Although the proposed method is claimed to mitigate the assumptions in previous works, the model convergence still relies on prior works' assumptions (vMF). Even though the assumptions do not influence the geometry of the latent space, the proposed method is still not viable without these techniques.

A.1. We apologize for the misunderstanding. We will add the following explanation to highlight our motivation.

• Previous work assumes $p_{\theta}(x)$ (denoted as p_x in the left) to be the vMF distribution without theoretical guarantee. The mismatch between the assumed and learned distributions will degrade detection performance.

• Our method, with the derived ELBO (Eq. 10), enjoys the theoretical guarantee that the learned p_x conforms to vMF distribution. Here, introducing vMF is to compute the normalization constant in a closed form.

• Our method can shape the learned p_x to fit various distributions, i.e., it is viable without the help of vMF. For example, if we, following [a], define $h(z,k)$ to be a Bregman divergence [b], the resulting p_x turns to be an exponential family distribution, which is left as our future work.

[a] ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection

[b] Clustering with Bregman Divergences

Q.2. The improvements in the standard benchmarks are not significant and convincing.

A.2. We kindly note that the reported results of most compared baselines in all tables of this paper originate from PALM [c]. We are sorry for not explicitly illustrating this to make the reviewer confused.

• While the reimplemented CIDER in PALM performs worse than the original CIDER on LSUN with CIFAR-100 as ID data, it is advantageous for the former to have better-averaged results across 5 OOD datasets than the latter. This is because OOD data tends to be from various domains rather than a specific one.

Following PLAM, the experiments on ImageNet-100 are conducted on ResNet-50. However, the CIDER results reported on the GitHub repository are based on ResNet-34 and thus can not be used for comparison. Thus, we never undergrad existing works.

[c] Learning with Mixture of Prototypes for Out-of-Distribution Detection

Q.3. The ablation study and analysis can be improved. Additional convergence analysis should be provided.

A.3. Thanks for your constructive advice. We have added the loss trajectory to the revision, as shown in the PDF file.

Q.4. It would be hard to justify the proposed DRL can completely avoid this issue. As it is still a sampling strategy, the uncertainty of occasional unconverge situations might still occur

A.4. We will add the following explanations.

• The proposed DRL is formulated into a provably convergent EM framework. The inconsistency issue that occurs in the batch-based training scheme originates from the nature of the EM framework rather than our methodological design: during batch-based training scheme, $p_{\theta}(k|x)$ is only updated once per epoch while $f_{\theta}$ keeps updated throughout batches (iterations). The EM framework has been widely used in deep learning literature [a,b,c,d,e,f,g,h], where inconsistency is an open problem.

• Our key idea is to update the model in the current iteration and take data from the upcoming iterations into consideration. This motivates sequential sampling to have $B\_m=(B\_m^{pre},B\_m^{next})$, where $B^{next}\_{m-1} = B^{pre}\_{m}$. In this way, by optimizing classification objective over $B\_m^{next}$ encourage $p\_{\theta_{m}}(k|x)$ to be close to the ground truth, the estimated $q\_{m+1}(k|x) = p\_{\theta_{m}}(k|x)$ can be reliable for $\mathcal{B}\_m^{next}$.

• Our key contributions show that one can deterministically shape p_x to conform to the known distribution defined via Eq. (7) to avoid the unalignment between the learned and assumed distributions. We also admit that the inconsistency issue is a fundamental topic, but addressing it with a theoretical guarantee is still an open problem in the literature and out of the scope of our work.

[a] Knowledge Condensation Distillation

[b] MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering

[c] Prototypical Contrastive Learning of Unsupervised Representations

[d] Joint Unsupervised Learning of Deep Representations and Image Clusters

[e] Deep Clustering for Unsupervised Learning of Visual Features

[f] Learning representation for clustering via prototype scattering and positive sampling

[g] Stable Cluster Discrimination for Deep Clustering

[h] Unsupervised Visual Representation Learning by Online Constrained K-Means

Q.5. To perform online approximation, I would like to know if the batch size is an important parameter

A.5. According to the definition in Line 159, the online approximation in Eq. (9) does not leverage data statistics within batches, i.e., it is independent of the batch size.

Q.6. The description in eq 7 (line 138) is missing

A.6. Thanks for pointing out the problem caused by typos. We have fixed this mistake: $p_{\theta}(x|k)=\frac{h(z,k)}{\Phi(k)},\quad \Phi(k) = {\int_{z \in \mathcal{Z}} h(z, k) \, d z}$, where $\mathcal{Z}$ is the latent feature space.

Q.7. In Tables 4 and 5, the proposed methods can hardly be comparable to PALM

A.7. We suspect this is because PLAM introduces more than one prototype for each ID class to learn better features regarding discriminating the aforementioned OOD dataset from the ID dataset. We kindly note that our method achieves a better average performance than PALM.

Q.8. Why is CIDER not in Table 5?

A.8. Unlike PALM proposed for unsupervised and supervised settings, CIDER is proposed for supervised settings. Thus, it is unclear how to extend CIDER to the setting. Thus, we exclude CIDER from Table 5.

Dear Reviewer #7foW,

The window for discussion is closing.

Thanks very much for your great efforts in reviewing and valuable comments. The discussion will end soon. At this final moment, we would sincerely appreciate it if you could check our responses and new results regarding your concerns.

1. The proposed method is still not viable without existing techniques.

We apologize for the misunderstanding. We have provided explicit explanations highlighting that the assumption used by previous methods is the motivation for our method instead of the basis for our work.

2. The improvements are not significant and convincing

We apologize for the misunderstanding. We have provided detailed explanations for the experimental results, i.e., we used results reported in the literature and the different results from the difference in model architectures.

3. Explanations for sampling strategy

We have provided detailed explanations highlighting the motivation of the proposed sampling strategy.

If you have any further concerns, we will respond instantly at this final moment. We would sincerely appreciate it if you could confirm whether there are unresolved concerns.

Best regards and thanks,

Authors of #8706

We sincerely appreciate your time and effort in reviewing. We have provided detailed responses below, hoping your concerns can be adequately addressed.

Q.1. Contradiction in Assumptions

A.1. We apologize for the misunderstanding and will provide explicit explanations highlighting that our method aligns with the motivation.

• As shown in section 2, existing feature-based and logit-based methods explicitly or implicitly assume $p_{\theta}(x)$ as the Gibbs-Boltzmann distribution and Gaussian mixture distribution, respectively. However, there is no theoretical guarantee to ensure these distributional assumptions hold under any discriminative models $f_{\theta}$ that are trained by minimizing the classification objective. This inconsistency would degrade OOD detection performance.
• Given that $p_{\theta}(x)$ is typically unknown after a discriminative model $f_{\theta}$ is trained by minimizing the cross entropy, our method proposes to deterministically shape $p_{\theta}(x)$ to conform to the known distribution defined via Eq. (7). This avoids the above unalignment issue. Namely, this leads to our motivation: is it possible to deterministically shape the ID feature distribution while pre-training a discriminative model? In this way, the defined distribution can be safely used for OOD detection since the involved distributional assumption can hold after training where Eq. (10) is minimized.

Q.2. Influence of class number (K)

A.2. Thanks for your kind suggestions. Our method's memory and time complexity linearly scales with K so that our method has the same scalability nature as the CE-based training.

• The parameters involved in our method are located in 1) backbone and 2) prototypes $\mu\_1,...,\mu\_K$. Since the backbone is orthogonal to our technical design, we omit the memory complexity of the backbone. Therefore, the memory complexity of our method is $O(K)$.
• Computing Eq. (14) for each x requires to compute $p\_{\theta}(x|k)$ for each $k$. By omitting the time complexity of the backbone and dot product, the time complexity of our method is $O(K)$.
• We conduct ImageNet-100 for large-scale OOD, aiming to keep consistent with prior works [a,b].

[a] Learning with Mixture of Prototypes for Out-of-Distribution Detection

[b] How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection

Q.3. Comparison with Density-Based Methods

A.3. While this paper has considered popular post-hoc density-based methods, including Energy, Maha, and CONJ, for comparison, as per your valuable advice, we compare to more baselines (Maxlogit [c] and GEM [d]) and report the averaged results across 5 OOD datasets used in this paper as follows:

We compare it to the training-based density-based method [e], where an extra flow model is introduced on top of the pre-trained model. Here, following [e], we evaluate our method on CIFAR-10 and report the averaged results across 6 OOD datasets (5 used in this paper+LSUN-Resize)

[c] Scaling Out-of-Distribution Detection for Real-World Settings

[d] Provable Guarantees for Understanding Out-of-distribution Detection

[e] FlowCon: Out-of-Distribution Detection using Flow-Based Contrastive Learning

Q.4. Justification of Sequential Sampling

A.4. We will add the following explanations.

• The proposed DRL is formulated into a provably convergent EM framework. The inconsistency issue that occurs in the batch-based training scheme originates from the nature of the EM framework rather than our technical design: during batch-based training scheme, $p_{\theta}(k|x)$ is only updated once per epoch while $f_{\theta}$ keeps updated throughout batches (iterations). The EM framework is widely used in deep learning literature [f,g,h,i,j,k,l,m], where inconsistency is an open problem.
• Our key idea is to update the model in the current iteration and take data from the upcoming iterations into consideration. This motivates sequential sampling to have $B\_m=(B\_m^{pre},B\_m^{next})$, where $B^{next}\_{m-1} = B^{pre}\_{m}$. In this way, by optimizing classification objective over $B\_m^{next}$ encourage $p\_{\theta_{m}}(k|x)$ to be close to the ground truth, the estimated $q\_{m+1}(k|x) = p\_{\theta_{m}}(k|x)$ can be reliable for $\mathcal{B}\_m^{next}$.

[f] Knowledge Condensation Distillation

[g] MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering

[h] Prototypical Contrastive Learning of Unsupervised Representations

[i] Joint Unsupervised Learning of Deep Representations and Image Clusters

[j] Deep Clustering for Unsupervised Learning of Visual Features

[k] Learning representation for clustering via prototype scattering and positive sampling

[l] Stable Cluster Discrimination for Deep Clustering

[m] Unsupervised Visual Representation Learning by Online Constrained K-Means

Q.4. Ablation on $\beta$

A.4. As per your advice, we conduct the ablation study on $\beta$ (with CIAR-10 as ID dataset) and show in the following:

Q.5 Description for Table 5

A.5. In Table 5, we extend our method to the unsupervised setting. Following DINO [n], we keep a momentum teacher to produce soft labels as the target. As with DINO, we use the centering operation to avoid collapse.

[n] Emerging properties in self-supervised vision transformers

Q.6 Equation Mistakes

A.6. Thanks for your comments. The revised paper has fixed the typos and mistakes you mentioned.

Q.7 Symbol Description in line 138

A.7. Thanks for pointing out the typo. We have fixed this in our revised paper and show the correct definition of $\Phi(k)$ in Eq. (7) $\Phi(k) = \int_{\mathbf{z} \in \mathcal{Z}} h(\mathbf{z}, k) d\mathbf{z}$

Dear Reviewer #s5Ce,

The window for discussion is closing.

Thanks very much for your great efforts in reviewing and valuable comments. The discussion will end soon. At this final moment, we would sincerely appreciate it if you could check our responses and new results regarding your concerns.

1. Contradiction in assumptions

We apologize for the misunderstanding. We have provided detailed explanations highlighting that the proposed method aligns with our motivation.

2. Influence of class number

We apologize for the misunderstanding. Our method's memory and time complexity linearly scales with K so that our method has the same scalability nature as the CE-based training.

3. Justification of sequential sampling

Following your valuable suggestion, we have provided explicit explanations highlighting the motivation for the widely adopted approach.

4. Comparison with density-based methods

Following your valuable suggestion, we have added experiments, results, and discussions, demonstrating the effectiveness of our method.

5. Ablation

Following your valuable suggestion, we conducted the ablation study on the hyper-parameter $\beta$ mentioned.

If you have any further concerns, we will respond instantly at this final moment. We would sincerely appreciate it if you could confirm whether there are unresolved concerns.

Best regards and thanks,

Authors of #8706