Learning with Mixture of Prototypes for Out-of-Distribution Detection

Q1. Discussions on hyperparameters for different models or datasets.

Fig. 5$c$ and 8(b) show the performance using different hyperparameters $K$ and $\alpha$, which are produced on the benchmark using CIFAR-100 as ID data and using others (such as SVHN, Place365, LSUN, etc) as OOD data. The curves in Fig. 5$c$ and 8(b) are the performance of the same model trained on the same ID data (e.g., CIFAR-100) on different unseen OOD datasets. They show that the proposed method can perform well and stably given various proper hyperparameters.

Considering the training is only applied on ID data (where OOD data can only be seen in testing), the setting of hyperparameters is mainly for reflecting the ID data characteristics from obtaining high-quality embedding space (with good OOD detection behaviour in testing). In experiments, we observed the method is robust to the hyperparameter setting on different training ID datasets (such as CIFAR-100, CIFAR-10, ImageNet-100). Although tuning the hyperparameters in a proper range may slightly increase or decrease the performance, we use the same default hyperparameters to produce the state-of-the-art results in Tables 1-8, demonstrating the robustness of hyperparameter settings on different datasets.

Q2. In-distribution classification accuracy.

Following the settings in KNN, SSD, and CIDER, we demonstrate the ID classification accuracy of the proposed method in Fig. 7. While achieving outstanding OOD detection capabilities, our method also performs well in ID classification. It either surpasses or is competitive with previous methods in this aspect. This highlights the effectiveness of the proposed research, showcasing its proficiency not only in identifying OOD samples but also in accurately classifying ID samples.

Q3. Varying behaviours across datasets in Fig. 5$c$ (i.e., Fig. 4$c$ in the revised version).

As discussed in Q1, the curves are for different OOD datasets used in inference, where the model is trained on CIFAR-100 as the ID dataset. It shows that the proposed method can perform well consistently with different settings of the prototypes. Although there may exist an ideal setting for a given training dataset, considering that the main role of prototypes is to regularize/shape the embedding space, a properly set (i.e., not too small or too large) parameter is enough for producing good results.

The behaviours of OOD detection on different testing OOD datasets are slightly different, mainly due to the difference in data characteristics and the gaps between ID and OOD data, which is not strongly associated to the intra-class variability in the OOD datasets seen only in testing. For example, while using the place recognition dataset Place365 as OOD data, an OOD input image from Place365 (with a place as the label) can easily contain similar semantic information as the images in CIFAR-100, making OOD detection on it difficult and leading to more unstable results.

Q4. Highlight the difference to CIDER.

Thank you for the comments. We highlight the significance of our method and the difference to CIDER.

Among distance-based OOD methods, the main objective is to learn the embedding space (using only ID data) that can discriminate the ID and unseen OOD sample. Different regularizations and supervisions are investigated in various methods based on similar and fair basic frameworks, such as CSI, SSD, CIDER, and our method PALM.

Despite the outstanding performance of CIDER, we identified the limitations of the straightforward (single-prototype) modelling in CIDER. 1) By analyzing the underlying insights, we first propose to use mixture modelling for better regularizing/shaping the embedding space and thus formulate the mixture vMF models different from CIDER. 2) Achieving multi-prototype modelling (i.e., moving from a single prototype to multiple prototypes) is not trivial, since, in single prototype modelling, all samples in a class have been assigned to a shared prototype (which is also the simplified assumption). To make multi-prototype modelling achievable, we propose a series of techniques, including prototype updating, sample-prototype assignment considering diversity, and assignment pruning. 3) Based on that, the novel MLE and contrastive losses based on prototypes use more flexible modelling to shape the embedding space. 4) Based on the carefully designed modelling and algorithm, the proposed method achieves state-of-the-art performances. We further conducted extensive analyses to analyze the behaviour of the methods. Overall, the relationship between the proposed method and CIDER does not influence the significance of our method.

Q5. Minor points for figure presentation.

Thank you for your suggestion. We have revised the presentation of Fig. 4-7 accordingly. 1) The order of Fig. 4 and 5 is changed. 2) The font size in Fig. 4 and 6 is adjusted to be larger.

Q1. Discussion on scalability with multiple prototypes and computational cost.

Although our method involves multiple prototypes per class, it introduces minimal additional overhead on both computation and memory, not influencing the scalability.

• In terms of memory, the additional parameters introduced by the multiple-prototype design are only the prototypes as vectors, and the assignment weights for each batch are temporally calculated during training. Under the default setting with 6 prototypes for each of 100 classes (i.e., 600 prototypes in total), it introduces 1.2% of additional parameters compared to the single prototype method. Specifically, based on ResNet-34 with 21.60M, CIDER introduces one prototype per class, increasing the parameters to 21.65M, while our method is 21.91M. The additional computation caused by ours is restricted.
• The additional computations of our method are mainly caused by calculating the soft assignment weight using the efficient Sinkhorn-Knopp approximation, which takes only 32 milliseconds each time. As discussed in Appendix C.1, the computation time is less than or similar to the previous efficient and powerful methods. With an optimized implementation, our method with multiple prototypes can be faster than CIDER. The computation times are based on the GeForce RTX 3090 GPU.

In the paper, we validated the proposed method with the ImageNet-100 (as ID data used in training), which is a relatively large dataset for OOD, where our method works well and performs better than other methods.

Q2. Influence of the quality of prototypes.

The prototype quality indeed influences the learning. We develop the method with various techniques to ensure the quality of the prototypes, such as the EMA estimation for stable prototype updating and assignment pruning to remove outliers in updating. We observe that, given proper initialization, the method can work effectively. In Appendix C - Table 7, we report the results based on different independent runs with different initializations, having a standard deviation on Average AUROC of only 1.08, implying the robustness of our method in producing high-quality prototypes. To further validate this, we report the results of using different initialization methods for the prototypes in Table 2 in the following. It shows that the proposed method can produce good prototypes and performances in different situations.

Table 2: Performance of PALM using different initialization methods for prototypes.

We have added the experiments and discussions in our revised version.

Q3. Discussion on performance on Texture dataset in Table 1, 2, and 3 of paper.

Our method can achieve state-of-the-art results on most of the metrics, especially among the methods with similar frameworks (e.g., distance-based methods), such as CIDER, KNN, and SSD. In Table 1, NPOS achieves the best FPR on Textures as the OOD data, since it directly generates OOD samples to boost training. The Textures dataset, characterized by its diverse textural patterns, may be more amenable to artificial OOD sample generation, especially when compared to the synthesis of natural images, which are more challenging to synthesise. Similarly, in Table 2, our method can perform very well via a simple extension from the labelled setting, while other methods include some specific designs for the unlabelled setting (without semantic labels in the ID training data). In Table 3, we validate the effectiveness of our method with different distance measurements, while the Mahalanobis distance-based version is the best.

Q1. Mixture modeling and normalizing to hyperspherical embedding space

In our method, we use the mixture of von Mises-Fisher (vMF) distributions (instead of the mixture of Gaussian distributions). The vMF distribution models the points on the hypersphere. Based on this model, we normalize the embeddings to the unit norm. The vMF distribution is commonly used to formulate and explain the embedding space from contrastive learning (Wang & Isola, 2020) ([1] in the following). Based on the contrastive representation learning on the hypersphere (and proper regularizations), the learned embeddings can be discriminative and beneficial for OOD detection, as commonly used in many methods (e.g., SimCLR, SupCpon, CIDER, and PALM) including our PALM. Different from CIDER using single vMF distribution for a class, we use the mixture modelling of vMF to capture the more complex and realistic data.

[1] Wang, T. and Isola, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning (pp. 9929-9939). PMLR.

Q2. Updating of prototypes.

Thanks for the comments. Different classes and datasets indeed have different characteristics. A more flexible and dynamic setting of the EMA hyperparameter $\alpha$ can be better but more difficult. However, the momentum parameter $\alpha$ mainly controls the updating strength in the iteration (like a learning rate), which does not influence too much the training behaviours on different data. As shown in Sec. C.1 (Fig. 8(b)), the experiments show that the influence of $\alpha$ is small. A properly set hyperparameter can work well for many cases.

During the updating of prototypes, the sample-prototype assignment weights are highly relevant to the class/data characteristics. We update with the robust cluster assignment method in Eq. (6) and use the assignment pruning method to alleviate the influence of unexpected points in prototype updating.

We revised the paper accordingly.

Q3. Estimation of the soft assignment weights $w_{i,k}$.

To avoid learning a collapsed model, we encourage samples of the same class in a batch to be assigned to diverse prototypes. We optimize $w_{i,k}$'s in $\mathbf{W}^c$ (for each class $c$) by solving the transportation polytope problem in Eq. (8) and (9) (in the revised paper) to maximize the similarity between samples and the assigned prototypes with the regularization. The solution is obtained via Eq. (6). More details of the updating problem are in Sec. 3.3 and Appendix A.2. Considering that, although some samples are the top-K neighbours of a prototype, some of them may still be far away like outliers, we further prune the assignment weights for more robust and stable updating, achieving better performance as indicated in Fig. 5(a).

Given the initialized prototypes, the assignment weights (between prototypes and each sample in the batches) in the iterations are calculated relying on the distances and the diversity regularization mentioned above, which avoids the potential trivial and collapsed solution. The prototypes and the subsequent weights are quickly updated to proper positions in the embedding space. Analyses in Appendix C (Tables 7 and 12) show the robustness of initializations (of the prototypes).

Q1. Experiments on OpenOOD benchmark. ``Most of the datasets used in experiments are based on standard benchmark datasets. How are these datasets, such as CIFAR are used for OOD in this work?''

Thank you for the suggestion. We evaluate the methods with the OpenOOD benchmark and report the results in the following Table 1. In OpenOOD, while using CIFAR-10 and CIFAR-100 as in-distribution data, other datasets, such as TinyImageNet and MNIST, are used as "Near-OOD" or "Far-OOD". As shown in Table 1, our method is better than or competitive with other methods on most metrics, which validates the effectiveness.

In the paper, the CIFAR datasets are used as in-distribution (ID) data, and other datasets are used as OOD data, which follows the standard experimental protocol widely used in previous works, e.g., CIDER, SSD, Energy, and KNN, as another benchmark. The OOD datasets used in this benchmark are different from OpenOOD. In both the standard and near-OOD experiments in Table 1 and Appendix C - Table 5, our method can achieve the best performance average.

We have added the results on OpenOOD in the paper and updated the discussions.

Table 1: Results on OpenOOD benchmarks.

Q2. Comparison with generative-based (GAN-based) methods.

Thank you for the suggestion. We discuss the generative-based (GAN-based) methods [1,2] for OOD detection in the revised paper. Considering the GAN-based method in [1] was originally designed for NLP tasks, we mainly conduct the comparison using the related method OpenGAN [2] on the OpenOOD benchmark, as shown in Table 1. The comparison further provides a broader perspective on the analysis. The results and discussions have been added to the revised version.

[1] Ryu, S., Koo, S., Yu, H. and Lee, G.G., 2018. Out-of-domain detection based on generative adversarial network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 714-718).

[2] Kong, S. and Ramanan, D., 2021. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 813-822).

Q3. Details of $\text{diag}(\mathbf{u})$ and $\text{diag}(\mathbf{v})$ on Eqn.(6).

As introduced in Sec. 3.3 and Appendix A.2, $\mathbf{u}$ and $\mathbf{v}$ are two non-negative vectors for obtaining the solution of $\mathbf{W}^c$ constrained by the transportation polytope $\mathcal{W}$ via Eq.(6). As shown in Lemma 2 in (Cuturi, 2013) in the paper, the solution of $\mathbf{W}^c$ is unique and can be obtained via the form of Eq.(6). $\mathbf{u}$ and $\mathbf{v}$ are two non-negative vectors for obtaining the solution, which can be computed with Shinkhorn's fixed point iteration $(\mathbf{u}, \mathbf{v}) \leftarrow (1/(K\mathbf{H}\mathbf{v}), 1/(B\mathbf{H}\mathbf{u}) )$. $\text{diag}(\mathbf{u})$ and $\text{diag}(\mathbf{v})$ are the diagonal matrix of the two vectors $\mathbf{u}$ and $\mathbf{v}$. The detailed proof and algorithms are referred to (Cuturi, 2013). We revised the paper and appendix A.2 with more details accordingly.