Kernel PCA for Out-of-Distribution Detection

Thanks for the detailed comments, each of which will be answered pointwisely below.
Due to length limit, we

• use "W", "Q" for Weakness, Question,
• put the references in global response,
• put the mentioned figures in the one-page rebuttal PDF file.

(W1) The usage of RFF approximation

• For the Cosine kernel, the RFF approximation is not needed, since the Cosine kernel $k_{\rm cos}(z_1,z_2)=\frac{z_1^\top z_2}{||z_1||\_2||z_2||\_2}$ has its exact feature mapping $\phi_{\rm cos}(z)=\frac{z}{||z||\_2}$, such that $k_{\rm cos}(z_1,z_2)=\phi_{\rm cos}(z_1)^\top\phi_{\rm cos}(z_2)$.
• For the Cosine-Gaussian kernel, the RFF is adopted to only approximate the Gaussian kernel part, i.e., $k_{\rm gau}(z_1,z_2)\approx\phi_{\rm RFF}(z_1)^\top\phi_{\rm RFF}(z_2)$ in Eqn.(2) in the paper. Specifically, the adopted feature mapping $\phi_{\rm RFF}(\phi_{\rm cos}(z))$ for the entire Cosine-Gaussian kernel is a composite of the exact feature mapping $\phi_{\rm cos}$ for the Cosine kernel part and the approximate RFF mapping $\phi_{\rm RFF}$ for the Gaussian kernel $k_{\rm gau}$ part.

(W2) Dimensionality of penultimate layer features $m$ and the number of RFFs $M$

The settings of $m$ and $M$ are provided in Table 6eWv-1, where $M$ is set as $4m$ for the CIFAR10 InD and $2m$ for the ImageNet-1K InD for all experiments in the paper.

Table 6eWv-1 The values of $m$ and $M$

Indeed, as $M$ increases, RFF can better approximate the Gaussian kernel. We have conducted sensitivity analysis on $M$ in line 605 and Fig.9 of Appendix D.3, showing that increasing $M$ boosts and then maintains rather steady OoD detection performances. Hence, we note that a substantially high number of RFF is not of significant necessity in the addressed task. To balance efficiency and effectiveness, we adopt $M$ as in Table 6eWv-1 with SOTA results under varied setups.

(W3) Computation complexity of RFF

With Eqn.(2), for the RFF mapping $\phi_{\rm RFF}$, $2M$ samplings for $\omega_i$ and $u_i$ respectively are required, and $\phi_{\rm RFF}$ includes $M$ dot products and $M$ additions. Accordingly, the memory and time complexity of RFF is ${\cal O}(M)$.

(W4) Organization of Sec.2.2 and Sec.3.1

Thanks for the nice suggestion. We will merge Sec.2.2 and Sec.3.1 as a 'background' section in the revised version.

(W5) Proposition 2 and the implementation of CoP and CoRP

Proposition 2 supplements the analysis for kernel implementation of CoP and CoRP.

CoP and CoRP leverage the covariance-based KPCA and are implemented via explicit feature mappings induced from the kernel, rather than via the more expensive kernel implementation. Proposition 2 provides the reconstruction errors when implemented with kernels, and serves as a supplementary analysis on our more efficient covariance-based CoP and CoRP.

(W6) Elaboration on the two kernels

The proposed OoD detector is a KPCA-based method with two alternative choices of kernels explained in the main context, i.e., Cosine kernel and Cosine-Gaussian kernel.

(Q1) How O(1) complexity is achieved with RFF

Thanks for the valuable comment. To align with the complexity analysis of KNN detector [7], the O(1) time and memory complexity in our work also refers to the inference procedure of calculating the OoD detection score, i.e., the reconstruction error, when given with features, so it indeed does not include the computation of the mapping. Regarding the raised comment in W3, we specify the complete computation complexity of CoP and CoRP:

• For CoP without RFF involved, the time and memory complexity in inference is O(1), since the feature mapping $\phi_{\rm cos}$ is obtained right away as the normalized $z$ without additional cost.
• For CoRP with RFF involved, when considering the computation of the feature mapping $\phi_{\rm RFF}(\phi_{\rm cos}(\cdot))$, we have the complete time and memory complexity in inference as ${\cal O}(M)$, since $\phi_{\rm RFF}$ requires an ${\cal O}(M)$ complexity.

(Q2) How to choose $q$

In CoP and CoRP, $q$ is selected as the number of principal components sufficing a certain explained variance ratio (e.g., 99%), which is also a common technique in (K)PCA-based methods. For example, given the ResNet50 on ImageNet-1K with a penultimate feature dimension $m=2048$, to preserve an explained variance ratio of 99%, $q=1346$ principal components are required in CoP.

We have conducted sensitivity analysis on $q$ in CoP and CoRP to comprehensively evaluate its effect. More details can refer to line 587 and Fig.7 in Appendix D.3.

(Q3) Kernel function in Fig.2

In the left panel of Fig.2, the adopted kernel function is the Cosine kernel function $k_{\rm cos}$. In the right panel of Fig.2, the adopted kernel function is the Gaussian function $k_{\rm gau}$ on $\ell_2$-normalized features, i.e., Cosine-Gaussian kernel.

(Q4) Smaller things

• Typos and literatures. We will revise all the typos and add the recommended literature [9] in the updated version. Also, we will consider the suggested Recursive Feature Machines [10] in learning algorithms for OoD detection in future works.
• Unclear phrasing. Regarding the 'under the same framework' in line 259, it indicates that our KPCA reconstruction error and the regularized reconstruction error in [6] are evaluated under the same fusion trick as [6] for a fair comparison.

Limitation on reproducibility

The source code for reproduction has been provided in the supplementary material, and will also be released publicly in github upon acceptance. In Appendix D.3, all hyper-parameters of CoP and CoRP are discussed in detail and evaluated with ablation studies.

Thanks for the detailed comments, each of which will be answered pointwisely below.
Due to length limit, we

• use "W", "Q", "L" for Weakness, Question, and Limitation,
• put the references in global response,
• put the mentioned figures in the one-page rebuttal PDF file.

(W1) More comparisons

As suggested, we add the post-hoc method ASH [3] and a training-based method DML [4] for more comprehensive comparisons, shown in Table w1ZW-1. Another suggested method ReAct [5] has been considered into comparisons in Table 3 and Table 5. Besides, we also supplement two latest post-hoc methods SCALE [1] and DDCS [2], as suggested by Reviewer sSKR.

Table w1ZW-1 Additional comparisons with more latest methods with ResNet50 on ImageNet-1K as InD data. Our CoRP detector is set upon ReAct [5,6] similar to Table 3 in the paper.

(W2) Ablation on more kernel designs

In our work, we have demonstrated the indispensable importance of the $\ell_2$-normalization feature mapping from Cosine kernel. On such basis, upon the Cosine kernel, we further deploy a Gaussian kernel to pursue greater separability between InD and OoD, where the $\ell_2$-distance along samples are preserved: $k_{\rm gau}(z_1,z_2)=e^{-\gamma||z_1-z_2||_2}$. Further investigations on more kernel designs are shown as follows. More details can also be found in Appendix D.1 and D.2.

• Cosine-Laplacian kernel. Aside from the Gaussian kernel preserving the $\ell_2$-distance on top on the indispensable Cosine kernel, we have considered another Cosine-Laplacian kernel, where the Laplacian kernel could keep the $\ell_1$-distance: $k_{\rm lap}(z_1,z_2)=e^{-\gamma||z_1-z_2||_1}$. Comparisons in Table w1ZW-2 demonstrate the superiority of Gaussian kernel, and refer to Fig.6 of Appendix D.2 for thorough results.
• Cosine-polynomial kernel. The polynomial kernel $k_{\rm poly}(z_1,z_2)=(z_1^\top z_2+c)^d$ is also investigated via the Tensor Sketch approximation [8] in Table w1ZW-2, and is shown to be unable to bring superior detection performances.
• Effects of individual kernels. We have also investigated the individual effect of each kernel in Table w1ZW-2, where the $\ell_2$-normalization mapping from Cosine kernel shows indispensable significance for OoD detection and the preserved $\ell_2$-distance from Cosine-Gaussian kernel brings the best results. More thorough results can refer to Fig.5 of Appendix D.1.

Table w1ZW-2 Comparisons among different kernels with ResNet18 on CIFAR10 as InD data.

(W3) Why not non-linear classifier & why linear separability matters

We would like to address this point from the following two aspects:

• Non-linear classifier is impractical for general OoD detection tasks. OoD detection is an unsupervised task, as the DNN is trained on InD data without knowledge about OoD data. In OoD detection, OoD data can be any data of any size drawn from different distributions w.r.t InD. Thus, it is impractical to particularly train a non-linear binary classifier with sufficient OoD data being another class. Then, it is justified to consider unsupervised models that brings separability between InD and OoD, such as the KNN-based detector in [7], and our KPCA-based detector.

• Linearity separability matters. In our work, we find that the linear principal components by simply applying PCA are insufficient to depict the informative patterns of InD features, as InD features are not even located compactly nor linearly inseparable with OoD features, as shown in Fig.1(a). Thus, PCA cannot achieve good reconstruction performances for all InD features, so that the reconstruction of InD and OoD features along such extracted principal components from linear PCA can both be bad, failing to well differentiate OoD from InD.

  Therefore, we introduce non-linear kernels with KPCA which map InD and OoD features into a new space, where the corresponding InD mappings are located more compactly and are almost linearly separable from OoD mappings. Thanks to such separability between InD and OoD, informative principal components are learned for InD and only works for InD with good reconstruction, as the separable OoD fails to be well characterized along such principal components, as shown in Fig.1(b). In this sense, good OoD detection performances are attained through the reconstructions along new axes by the extracted components in the mapped feature space with KPCA.

(W4) Important assumptions for the theory

Important assumptions for the theory in the paper are listed.

• For the RFF approximation outlined in Sec.2.2, the theory basis of RFF methods is the Bochner's theorem, which assumes a continuous and shift-invariant kernel (line 118). In our work, we apply RFF to approximate the Gaussian and Laplacian kernels, both of which satisfy the assumption in the Bochner's theorem.
• For the analytical analysis in Sec.5, the two propositions have no particular assumptions, and can be applied to any standard KPCA methods under both covariance and kernel setups.

Dear Reviewer,

We hope that our responses above could address your concerns and be helpful for final evaluations on our work.

As the author-reviewer discussion is approaching the end, we would like to inquiry if there is any question, and we would be willing to discuss.

Sincerely, Authors.

Thanks for the detailed comments, each of which will be answered pointwisely below.

(Weakness 1 & Question 1) Selection of kernels/feature mappings

We would like to firstly elaborate the effectiveness and novelty of our KPCA method (Weakness 1), and then discuss more kernel selections (Question 1).

(A1) Novelty of KPCA with the designed two effective kernels for OoD detection

• The key of our KPCA detector is to find qualified kernels (or their explicit mappings) that can capture informative patterns of InD and meanwhile well distinguish InD and OoD in the mapped space.
  From our extensive evaluations, we have highlighted that: the generic KPCA method can adapt to a wide variety of kernels, but only specific designs of kernels are competent for OoD detection, which is justified in detail in A2 below. Thus, although the KPCA detector is not entirely new in the sense of kernel methods, we have brought new insights into OoD detection from a kernel perspective, which is novel to the research community and the first trial on practicable and efficacious kernel methods for OoD detection.
• The efficiency of our KPCA detector is underlined for its cheap O(1) complexity from the reconstruction-error score in OoD inference over the O(N) of KNN. Besides, explicit feature mappings instead of expensive kernel matrix computation are deployed in our detector, resolving the inefficiency of KPCA on large-scale data.
• Despite the used $\ell_2$-normalization technique, our method is significantly different from KNN, which will be discussed in the response to Question 2.

(A2) Investigations on more kernel choices
The devoted two kernels provide the following in-depth insights about InD and OoD that should be considered in designing more viable kernels:

• The $\ell_2$-normalization from Cosine kernel plays a pivotal role in distinguishing OoD from InD by balancing the norm of InD and OoD features.
• The $\ell_2$-distance preserving property of the shift invariant Gaussian kernel can further promote the separability that benefits OoD detection performances.

Therefore, when devising or learning new kernels, we should consider those kernels that can normalize feature norms or preserve the distance relationships. For example, the ablation studies in Fig.6 of Appendix D.2 discuss an alternative cosine-Laplacian kernel, where the $\ell_1$-distance gets preserved via the Laplacian kernel. Another common polynomial kernel is also investigated in Table LuQe-1.

Table LuQe-1 More kernel choices with ResNet18 on CIFAR10 as InD.

(Weakness 2) Limitation of learning kernels as future works

Learning kernels as a potential area may not be redundant for OoD detection.

• Despite the training on DNNs from scratch, it is also a widely-accepted research paradigm to further perform learning with features from a pre-trained DNN. A typical example is the celebrated Deep Kernel Learning (DKL, [1]), where a kernel learning procedure is imposed upon the features learned from a pre-trained DNN.
  With particular regards to OoD detection, an additional learning step can be taken for the specified goal. Here, DKL could be considered as an alternative choice so as to pursue stronger kernels that can better characterize InD and OoD with enhanced detection performance. In contrast, we currently leverage the kernel method under typical non-parametric setups, rather than learning-based ones.
• Although we leverage the non-parametric kernel method, it still requires a few hyper-parameters. In this work, we tune the parameters $q$, $M$ and $\gamma$ of our proposed CoP and CoRP and use the suggested default settings according to our extensive evaluations. In future work, it would be of greater convenience and potentials, if such kernels can be learned with data available at hands.

[1] Wilson, et al. "Deep kernel learning." AISTATS'2016.

(Question 2) Justification between KNN and KPCA

KNN relies on the discrete distance, while the analysis on KPCA comes from the feature space w.r.t Mercer kernel operators. Thus, it is difficult to align theoretical justifications between such two methods, nor to make comparisons straight away. Despite the $\ell_2$-normalization technique, our KPCA detector has significant differences with KNN, regarding the designed kernel tricks and O(1)-complexity detection score. With particular interest to OoD detection, the following two aspects are elaborated to discuss the superiority of our KPCA detector over the KNN one.

OoD detection score design. KNN sets the Euclidean distance between samples as the detection score, while KPCA adopts the reconstruction error along the extracted principal components. For the former, the k-th smallest Euclidean distance to InD data determines whether a sample is OoD or not; thus, only the nearest affinity neighborhood in InD data is used, where the full information of InD is missing in calculating the detection score. In contrast, in the latter, all InD data contributes to learning the principal components for the reconstruction; accordingly, our KPCA-based method goes beyond the affinity neighborhood in KNN, but makes full use of InD data in designing the detection score, facilitating stronger OoD detection.

Low-dimensional subspace modeling. Another key advantage of KPCA comes from the low-dimensional subspace learned from InD data, where the most informative patterns of InD data (e.g., 99% explained variance) is kept and redundant information in InD data is removed. Such a subspace is learned with specification to InD data and is thereby less sensitive in revealing the pattern of OoD data than using the original all dimensions of KNN. Hence, OoD data can get more easily differentiated.

Thanks for the detailed comments, each of which will be answered pointwisely below.
Due to length limit, we

• use "W", "Q", "L" for Weakness, Question, Limitation,
• put references and mentioned figures in global response and one-page rebuttal PDF file,
• only show the average FPR in tables and put the results on each OoD dataset in global response.

(W1 & Q1) Evidence against PCA

The primary goal in OoD detection is to differentiate OoD and InD data. To this end, we provide the following evidence on the significance of non-linear KPCA upon linear PCA.

• As shown in Fig.1(a), InD features from the DNN backbone are not compactly located nor easily separable w.r.t OoD features. Then, it is difficult to rely on PCA to find a new axis system simply through linear transformation, where InD and OoD features can be well separated, since both InD and OoD features can have worse reconstruction along the axes.
• In contrast, in Fig.1(b) with our KPCA-based method, InD features are located compactly. In this way, InD features can be well reconstructed along the informative axes extracted for InD, while OoD features cannot be well reconstructed with such axes and thus are easily separated from InD.
• An ablation study is also given in Table sSKR-1, further verifying the significance of KPCA over PCA.

Table sSKR-1 Comparisons on PCA and KPCA with ResNet18 on CIFAR10 as InD.

(W2 & Q2 & L2) Outdated Baselines

As suggested, we additionally compare with four latest detection methods, SCALE [1] (ICLR'2024), DDCS [2] (CVPR'2024), ASH [3] (ICLR'2023) and DML [4] (CVPR'2023). Table sSKR-2 shows that our method maintains superior performances and will be incorporated into next version.

Table sSKR-2 Comparisons on latest baselines with ResNet50 on ImageNet-1K as InD, where our CoRP is set upon ReAct [5,6] similar to Table 3 in the paper.

(W3) Insufficient demonstration on $\ell_2$-normalization significance

We will elaborate the significance of $\ell_2$-normalization, i.e., Cosine kernel, from two aspects.

• Promoting separability between InD and OoD features
  • Motivation. Kernels can measure (dis)similarities between samples, e.g., a pair of InD features, such that $k(z_1,z_2)=\phi(z_1)^\top \phi(z_2)$. The norms of InD features from the DNN backbone can vary drastically, even after feature mapping $\phi$. This may pose numerical defects to kernel methods, as any two InD features $z_1,z_2$ can have varied magnitudes with very large $\phi(z_1)^\top\phi(z_2)$, despite of their true (dis)similarities. This does not help reveal the intrinsic connections between samples, hindering the extraction of informative principal components of InD features. Then, both InD and OoD features can have bad reconstruction and are thereby not distinguishable.
  • Numerical Tests. In Fig.4 and Fig.5 of Appendix D.1, detailed discussions are presented. More specifically,
    • Fig.4 (left) shows the inseparability of InD and OoD features without normalization.
    • Fig.4 (middle) still shows the inseparability of InD and OoD features even applied with a Gaussian kernel, indicating that a non-linear mapping alone fails to capture informative patterns of InD features, nor to separate InD and OoD features.
    • Fig.4 (right) shows distinctive separability of $\ell_2$-normalized (with Cosine kernel) InD and OoD features, indicating that $\ell_2$-normalization is critical to bring meaningful measures in kernel methods for InD features.
    • Ablation studies are provided in Table sSKR-3 with thorough results in Fig.5.

Table sSKR-3 Ablation on Cosine kernel in CoP and CoRP with ResNet18 on CIFAR10 as InD.

• Alleviating imbalanced feature norms for distinguishable reconstruction errors
  The norms of InD features are commonly larger than that of OoD features, as shown in Fig.3 in Appendix D.1, which was also pointed in [7]. Recall the reconstruction error in Eqn.(4), where $\mu$ is for centering: $e(z)=||U_qU_q^{\top}(z-\mu)-(z-\mu)||_2\leq||U_qU_q^{\top}-I||_2\cdot||z-\mu||_2$ As $||z-\mu||_2$ of InD are commonly larger than OoD, the reconstruction error of OoD features can still be small, due to the imbalanced norms shown in Fig.3, which brings difficulty to differentiate OoD and InD through reconstruction errors. With our deployed $\ell_2$-normalization, the issue of imbalanced norms is greatly alleviated.

(W4 & L1) Manual kernel selection and parameter tuning

The choices of kernels and their tuning parameters are common in kernel methods. Our comprehensive evaluations on OoD detection show:

• the $\ell_2$-normalization from Cosine kernel plays a pivotal role, where the explicit feature mapping has no hyper-parameters;
• the $\ell_2$-distance preserving property of the shift invariant Gaussian kernel further promotes separability between InD and OoD, where only $M$ (dimensionality in RFF) and $\gamma$ (band width) need to be tuned. Our practice shows that our default settings of $M=2m$ ($m$ is the InD feature dimension) and $\gamma=1$ with superior performances in most cases can be a mild suggestion for practitioners.

In Sec.6, we have discussed this limitation as mentioned by the Reviewer. Nevertheless, it seems that in practical setups, the kernel choice is relatively simple and does not suffer from exhaustive tuning. We would also like to highlight that our work takes the first step of casting a kernel perspective into OoD detection, advocating more future work.

Dear Reviewer,

We hope that our responses above could address your concerns and be helpful for final evaluations on our work.

As the author-reviewer discussion is approaching the end, we would like to inquiry if there is any question, and we would be willing to discuss.

Sincerely, Authors.

(W2) Our method and the boosting scheme

Our proposed method is not simply a boosting scheme, but a detection method via the KPCA reconstruction error being the detection score.

The "bad" results in Table 1
In Table 1, CoRP is under a fair comparison with the KNN method [2], where the detection procedures are both straightforwardly proceeded with the DNN features. Besides, the results in Table 1 are not truly bad, since CoRP outperforms the related KNN method in performances and also efficiency as in Table 2. Note that CoRP also achieves better detection than other compared popular baselines, which do not involve boosting techniques.

The boosting scheme
In Table sSKR-2 of our rebuttal and Table 3 of Sec.4.2, the boosting scheme is adopted, as we aim to present a fair comparison with the relevant PCA-based method [6]. The PCA method alone cannot achieve reasonable detection performance, unless its detection score gets fused with other methods as shown in [6], e.g., ReAct+PCA. Therefore, we report the results of our KPCA detector under the same boosting scheme, e.g., ReAct+CoRP, which refers to the "ours" in Table sSKR-2 above.

Remark: We would like to point out that the strong SOTA methods compared in Table sSKR-2 and Table 3 are specifically designed with incorporating boosting schemes, e.g., the "DICE+ReAct" in Table 3 and SCALE [1], DDCS [2] in Table sSKR-2, where the feature-pruning-based detectors SCALE [1] and DDCS [2] are boosting detection scores on logits by clipping features. Therefore, in this set of experiments, our boosting implementation is fair for comparisons. Without the boosting technique, our method also achieves superior performances than the compared methods in Sec.4.1.

(W3) Hyper-parameter selection

Maintaining a high true positive rate (TPR) and our hyper-parameter tuning principle are elaborated below.

• Maintaining a 95% TPR is a common setup in current evaluations on OoD detection methods. Specifically, for any detection method, it is required to achieve a 95% TPR on the InD test set firstly to assign the value of $s$ in Eqn.(1), then, based on the assigned $s$, the detection results on OoD datasets (FPR and AUROC) are determined, which is a widely-adopted procedure in evaluating OoD detection methods.

• The hyper-parameter selection is to obtain the best detection FPR values on OoD datasets, in order to find the best performance that can be achieved by the detection method, which is also a common setup in existing works. For example, in prevailing feature-pruning-based OoD detectors [1,2,3,5] with the pruning thresholds as hyper-parameters, their detection results are recorded under a variety of different thresholds and the best results are reported finally, which provides a comprehensive understanding on the feature pruning. Regarding our KPCA detector, we also give a detailed analysis on the effect of each hyper-parameter in our method in Appendix D.3, for in-depth insights on the KPCA detector.

Dear Reviewer,

Thanks for your kind reply. Below we provide a pointwise response to each of the further comments.

(W1) Motivation behind our work

(A1) The motivations from PCA to KPCA are elaborated below.

• The linear transformations by simply applying PCA [6] cannot capture informative principal components of InD features. As shown in Fig.1(a), InD features are not even located compactly nor linearly-separable with OoD features. In this case, the axes extracted from PCA are insufficient to depict the intrinsic patterns of InD features, so the both InD features and OoD features can have bad reconstructions, hindering to well differentiating OoD from InD.
• Despite of the current bad performance of PCA, the low-dimensional subspace property is commonly considered to boost robustness and thus should be of great interest in OoD detection, as long as informative components can be extracted, such that the intrinsic patterns of InD features are kept, while removing redundant information. Hence, introducing appropriate non-linear feature mappings is promising as a remedy to the existing obstacle, motivating us to construct our KPCA-based detector.
• With KPCA, we can proceed the task in a new non-linear feature space, where with proper kernels (new feature spaces), the corresponding InD mappings are located more compactly and even almost linearly separable from OoD mappings, as shown in Fig.1(b). In this mapped space, KPCA can learn informative principal components from InD data which produce distinguishable reconstruction errors w.r.t OoD.

For the raised two questions, we answer below.

(Q1) Does PCA significantly impact OoD detection?
Yes. (K)PCA learns a low-dimensional subspace spanned by the (nonlinear) principal components from the InD data. All InD data are utilized to learn such a low-dimensional subspace, while KNN [2] only utilizes the nearest affinity neighborhood in InD data. With the low-dimensional subspace technique, it is expected that the most informative patterns of InD are kept and redundant information in InD is removed, and meanwhile OoD features are easily separable in such space. In this way, based on projections onto the subspace, the good reconstruction of InD could well differentiate InD and OoD and serve as an effective OoD detection score. Here, the critical problem is how to find an effective low-dimensional subspace in an efficient way, which is well resolved by our proposed KPCA-based detector. Figure 1 can be referred for illustration.

(Q2) Is the importance of non-linear, low-dimensional structures evident in OoD detection?
Yes. The importance of low-dimensional structures has been elaborated in the response to (Q1) above. This can also be observed from the improved performance of CoRP than KNN, which all utilize the $\ell_2$ distance/similarities between normalized InD features in Table 1. The non-linearity lies in the non-linear kernel which is in correspondence to a mapped feature space, elaborated in (A1) above. With proper non-linear transformations, significantly improved separability between InD and OoD is attained, as shown by the improved performance of CoP/CoRP than PCA in Table sSKR-1.

Thank you for catching this issue. The pseudo-inverse is indeed missing in Eqn.(12). Apologies for this typo.