We thank this reviewer for the feedback on how to improve our paper. We updated the paper to increase its clarity based on the feedback given (e.g. on distance, bullet points, related works section, small typos). We proceed to answer other questions.

Why VC dimension?

As mentioned in the global response, VC dimension analysis shows that equality separation (ES) is more expressive than halfspace separation (HS), not just a linear solution to XOR. The generalization bound can be directly obtained with VC dimension if desired.

Tutorial-like narration

We explain what locality is to motivate the need for quantitative metrics (closing numbers). It was our goal to explain to a wider audience who may not be familiar with both learning theory and anomaly detection (AD) literature. If this presentation is problematic, we ask this reviewer to suggest how they think we should edit the presentation.

Why ES? Why not kernel SVMs? Can ES work without kernels?

XOR is our starting point: we point out that the widely known “fact" that XOR cannot be classified with linear models is indeed false. No kernels needed.

In general, the motivation of using any classifier depends on the use case. In our work, we observe interesting properties of ES for AD and corroborate this in our experiments. For instance, we observe that kernel ES perform much better than kernel SVMs in our synthetic supervised AD experiment – it is ES that performs well, the kernel merely does feature embedding.

We do not over-claim that (linear) ES works for every task: raw data may not be linear classifiable. We use standard methods to embed raw data into feature space before linear classification: RBF kernels and neural networks in tabular datasets, and a pre-trained computer vision model (DINOv2) for MVTec defect dataset. What we claim is that the paradigm of ES is better for AD, that we should impose one class (the normal class) to have more structure than the other (the anomaly class). We show this theoretically with closing numbers and empirically with supervised AD experiments.

Failure modes? E.g. Fig 1e.

If unseen anomalies fall in the margin, they will indeed be misclassified. Hence, the need for finite-volumed decision regions is important, and we may not be able to achieve that with just 1 ES. This observation motivates our introduction of closing numbers (especially when paired with neural networks, where neurons in a hidden layer work together), which balances the learnability and the openness of the decision region (Figure 2).

Although ES may not classify everything correctly, we observe that it can classify the brown class better than the corresponding HS: the set of misclassified non-brown points from ES is a proper subset of misclassified non-brown points by HS.

Discussion on $\epsilon$?

The margin width is $\frac{2\epsilon}{||\mathbf w||_2}$. In Appendix F1 and F3 (Figure 8), we discuss with respect to the reparameterization of $\epsilon$ to $\sigma$ in the bump function.

Our main goal was to assess the separation of normal data from anomalies – if there is no/little separation, choosing a threshold would be moot. Hence, we use AUPR as our metric. During inference, the (output) threshold depends on the tolerance on the false positive rate. Since the tolerance is domain-dependent, we do not prescribe a threshold in our general framework.

AD Benchmarks

NSL-KDD is an improved version of the KDDCUP99 dataset mentioned in the website given by this reviewer — it is a standard cyber-security benchmark. In this revision, we included the standard MVTec defect dataset used in AD (for instance, in Explainable Deep One-Class Classification [1]) and thyroid dataset from the suggested website.

Deep One-Class Classification (Deep SVDD) discussion?

In our paper, we use the improved Deep Semi-supervised Anomaly Detection (SAD) as a SOTA benchmark, which is Deep SVDD that can use labels. This discussion is in section 3.1 and 4.2 e.g. “SAD is … an RBF separator…”.

Possible to graphically show other examples?

We are unsure of what this reviewer means and would like to ask for clarification.

Modeling halfspace with ES and ReLU?

As per section 2.2, ES acts as a HS by splitting the space into the positive and the zero parts of the ReLU.

Optimality of 2 ES for HS?

We answer this in the last 2 sentences of section 2.2 from a robust perspective. Support vectors may be outliers far from high density regions, so using them may not be as robust as fitting a hyperplane for each class (which depends more on the center of mass).

Meaning of “conservative”?

A more conservative classifier is less likely to classify a datum as positive, in line with density level set estimation. Closing numbers is a property of a class of classifiers that quantify conservativeness, motivated by using these classifiers in neural networks.

[1] Liznersk et al., Explainable Deep One-Class Classification. ICLR 2021.

We thank this reviewer for affirming the creativity of equality separation, the theoretical analysis provided and the communication of our ideas. We acknowledge this reviewer’s request for more datasets in our global response (adding MVTec and thyroid dataset results), and thank this reviewer for noting the thorough experimentation on NSL-KDD dataset. We address other concerns below.

Why don’t equality separators have the best performance for NSL-KDD dataset in all metrics, especially over (unsupervised) baseline methods? Are there intuitions why?

In short, equality separation is an attempt to merge the benefits of unsupervised methods into supervised methods. We observe a significant improvement over supervised methods for unseen anomalies while maintaining supervised-level performance on seen anomalies.

We do not claim that equality separation is or should be the final solution to supervised anomaly detection (AD). We do claim that, among methods that use label information via standard empirical risk minimization, equality separation is more suitable, especially to detect unknown anomalies. The experiments support this claim: for instance, in regular binary classification, equality separation significantly improves AUPR on detecting unseen anomalies (privilege and remote access attacks) by 3-fold compared to halfspace separation.

Comparing equality separation to unsupervised AD baselines, we especially see that OCSVM beats all other methods in detecting unseen anomalies (privilege and remote access attacks). However, unsupervised methods like OCSVM cannot capitalize on labels, hence our comment in footnote 5 that unsupervised methods are not our main apples-to-apples comparison. The inability to use labels impacts detection on seen anomalies (DoS attacks). This disadvantage is in addition to the computational load of computing kernels (e.g. for OCSVM) for large datasets like NSL-KDD. Furthermore, equality separation outperforms unsupervised baselines Isolation Forest and Local Outlier Factor on the unseen remote access attacks, while other regular binary classifiers are far from it. We conclude that equality separation is a step in the right direction to use limited anomaly labels to detect both seen and unseen anomalies.

Our intuition on why equality separation may not be beating the OCSVM baseline in detecting unseen anomalies relates mainly to implementation details – given that equality separation had excellent performance in our synthetic experiments, we posit that higher dimensional data amplify certain difficulties. We observe that optimization may be more difficult because of more sensitivity in initializations and vanishing gradients. We are currently working on understanding how the activation function, loss function and optimization scheme (including initializations) work together to form closed decision regions, which we posit may be the key to further improvements.

Bold Results for HS-NS in Table 3?

We consider both the mean and standard deviation when determining which is the best. This is to account for variability in running the optimization that arises from different initializations. To avoid confusion, we account for this consideration in our discussion and appendix instead and remove the double bold text in the table in the main body.

We thank this reviewer for appreciating the innovation of equality separation, effective visualizations and theoretically sound analysis. We address this reviewer’s question on general applicability (to other problem domains like computer vision) in our global response (adding MVTec and thyroid dataset results). We address other questions here.

There are some useful sections in the appendix that would benefit to be in the main body.

We thank this reviewer for reading through the details in the appendix. We are willing to move relevant sections of the appendix to the main paper. Which sections does this reviewer think are relevant to move to the main body?

The proposed method is tested only on straightforward datasets. Can it be generalized to other datasets such as images?

Although we use the toy dataset for enhanced visualizations and understanding (both in the main body and appendix), we use NSL-KDD as a real cyber-attack dataset. NSL-KDD is an (improved) edited version of the KDDCUP99 dataset, which was part of USA DARPA’s Intrusion Detection System evaluation program, with “about 4 gigabytes of compressed raw (binary) tcpdump data of 7 weeks of network traffic, which can be processed into about 5 million connection records, each with about 100 bytes. The two weeks of test data have around 2 million connection records” [1]. Real cyber-attack datasets allow us to test methods against adversaries who are actively trying to circumvent and break the system.

Our method can indeed be generalized to other domains – there is nothing specific to tabular or cyber-attack data. We have included results for MVTec image defect dataset, showing better AUPR in 5/6 objects tested. We also experiment with the thyroid dataset, which focuses on a medical application. Our method once again shows better performance in detecting seen and unseen anomalies.

Will gradients vanish in deep networks?

Gradients will vanish outside the margin (proportional to $\frac{1}{||\mathbf w||_2}$) as plotted in Figure 4 (Appendix E1), similar to vanishing gradients in sigmoid or the dead neuron problem in ReLU. However, like sigmoid, gradients are the greatest in the regions close to the boundary between positive and negative (i.e. regions of aleatoric uncertainty).

There are 2 sides of this coin. The conventional perspective is that gradients that vanish inhibit learning, and we mention in Section 4 and Appendix F1 that we can mitigate this by initializing $\sigma$ to be large (e.g. $\sigma=10$). This decreases the sensitivity in weight initialization.

On the flip side, there is a self-regularizing effect, where the neuronal activation is very selective. This is indeed what we desire in anomaly detection, where unseen inputs should not activate the network. In Appendix F3 (Figure 8), we show in Figure 8 how the margin width changes across hidden layers (grows across time in earlier layers for more information flow and shrinks in later layers for selectivity of information/features).

Advantages of doubled VC dimension in contemporary neural network architecture?

As mentioned in our global response, the VC dimension analysis was not aimed at a neural network analysis. It shows that equality separation is indeed more expressive than halfspace separation (i.e. not that equality separation works for XOR and nothing else).

In neural networks trained for anomaly detection, we champion for using the equality separation paradigm, which has an inductive bias towards closed decision regions while maintaining the capacity to learn. We think of closing numbers to anomaly detection and forming closed decision regions as VC dimension is to risk minimization – closing numbers is a dimension-like combinatorial number describing the capacity of a hypothesis class to minimize open space risk. Closing numbers also have a direct interpretation in a neural network setting, as we described in section 3.3.

[1] M. Tavallaee, E. Bagheri, W. Lu and A. A. Ghorbani, "A detailed analysis of the KDD CUP 99 data set," 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications, Ottawa, ON, Canada, 2009, pp. 1-6, doi: 10.1109/CISDA.2009.5356528.