SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors

Normalizing anomaly scores via quantiles (Sec. 3.2) addresses score incomparability. The use of APS (Averaged Precision Score) is appropriate for imbalanced anomaly detection.

Theorem 3.1 cites regret bounds from FTRL literature but does not provide a self-contained proof. While plausible, the paper assumes anomalies are "rare" for at least one detector; this assumption’s impact on regret guarantees is not analyzed.

Comprehensive evaluation across 15 datasets with varying anomaly rates and dimensions. Runtime comparisons with MetaOD (Table 5) highlight streaming efficiency. No ablation study on SEAD’s hyperparameters (e.g., learning rate η, regularization λ). Missing comparison to state-of-the-art streaming ensembles (e.g., LODA, xStream variants).

SEAD addresses a gap in unsupervised online model selection for AD

Most of the references were provided and due to my lack of knowledge in the field, it is difficult for me to provide an accurate judgement.

The method is evaluated on sufficient datasets.

N/A

The reported variances seem very large. Not clear why this happens for SEAD, but this raises questions about the stability of the method.

The paper studies anomaly detection in streaming data, where the assumption is that data distribution can change over time. This is an interesting topic which is understudied in literature.

N/A

SEAD needs to calculate anomaly scores for all base detection algorithms and perform MWU weight updates, which may be computationally expensive. In large-scale streaming data, calculating anomaly scores for all detection algorithms and normalizing them may be too expensive. SEAD++ attempts to reduce computation by subsampling detectors, but still needs to maintain and adjust weights between all detectors.

n/a

1. Only Averaged-Precision score (APS) is used for evaluation , which is insufficient . I suggest the authors to include more comparisons.
2. The experiments in the paper are mainly based on standard datasets (ODDS, USP database, etc.), but lack real industrial data verification. These datasets may not fully simulate the abnormal distribution of the real world (such as network security, industrial fault detection, etc.). The paper only contains an internal telemetry dataset, but does not provide detailed analysis or reproducible experimental details.
3. The paper does not compare with the latest unsupervised anomaly detection model selection methods (such as the time series anomaly detection model selection method proposed by Goswami et al., 2023). The paper is only compared with MetaOD (offline method) and not with other online unsupervised methods. I suggest the authors to choose the latest unsupervised methods for a fairer experimental comparison.

[1] SEAD relies on the accumulated information of historical anomaly scores for detector weighting, but does not explicitly handle concept drift. If the data stream distribution changes significantly (such as anomaly pattern changes), SEAD may maintain outdated detector weights, resulting in performance degradation. Some concept drift literature should be discussed.

[2] Not compared with "Unsupervised Model Selection for Time-series Anomaly Detection". ICLR 2023 spotlight.

n/a

The proposed methods and evaluation criteria are appropriate for the problem of unsupervised anomaly detection in streaming environments, as they align with the challenges of real-time detection, model selection without labels, and handling non-stationary data. The use of a diverse set of anomaly detection algorithms as base models ensures that SEAD’s model selection approach is tested across different detection paradigms. The reliance on anomaly score weighting instead of labeled data makes sense in real-world applications where obtaining labeled anomalies is infeasible. The choice of evaluation datasets is also relevant, as the authors include 15 publicly available datasets that cover different types of anomalies and distributions, ensuring that SEAD is tested in diverse scenarios. The Averaged Precision Score (APS) metric is an appropriate choice for evaluating anomaly detection performance, as it accounts for the highly imbalanced nature of anomaly detection tasks. However, the evaluation could be expanded by testing SEAD’s performance in high-dimensional datasets or comparing it against deep learning-based anomaly detectors, which are increasingly used in large-scale anomaly detection problems.

The paper presents theoretical guarantees for SEAD, particularly in the form of a regret bound. This theoretical result aims to justify that SEAD's selection strategy will be competitive as the performance of the best anomaly detector in hindsight, even without labeled data.

The experimental design in the paper appears well-structured, with evaluations conducted on 15 public datasets, covering a diverse range of anomaly detection scenarios. One aspect that could be further scrutinized is the robustness of the experimental setup across datasets with significantly different anomaly distributions. The paper does not provide a detailed breakdown of how SEAD performs on datasets with varying anomaly rates, which could impact its effectiveness.

The key contributions of the paper are well-situated within the broader anomaly detection and online model selection literature, but their novelty and impact appear somewhat incremental rather than groundbreaking. The idea of leveraging multiple anomaly detection models in a streaming setting is an interesting direction, aligning with prior work on ensemble-based anomaly detection and meta-learning for model selection. However, the main theoretical foundation of the method—Multiplicative Weights Update (MWU)—is not novel and has been extensively studied in online learning literature. While the paper applies MWU in a new context by using it for online model selection in anomaly detection, the underlying theory itself is borrowed from well-established frameworks. From an empirical standpoint, SEAD demonstrates competitive but not significantly superior performance compared to existing baselines. The results indicate that while SEAD adapts dynamically to different data distributions, its overall detection accuracy does not show a substantial improvement over simpler ensemble baselines such as mean and max aggregation of anomaly scores. This suggests that while the method is effective, its contribution in terms of practical anomaly detection performance is not dramatically beyond existing approaches. Furthermore, while the experimental validation is comprehensive in terms of the number of datasets (15 public datasets), the selection is somewhat limited in diversity, focusing primarily on tabular datasets. The inclusion of vision-based anomaly detection benchmarks or datasets with high-dimensional, structured data (e.g., image or sensor data) could have strengthened the paper’s practical contributions by demonstrating the generalizability of SEAD beyond classical tabular anomaly detection tasks. Expanding the evaluation scope would provide stronger evidence of SEAD’s applicability in real-world, high-dimensional anomaly detection problems. In summary, while SEAD presents an interesting adaptation of MWU to streaming anomaly detection, its contribution is somewhat incremental in both theoretical and empirical dimensions. The method does not introduce a fundamentally new learning principle, and its empirical improvements over baselines are relatively modest. A broader experimental validation across diverse domains, such as vision-based anomaly detection, would have further enhanced its practical significance.

While there may be newer or alternative approaches to anomaly detection ensembles that are not explicitly cited, none appear to be critical omissions that would significantly alter the context or positioning of SEAD’s contributions.