Uncertainty-aware Fine-tuning on Time Series Foundation Model for Anomaly Detection

1. The paper directly uses several existing techniques, such as MoE, LoRA, probability modeling and Gumbel-Softmax, into existing time series foundation models for fine-tuning. The contributions and novelty are limited.

2. There are no experiments to show or theoretical proof of why MoE and LoRA can learn the boundaries of normal and anomalous patterns. More experiments or analyses would help demonstrate how or why MoE and LoRA learn these boundaries.

3. More experiments or analyses would help demonstrate why probability modeling can solve the anomalies in the dataset. However, I do not think it can solve the challenges in this paper. The probability modeling anomaly scores in this paper are also optimized for the anomalies in the dataset. I believe it cannot solve the challenges in this paper.

4. The calculations of MoE and a set of reconstruction samples are much more complex than the baselines in both time and space complexity. No analysis is conducted on efficiency and effectiveness. More experiments or analyses would help demonstrate the tradeoff.

5. As the paper uses the probability of a set of reconstruction samples as the anomaly score as contributions, the authors should also include related works of existing probability models and compare them by novelty and effectiveness, such as [1] and [2]. The authors should also include a specific comparison section that analyzes how their approach differs from and potentially improves upon these existing probabilistic methods for time series analysis.

   [1] CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation. Yusuke Tashiro et al., NeurIPS 2021.

   [2] Diffusion-TS: Interpretable Diffusion for General Time Series Generation, Xinyu Yuan et al., ICLR 2024.

6. More benchmarks, such as UCR [3], and metrics, such as PR-AUC and ROC-AUC, should be considered. The authors should justify their choice of benchmarks and metrics, and explain why they believe their current selections are sufficient or how they plan to expand their evaluation.

   [3] Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress. Eamonn J. Keogh et al., TKDE 2023.

7. No codes available. I find the results in this paper are not reliable. From the original paper such as their baseline D3R, the F1 results are 0.7609, 0.8682, and 0.7812 for PSM, SMD, and SWaT, which are much larger than the results in this paper. Can you explain what is the reason and provide the codes to evaluate your results and ensure reproducibility?

8. The titles are different on the PDF and submission page. The authors should ensure consistency between the title on the PDF and the submission page, and clarify which title they intend to use.

9. Many writing errors need to be improved, such as line 88 $X$ should be $\hat{X}$.

• The claim regarding the contamination. Authors explain in the abstract that the "effectiveness in anomaly detection is often inferior (...) due to anomaly contamination in the training data". The relation between performance and contamination is difficult to assess based on the experiments performed by the authors. For instance, SWaT is not contaminated (as explained by the provided reference Xu et al.), while the performance with ULoRA-MoE is still better. I suggest, if this is a claim by the authors: (a) to provide an estimated contamination for each dataset, (b) to perform an ablation with increasing value of contamination (maybe following Xu et al. on Epilespy and DSADS?)

• Availability of the code/results: is there a confirmation that the code will be provided? In addition, it would be preferable to include the pre-computed anomaly scores for each (method, dataset) of the tables, so that other researchers can directly check for other thresholds/metrics.

• Questions/comments below are related to some missing important details that impact the current rating and soundness assessments.

• The paper is poorly written, and often lacks intuitions and justifications. The paper needs clear motivations on why use MoE and LoRA, and needs clarifications on how MoE and LoRA solve their challenges. There are also issues with symbol definitions, such as the repeated use of \hat{x}, which creates reading obstacles.

• The paper simply combines several existing fine-tuning techniques into existing foundation models without novel contributions. The MoE and LoRA seem not to have relevance to the boundaries of normal and anomalous patterns. The anomaly score function is applied during the inference stage and thus it cannot address the impact of anomalous data during the fine-tuning process.

• The main claims of the paper are not adequately supported with evidence. The paper needs to provide discussion and analyses on why the proposed method can address the challenges mentioned in the introduction and how it effectively tackles them. For example, why and how the sampling-based routing strategy quantify the epistemic uncertainty?

• I find it difficult to believe that various modules proposed by the authors are effective and there are no codes for evaluation. The experiments are incomplete, and the analysis is inadequate. The authors merely list the performance of the model without validating the effectiveness of individual components, such as the Gumbel-Softmax sampling approach and the proposed anomaly score. Simply stating that the methods is SOTA is insufficient. I would prefer to understand which part plays a critical role and how they impacted the results. More insight analysis should be provided, including but not limited to thorough ablation studies, hyperparameter analysis, and visualization analysis.

• The paper should consider additional metrics to provide a more comprehensive evaluation of the model. Besides, using only two foundational models is insufficient. More models, such as Moment[1] and Timer[2], should be considered.

[1] Goswami M, Szafer K, Choudhry A, et al. Moment: A family of open time-series foundation models. ICML'24. [2] Liu Y, Zhang H, Li C, et al. Timer: Generative Pre-trained Transformers Are Large Time Series Models. ICML'24.

1. Much of the methodology follows standard approaches for fine-tuning foundation models (e.g., LoRA and MoE). The uncertainty-aware component based on negative log-likelihood appears limited, as it mainly penalizes highly uncertain predictions. Exploring alternative distributions and more advanced uncertainty quantification methods could strengthen this aspect. 2. The fine-tuning is only tested on GPT4TS and UniTS models, omitting other popular foundation models like Lag-Llama, TimesFM, and TimesGPT, which limits the generalizability and impact of the approach. 3. While the authors claim effective epistemic uncertainty quantification, the proposed approach primarily involves sampling and a negative likelihood score. A deeper investigation into producing confidence bands or capturing anomaly score distributions would better support this claim.

1. Ablation study on the number of LoRA experts is missing. What happens when this increases/decreases instead of being fixed at 5.
2. The authors claim that most anomaly detectors aren't suitable because the training data might be contaminated. They also state that they are in a self-supervised paradigm. Taking into consideration these two statements, most anomaly detectors rely on the LPUE [4] technique and train on only normal data to build a notion of what normality is. Then, they test on both normal and abnormal instances and measure their performances. For fine-tuning where the fine-tuning data might be contaminated with anomalies, I get ULoRA-MoE's contribution. But for fine-tuning data where the contamination is negligible, why wouldn't a pre-trained anomaly detector be sufficient and suitable? (see Question 4).
3. In lines 58-60, the authors claim that anomaly detectors aren't able to have boundaries between anomalies and normal data (whatever "boundaries" means here...). However, although in video anomaly detection, MoCoDAD [3] doesn't need to have knowledge about anomalies whatsoever - again by relying on LPUE. The authors of this paper tackle this "lack of knowledge for anomalies" by conditioning their proposed diffusion-based model with past observations. The gist of the paper is that if the past was indeed abnormal, then the future is most probably anomalous as well and they show it by observing higher reconstruction errors.
4. The authors claim that ULoRA-MoE learns different boundaries of anomalies vs. normalities and that each MoE can capture different types of anomalies. However, the authors fail to show what kind of boundaries are we talking about here. Can these boundaries be used to actually subcategorized what anomaly is happening at timestamp $t$? How come these boundaries are better than just having a normalcy manifold and everything falling outside of this manifold is considered an anomaly? What are these boundaries? How are they traced? Via the proposed epistemic uncertainty?

Minor concerns:

5. Why don't you compare with HypAD [1] and BAE [2], although a bit old, which are relevant to your research?

Missing references:

Uncertainty estimation for free in the hyperbolic space $\rightarrow$ [1] Flaborea A, Prenkaj B, Munjal B, Sterpa MA, Aragona D, Podo L, Galasso F. Are we certain it's anomalous?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 (pp. 2897-2907).

MoE related (boosting-based mechanism) $\rightarrow$ [2] Sarvari H, Domeniconi C, Prenkaj B, Stilo G. Unsupervised boosting-based autoencoder ensembles for outlier detection. In Pacific-Asia Conference on Knowledge Discovery and Data Mining 2021 May 9 (pp. 91-103). Cham: Springer International Publishing.

[3] Flaborea A, Collorone L, Di Melendugno GM, D'Arrigo S, Prenkaj B, Galasso F. Multimodal motion conditioned diffusion model for skeleton-based video anomaly detection. InProceedings of the IEEE/CVF International Conference on Computer Vision 2023 (pp. 10318-10329).

[4] Zhang B, Zuo W. Learning from positive and unlabeled examples: A survey. In2008 International Symposiums on Information Processing 2008 May 23 (pp. 650-654). IEEE.