Towards a General Time Series Anomaly Detector with Adaptive Bottlenecks and Dual Adversarial Decoders

We sincerely thank reviewer EBSp for the valuable comments and suggestions. We have revised our paper accordingly and the changes are marked in blue.

W1 & Q1: Difference between pre-training and evaluation datasets
We list our evaluation datasets in $\underline{\text{Appendix A.1}}$ and our pre-training datasets in $\underline{\text{Appendix A.2}}$.
The differences between the pre-training and evaluation datasets are (1) They are non-overlapping. None of the datasets used for pre-training are included in the evaluation. (2) They come from diverse domains. They include datasets where the domains are similar but the datasets are different, as well as datasets where the domains are significantly different. This indicates that their data patterns are different. (3) Each evaluation dataset comes from a specific domain. The whole pre-train dataset contains multiple domains.
How we select the datasets. We choose 10 commonly used datasets in the anomaly detection field as evaluation datasets to assess model performance comprehensively. For pre-training, we gather datasets from various domains within anomaly detection. Additionally, we use the large-scale Monash dataset to further supplement the pre-training data. $\underline{\text{Table 7 in Appendix A}}$ provides a detailed description of the overall composition of the pre-training datasets we use.

W2 & Q2: More experiment details
Thanks for your suggestion. $\underline{\text{In Appendix A}}$, we introduce the collection and processing methodology for our datasets, covering pre-training data, evaluation data, and anomaly injection. The model processes these data using a channel-independent approach. $\underline{\text{In Appendix A.3 (in the revised paper)}}$, we have added a more detailed explanation of the types of anomalies injected and a comprehensive description of the entire anomaly injection process. Furthermore, $\underline{\text{in our code repository}}$, we have included the relevant code and scripts for anomaly injection.

W3 & Q3: Computation cost
The overall time complexity of the model is O(Encoder + AdaBN + Decoders). We adopt a sparse design in AdaBN where data goes through only top-k bottlenecks, and each bottleneck in AdaBN is designed to be lightweight. Each bottleneck and decoder is implemented using a simple MLP, so O(Encoder)>>O(AdaBN), O(Decoders). Therefore, the time complexity is approximately O(Encoder). We calculate the time cost of a single forward pass for each important module. As shown in the table, the main computational cost of the model comes from the Encoder. Introducing AdaBN and Abnorm Decoder has a minor impact on the model, which is acceptable, considering the improvements in zero-shot performance.

Dear Reviewer EBSp:

We would like to express our sincere gratitude for your time in reviewing our paper and your valuable comments.

Since the rebuttal period is nearing its end, we are wondering whether our response has sufficiently addressed your questions. If so, we would greatly appreciate it if you could consider updating the score to reflect this. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper.

Thank you once again for dedicating your time to enhancing our paper!

All the best,

Authors

We sincerely thank you for the insightful reviews and valuable comments, which are highly instructive for further improving our paper.

In this work, we propose the first general time series anomaly detection model, and enhance its generalization ability and detection performance by introducing novel perspectives. After pre-training, our method, as a zero-shot anomaly detector, achieves state-of-the-art performance compared to models trained specifically for each dataset.

Since the rebuttal period is nearing its end, we kindly ask you to consider raising the score if we have addressed the concerns, which would provide us with a greater opportunity to present our work at the conference.

Thank you once again for dedicating the time to enhancing our paper!

W4: Labeling mistakes
Thank you very much for your detailed review. We have corrected labeling mistakes in the revision.

Q1: Inference time
We compare DADA with other models on the SMD dataset in terms of training time, inference time, and F1 score. Training time refers to the time required to train the model for one epoch on the target training dataset, while inference time refers to the time required for a single forward pass on the target testing set. We compare the baseline models A.T. and ModernTCN, as well as the pre-trained model GPT4TS. Additionally, we pre-train GPT4TS on multi-domain time series and include it in the comparison.
It can be found that ModernTCN is advantageous in terms of inference time, but it still requires training data to train the model on the target dataset. After pre-training on multi-domain datasets, GPT4TS (multi-domain) and DADA can perform zero-shot on the target dataset. However, GPT4TS (multi-domain) requires a longer inference time and, due to not being specifically designed for anomaly detection, its detection performance is not ideal.
In comparison, DADA has a certain advantage in inference time and can effectively perform zero-shot learning without target training data. Thanks for your suggestion, and we have added this experiment $\underline{\text{in Appendix E.11}}$.

Q2: How to differentiate between masked and non-masked portions
$\underline{\text{In Section 3.2 of the paper}}$, we omit the patch embedding process. In fact, after dividing the original sequence into patches, the sequence first undergoes embedding before being masked. This is consistent with the provided code. Points in the original sequence that are 0 are converted into high-dimensional vectors along with other points in the same patch. The mask operation sets the entire patch embedding to 0, distinguishing it from the original 0 points. We apologize for any confusion this may have caused. We have clarified the patch embedding process in the revised paper and revised the dimension of $\mathbf{X}$ in the paper to $P\times d$, where $d$ is the dimension of patch embedding.

We sincerely thank reviewer e5Ze for the valuable comments and suggestions. We have revised our paper accordingly and the changes are marked in blue.

W1: Analysis of the mask rate's effect
Thanks for your suggestion. Since we use complementary masks, when data is masked 30%, its complementary part is masked 70%. Therefore, we analyze the effects of mask rates ranging from 10% to 50%. From the experimental results, we find that maintaining a complementary mask ratio closer to 50% yields better performance. When the mask ratio is too high, the model's reconstruction becomes very difficult; when the mask ratio is too low, it becomes too simple, leading to poorer model performance. We have added the experiment $\underline{\text{in Appendix E.10 (in the revised paper)}}$.

W2: About the fine-tuning result
Thank you for pointing this out. This work focuses on learning strong generalization and anomaly detection capabilities from multi-domain data to create a general anomaly detection model. The good zero-shot results indicate that the model has learned strong generalization capabilities from multi-domain pre-training, which reduces the gains on target datasets. Additionally, the fine-tuned results are 2.5% higher than SOTA using the same training data, which is a significant improvement. We will also explore how to better fine-tune a general anomaly detection model for target datasets in our future work.

W3: Explanation of why maximizing the anomalous time series leads to increased confusion
Thank you for your suggestions and we have added the explanations in the Ablation study $\underline{\text{in Section 4.2 (in the revised paper)}}$.
For normal time series, our model utilizes the encoder and the normal decoder to minimize reconstruction errors, thereby learning normal patterns. For anomalous time series data, the model passes it through the encoder and then to the anomaly decoder. The anomaly decoder aims to minimize the reconstruction loss for abnormal series, allowing it to learn common abnormal patterns. However, the encoder aims to maximize the reconstruction loss, thereby enhancing the model's ability to discriminate between normal patterns and common abnormal patterns, which is a key aspect of the adversarial training.
As illustrated in Table 3, removing the adversarial mechanism leads to a scenario where both the encoder and the anomaly decoder directly maximize the abnormal reconstruction loss simultaneously. This results in the anomaly decoder being unable to learn meaningful knowledge about abnormal patterns, as it can achieve this goal simply by generating arbitrary reconstruction results. Directly maximizing the abnormal reconstruction loss does not teach the model anything and even adds unnecessary noise to the pre-training process, weakening the encoder's ability to model normal patterns. So we call it "confusion".

Dear Reviewer e5Ze:

We would like to express our sincere gratitude for your time in reviewing our paper and your valuable comments.

Since the rebuttal period is nearing its end, we are wondering whether our response has sufficiently addressed your questions. If so, we would greatly appreciate it if you could consider updating the score to reflect this. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper.

Thank you once again for dedicating your time to enhancing our paper!

All the best,

Authors

We sincerely thank you for the insightful reviews and valuable comments, which are highly instructive for further improving our paper.

In this work, we propose the first general time series anomaly detection model, and enhance its generalization ability and detection performance by introducing novel perspectives. After pre-training, our method, as a zero-shot anomaly detector, achieves state-of-the-art performance compared to models trained specifically for each dataset.

Since the rebuttal period is nearing its end, we kindly ask you to consider raising the score if we have addressed the concerns, which would provide us with a greater opportunity to present our work at the conference.

Thank you once again for dedicating the time to enhancing our paper!

We sincerely thank reviewer e12v for the valuable comments and suggestions. We have revised our paper accordingly and the changes are marked in blue.

W1: Anomaly contamination in training sample
Thank you for your suggestions. We have added this experiment $\underline{\text{in Appendix E.8 (in the revised paper)}}$. Existing methods for time series anomaly detection typically assume that the training data are normal. Exploring the impact of potential anomalies in training data on the model is indeed an interesting research direction, but a major challenge is the lack of labels in training data to identify potential anomalies. To explore this impact, we additionally inject anomalies into our pre-training normal data at rates of 0.1%, 5%, and 10%, and treat these data as if they were normal. Our results show that the model's performance is affected to some extent as the anomaly contamination ratio increases. Since 10% is already much higher than the potential contamination in the actual training set, this result is acceptable and our method is generally robust to some anomaly contamination.

W2: About loss curve
Thanks for your suggestions. We have added the loss curves $\underline{\text{in Figure 13 in Appendix E.9 (in the revised paper)}}$. We present the pre-training loss curves for the abnormal and normal parts, denoted as Loss_norm and Loss_abnorm. As training progresses, Loss_norm steadily declines, indicating improved reconstruction of normal data. In contrast, Loss_abnorm initially decreases but then rises slowly, stabilizing later, and is always clearly higher than Loss_norm. This is the expected effect of the adversarial training mechanism, as shown $\underline{\text{in Eq.(7) in the paper}}$, where the Encoder maximizes the abnormal loss, and the anomaly decoder minimizes the abnormal loss. Eventually, Loss_abnorm stabilizes at a steady value, indicating that the balance between the Encoder and the abnormal decoder has been reached and normal and abnormal data can be clearly distinguished. Therefore, the entire training process is relatively stable.

Q1: More details about anomaly injection
Thank you for your suggestions. For anomaly injection, we randomly select a subsequence within the original time series and an anomaly type. If the subsequence has not previously been injected with an anomaly, we inject this anomaly into it. We repeat this process until the anomaly ratio in the entire sequence reaches the pre-defined number.
We have added a more detailed explanation of the types of anomalies injected and a comprehensive description of the entire anomaly injection process $\underline{\text{in Appendix A.3 (in the revised paper)}}$. Additionally, we have added the relevant code and scripts for anomaly injection $\underline{\text{in our code repository}}$.

Q2: How can baseline pre-train and zero-shot
To handle the issue of pre-training on multi-domain datasets, we adopt the channel-independent approach for processing multi-domain data for these baseline models. These models continue to be trained according to their original training methods with multi-domain data and then directly tested on downstream datasets. For these models, the only change is that the training set of the downstream dataset is replaced by the multi-domain datasets, and they are not further trained on the downstream dataset, which allows them to perform zero-shot anomaly detection.

Q3: How can DADA fine-tune the target datasets
For fine-tuning, we discard the abnormal decoder and only use the normal decoder because the boundaries between normal and anomalous patterns are clearer with the target domain training data. We freeze the encoder and AdaBN modules and use $\underline{\text{Eq. (5) in the main text}}$ to fine-tune the normal decoder using the training set of the target dataset.

Dear Reviewer e12v:

We would like to express our sincere gratitude for your time in reviewing our paper and your valuable comments.

Since the rebuttal period is nearing its end, we are wondering whether our response has sufficiently addressed your questions. If so, we would greatly appreciate it if you could consider updating the score to reflect this. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper.

Thank you once again for dedicating your time to enhancing our paper!

All the best,

Authors

We sincerely thank you for the insightful reviews and valuable comments, which are highly instructive for further improving our paper.

In this work, we propose the first general time series anomaly detection model, and enhance its generalization ability and detection performance by introducing novel perspectives. After pre-training, our method, as a zero-shot anomaly detector, achieves state-of-the-art performance compared to models trained specifically for each dataset.

Since the rebuttal period is nearing its end, we kindly ask you to consider raising the score if we have addressed the concerns, which would provide us with a greater opportunity to present our work at the conference.

Thank you once again for dedicating the time to enhancing our paper!

We sincerely thank reviewer hvUu for the valuable comments and suggestions. We have revised our paper accordingly and the changes are marked in blue.

W1: Comparison between AdaBN and MoE
The differences between AdaBN and the previous MoE models go beyond just dimensional variations. Our experiments, as detailed $\underline{\text{in Table 10 in Appendix E.3}}$, demonstrate that AdaBN outperforms the previous MoE models. Thank you for your suggestion, and we have added the reference you suggested.
Different intuitions. AdaBN is specifically designed for multi-domain pre-training in time-series anomaly detection, focusing on tailoring appropriate information bottlenecks for various datasets within a unified model. It ensures that our model can flexibly adjust the bottleneck size to adapt to the different reconstruction requirements of different data. MoE, however, focuses on enhancing the model's representation capabilities by increasing the number of parameters while avoiding additional computational overhead. MoE cannot solve the problem that different data require different bottlenecks in general time series anomaly detection.
Different structures. AdaBN establishes a bottleneck pool containing various bottlenecks with different latent space sizes between the encoder and decoder. It selects an appropriate bottleneck size for the input and then performs a reconstruction. However, MoE replaces the block in the original model by employing multiple experts with identical structures. These multiple experts capture different features and then integrate them.

W2 & Q1: The effect of AdaBN

1. Explanation of Ablation Study. We would like to clarify the misunderstanding. The comparison between the fourth row and the sixth row (final model) in the ablation study illustrates the impact of the Dual Adversarial Decoders. The average improvement of 1.68% is significant. The comparison between the first row, i.e. w/o AdaBN, and the sixth row, demonstrates the improvements brought by adding the AdaBN module. The average improvement of 5.04% is also clearly substantial.
2. Experimental comparison between AdaBN and MoE. Directly comparing the result of MoE 256 in Table 10 with the fourth row in Table 3 is unfair. MoE 256 includes both MoE and Dual Adversarial Decoders, whereas the fourth row only has AdaBN. This comparison does not fairly demonstrate whether MoE outperforms AdaBN. The MoE 256 should be compared with the final model DADA $\underline{\text{in Table 10}}$, and AdaBN outperforms MoE on all datasets.
3. MoE does not always perform well. $\underline{\text{In Table 10 in the appendix}}$, the performance of MoE is comparable with AdaBN on MSL, SMAP, and SMD datasets, but it is clearly worse than AdaBN on SWaT and PSM. From $\underline{\text{Figure 5(left) in the main text}}$, we can see that MSL, SMAP, and SMD datasets are well-suited for a bottleneck size of 256, making it easier for MoE 256 to achieve good performance on these datasets. However, on datasets where the bottleneck size of 256 is less effective, such as SWaT, MoE 256's performance is not as good as AdaBN. This is further validated by adding the MoE 256 results on the Creditcard and GECCO datasets. It shows that MoE cannot solve the problem that different data require different bottlenecks in general time series anomaly detection. In contrast, AdaBN performs consistently across all datasets and shows significant improvements on SWaT, GECCO, and Creditcard compared to MoE 256, aligning with the motivation behind AdaBN's design that the model's bottleneck needs to be adaptively selected.

4. Overall, We conducted extensive experiments to demonstrate the effectiveness of AdaBN. In the ablation study in $\underline{\text{Table 3}}$, AdaBN brought an average improvement of 5.04%. $\underline{\text{Figure 5(left)}}$ illustrates the preferences of different datasets for different-sized bottlenecks. $\underline{\text{Figure 12(left) in Appendix E.5}}$ illustrates that AdaBN indeed can adaptively select an appropriate bottleneck based on the data, further substantiating the importance of AdaBN.

Q2: About complementary mask
The complementary mask is a way to enhance utilization of training samples during mask modeling and we did not claim it as a main technical contribution of our paper. We mention this design for better reproducibility of our method. To further analyze it, we have added a comparison between the ordinary masks that reconstruct the masked part from the unmasked part once and the complementary masks $\underline{\text{in Table 11 in Appendix E.7 (in the revised paper).}}$ In terms of results, the complementary mask slightly outperforms the ordinary mask.

Q3: Visualization analysis
Thank you for your suggestions. We use the PSM dataset, and we have added it in the caption of $\underline{\text{Figure 6 (in the revised paper)}}$. Following your suggestion, we have added a visualization of the original data and a visualization comparison with other baselines in $\underline{\text{Figure 11 in Appendix E.1 (in the revised paper)}}$. From the visualization results, it is clear that DADA distinguishes anomalies from normal points more effectively than ModernTCN and GPT4TS, avoiding many false alarms.

Q4: Generation of Monash+
Thank you very much for your suggestions. We have updated the generation script for Monash+ $\underline{\text{in our code repository}}$. After downloading the data from the Monash dataset link provided in the repository we can generate Monash+ with the script.
$\underline{\text{In Appendix A.3}}$, we describe and visualize the anomaly injection. To further explain the anomaly injection, we have added a more detailed explanation of different types of anomalies injected and a more comprehensive description of the entire anomaly injection process $\underline{\text{in Appendix A.3 (in the revised paper).}}$

Dear Reviewer hvUu:

We would like to express our sincere gratitude for your time in reviewing our paper and your valuable comments.

Since the rebuttal period is nearing its end, we are wondering whether our response has sufficiently addressed your questions. If so, we would greatly appreciate it if you could consider updating the score to reflect this. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper.

Thank you once again for dedicating your time to enhancing our paper!

All the best,

Authors

We sincerely thank you for the insightful reviews and valuable comments, which are highly instructive for further improving our paper.

In this work, we propose the first general time series anomaly detection model, and enhance its generalization ability and detection performance by introducing novel perspectives. After pre-training, our method, as a zero-shot anomaly detector, achieves state-of-the-art performance compared to models trained specifically for each dataset.

Since the rebuttal period is nearing its end, we kindly ask you to consider raising the score if we have addressed the concerns, which would provide us with a greater opportunity to present our work at the conference.

Thank you once again for dedicating the time to enhancing our paper!

We sincerely thank reviewer WTbT for the valuable comments and suggestions. We have revised our paper accordingly and the changes are marked in blue.

W1 & Q1: Differences between Pathformer and AdaBN

1. The motivation of the router is different. The router in DADA is specifically designed for multi-domain pre-training in time-series anomaly detection, focusing on the importance of the bottleneck size for generalization capabilities. It ensures that our model can flexibly adjust the bottleneck size to adapt to the different reconstruction requirements of different time series data, capturing the essential patterns while discarding irrelevant noise. The router in Pathformer primarily focuses on the forecasting task within a single downstream dataset. It emphasizes that the dataset can be modeled at different scales through the use of a routing function to select various patch sizes.

2. The mechanism and implementation of the router are different. DADA selects an appropriate bottleneck size for the representation of time series and then passes the representation through the corresponding bottleneck for suitable reconstruction. Pathformer first selects different patch sizes for time series and divides it into different patches, and then feeds these patches into different transformer blocks for modeling at various scales.

W2 & Q2: Explanation of the reconstruction process
Thank you for your suggestions. We have added more detailed explanations $\underline{\text{in Section 3.2 (in the revised paper)}}$. The Recon(·) function passes its input through the Encoder, AdaBN, and Dual Decoders modules to obtain the reconstruction results. Since some elements of the input are masked as 0, the model attempts to reconstruct these masked elements back to their original values. It can be formalized as:

which is the same as the equation $D(G(X))$ $\underline{\text{in Eq.(5) and Eq.(6)}}$ in the main text, where $D$ is the Decoder, $G$ is the AdaBN and Encoder, and $\mathbf{X}$ is the input. When the input is normal data, we use the normal decoder $D_n$; otherwise, the abnormal decoder $D_a$ is used.

Dear Reviewer WTbT:

We would like to express our sincere gratitude for your time in reviewing our paper and your valuable comments.

Since the rebuttal period is nearing its end, we are wondering whether our response has sufficiently addressed your questions. If so, we would greatly appreciate it if you could consider updating the score to reflect this. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper.

Thank you once again for dedicating your time to enhancing our paper!

All the best,

Authors

We sincerely thank you for the insightful reviews and valuable comments, which are highly instructive for further improving our paper.

In this work, we propose the first general time series anomaly detection model, and enhance its generalization ability and detection performance by introducing novel perspectives. After pre-training, our method, as a zero-shot anomaly detector, achieves state-of-the-art performance compared to models trained specifically for each dataset.

Since the rebuttal period is nearing its end, we kindly ask you to consider raising the score if we have addressed the concerns, which would provide us with a greater opportunity to present our work at the conference.

Thank you once again for dedicating the time to enhancing our paper!