Conformal Anomaly Detection in Event Sequences

Thank you for your appreciation of our work and suggestions. We provide answers to your concerns as follows:

Q1: While the authors provide theoretical guarantees on FPR control, the practical implication and significance of these guarantees in real-world scenarios could be explained in more detail.

R1: FPR control guarantees are crucial for safety-critical applications, such as cybersecurity, finance, and healthcare. For example, in electronic health records (EHRs), maintaining low false positive rates is vital. High false positives could lead to incorrect diagnoses or unnecessary treatments, potentially compromising patient safety. Our method ensures that detected anomalies are more likely to be genuine, helping healthcare professionals make more reliable decisions. We will add this content in our paper.

Q2: The paper could explore the potential for extending the proposed method to other types of sequential data, such as time series or spatial-temporal data, to broaden its applicability.

R2: In this work, we primarily focus on anomaly detection in event sequences. Unlike other types of sequential data, event sequence data is asynchronous and of variable length, commonly modeled using temporal point processes. As you suggested, we plan to extend our method to other types of sequential data as future work.

Q3: It would be clearer to denote the new observation $X$ in the problem statement of Section 2 as $X_{\text{test}}$.

R3: Thank you for pointing this out. We have already made the replacement in our manuscript.

Thank you for your recognition and feedback. We provide answers to your concerns as follows:

Q1: It would be more comprehensive if the literatures on applying conformal inference to time series were discussed.

R1: Thank you for your valuable suggestion and the references you provided. We will incorporate a discussion of them in the paper. In this work, we specifically apply conformal inference to event sequences. Although both time series and event sequences are types of sequential data, they differ significantly [1]. Specifically, in time series, time serves only as the index to order the sequence of values for the target variable. In event sequences, time is treated as a random variable representing the timestamps of asynchronous events, commonly modeled using temporal point processes. In addition, the references you mentioned primarily focus on prediction tasks, while our work addresses the anomaly detection task.

Q2: What would the experimental performance be like if the KL divergence in Eq.(7) and Eq.(8) is replaced by the $L_2$ norm?

R2: We conducted experiments by replacing the KL divergence with the $L_2$ norm on real-world datasets, referring to this method as CADES-$L_2$. The results demonstrate that CADES with KL divergence outperforms CADES-$L_2$ in terms of AUROC.

• AUROC (%) results

Reference:

[1] Xiao S, Yan J, Yang X, et al. Modeling the intensity function of point process via recurrent neural networks. AAAI, 2017.

Thank you for your insightful comments and questions. We provide answers to your concerns as follows:

Q1: It is suggested to explain the reason behind the poor performance of the 3S statistic under the Uniform scenario in Section 4.1.

R1: This is because the 3S statistic is not sensitive to relatively uniform spacings (i.e. inter-event times). Specifically, according to Proposition 1 in [1], the value of the 3S statistic for a standard Poisson process realization (i.e. an ID sequence) is around 2. In the Uniform scenario, the spacings of the OOD sequences are identical. For example, when the detectability parameter $\eta = 0.5$, the spacings are all 2. In this case, the 3S statistic for an OOD sequence is equal to 2, making it unable to distinguish between ID and OOD sequences. We will add this explanation in Section 4.1.

Q2: In Figure 7, why are the distributions of ID scores different in the RenewalA and SelfCorrecting scenarios?

R2: Since the bandwidth of the non-conformity score $s_{\text{arr}}(X)$ is selected differently in the RenewalA and SelfCorrecting scenarios, the value of $s_{\text{arr}}(X)$ is different, leading to differences in the distribution of ID scores.

Q3: Could you provide the computational time of the CADES method?

R3: During the training phase, for fairness, CADES employs the same neural TPP model as the baseline methods. Thus, its training time is similar to that of the baselines. We compare the inference runtimes of CADES and the baseline methods on real-world datasets in Appendix D.1, showing that CADES achieves comparable runtimes. For convenience, we also present the experimental results below:

• Inference runtimes (in seconds)

Q4: How would CADES perform in scenarios where event sequences have sparse data?

R4: Our method is flexible, allowing any neural TPP model to be plugged into CADES. Therefore, we can use more advanced neural TPP models or models specifically designed for sparse event data to capture the distribution of the normal training data, and then apply the proposed test procedure for inference on the test data.

Reference:

[1] Shchur O, Turkmen A C, Januschowski T, et al. Detecting anomalous event sequences with temporal point processes. NeurIPS, 2021.