Con4m: Context-aware Consistency Learning Framework for Segmented Time Series Classification

W1. We apologize for any confusion caused. Fig. 1 is intended to emphasize that, despite the similarity in seizure patterns, different annotators can still provide inconsistent labels for the same type of seizure and across different recordings from the same patient. For simplicity, we only illustrated a segment of brain signals to represent a seizure pattern type, rather than a single instance. This will be explicitly stated in the revised version. Additionally, inconsistent labels do not equate to erroneous labels, as there is a lack of unified quantitative metrics to define boundaries between different classes.

---

W2. Our focus is on supervised TSC, which assigns each segment a single label, as stated in the problem definition in Sec. 3. We acknowledge that the studies you referenced consider temporal consistency. However, the first paper focuses on clustering, which is fundamentally distinct because it involves setting/learning the number of clusters within a sequence. The second paper concentrates on active learning. While these studies leverage temporal consistency, they do not align directly with the supervised TSC models we discuss. And our work explicitly integrates temporal consistency into the modeling of supervised TSC from both the data and label perspectives.

---

W3. The only similarity between the two is the parallel computation of $G^l$. Anomaly Transformer uses the similarity between inter-sequence and intra-sequence distributions to measure anomaly scores, whereas our approach aims to achieve smoother representations through fusion. We have cited this work in the main text (line 143) and have not referred to its code.

---

W4. While Theorem 2 may seem intuitively obvious, we provide a more formal proof from an information-theoretic perspective (lines 109-121). Based on this theorem, we designed our model from both the data and label perspectives, as detailed in Sec. 3.1 (lines 142-149) and 3.2 (lines 170-173, 182-187).

---

W5. Assuming the number of consecutive input time segments is $L$, the hidden representation dimension is $D$, the number of classes is $C$, and the internal iteration steps of the Tanh function fitting are $I$, the time complexity of the function fitting module is $\mathcal{O}(ICL)$. The complexity of the encoder’s final prediction layer is $\mathcal{O}(DCL)$. In practice, keeping $I < D$ helps manage the time complexity of the function fitting module. In our experiments, we set $I=100$ and $D=128$, with average training times reported per epoch.

The additional computational cost is higher for the Sleep dataset due to its classification as a five-class problem. SEEG, like fNIRS, is a binary classification problem. However, the additional overhead from function fitting is relatively smaller due to the larger input sample window length and the total number of training samples. As a result, the function fitting module has a higher computational overhead on datasets with more classes, which is a limitation of our model design. However, this overhead becomes less significant with larger datasets.

---

W6. Our assumption is that annotators “tend to reach an agreement on the most significant core part of each class,” which is common in temporal action segmentation. We apologize for any confusion caused by Fig. 1; we used real data, so the illustration does not depict an entire seizure episode due to its length. In practice, doctors often disagree on both the exact start and end times of seizures. While domain knowledge can aid in sample selection, our design remains general when such prior knowledge is unavailable.

---

W7. We acknowledge in lines 334-335 that modeling transitions for a single class is a limitation. However, based on domain knowledge and statistical information from the datasets, we can select an appropriate window length that meets this condition. For the auxiliary parameters $N^l$ and $E_g$, we do not consider them core hyperparameters of the model. We ensure that the model’s loss approaches convergence for the new level data within $E_g$ epochs. See more details in global rebuttal G2.

---

W8. Although we tuned $E_\eta$ on the SEEG dataset, we used the same hyperparameter across all 4 datasets, demonstrating Con4m’s robustness. According to Appendix C, $E_\eta$ should not be set too low. For practical use, start with a value of 10 and adjust in intervals of 10 until the validation set performance peaks or slightly declines.

---

W9. The fNIRS and Sleep datasets are publicly available with curated labels, so they are considered noise-free. In contrast, SEEG data is derived from real clinical datasets and annotated by multiple experts, resulting in naturally inconsistent labels. We employ a voting mechanism (Appendix G, lines 746-749) to minimize discrepancies in test labels. However, due to the lack of a unified standard, inconsistencies in SEEG do not imply “incorrect” labels but rather reflect differences in experiences. Unlike video segmentation, where single-frame semantics are clearer, SEEG annotations can vary significantly.

Our goal is to harmonize, not correct, these labels using methods inspired by noisy label learning. Therefore, we cannot directly calculate an r value for SEEG but instead use indirect label substitution experiments (Sec. 4.3) to validate Con4m’s effectiveness in label harmonization. Comparisons with other noisy label learning baselines and ablation studies further demonstrate Con4m’s ability to handle inconsistent labels.

Thank you for your responses. I have carefully read the authors' rebuttal.

(1) As I mentioned earlier, the overall presentation is somewhat confusing. I now better understand the authors' original intention. However, the current draft needs a major rewrite. The differences between erroneous and inconsistent boundary labels are still not clear to me. To my understanding, this paper deals with possibly erroneous boundary labels caused by inconsistent annotations. Because we cannot review the revised draft, it would be hard to increase the current rating.

(2) The responses on the novelty issues (W2 and W3) and the generalizability issues (W7) are not very convincing to me.

(3) The in-depth analysis w.r.t. the degree of noise or inconsistency (W9) would be necessary to further strengthen this work. A voting mechanism applied to SEEG means curation? If you apply the voting mechanism to the training set of SEEG, it becomes the same status as fNIRS and Sleep? There are several unclear points regarding the motivation and experiment setting.

Overall, I would like to keep my current rating.

W1&Q1. We consider $N_l$ and $E_g$ as auxiliary hyperparameters that work together rather than core hyperparameters of the model. When selecting these values, we focus on ensuring that the model’s loss approaches convergence for the newly added level within $E_g$ epochs. In contrast, $E_\eta$ has a more significant impact on the model’s performance. See more details in global rebuttal G2.

---

W2. From an architectural perspective, Con4m is built on the Transformer framework, allowing for scalability by adjusting the hidden layer dimensions or stacking layers. From a data perspective, time series data differs from NLP sequences as it consists of continuous numerical values rather than discrete tokens. By adjusting the duration of each time segment (patch), we can control the sequence length L. Consequently, Con4m can be scaled to handle longer time series by appropriately tuning the patch length, the number of CNN layers, and the size of the convolutional kernels.

---

W3&Q4. Our method differs from semi-supervised TSC methods that assume reliable labels on partially labeled datasets. In contrast, we begin with initial labels for the entire dataset, but their reliability remains unclear. Our model is specifically designed to evaluate label reliability as part of its learning process, addressing inconsistencies not typically covered by semi-supervised methods. Therefore, we cannot consider semi-supervised or few-shot methods as a baseline or apply them to our task due to these differences in task settings.

---

W4. Thank you for your valuable suggestion. Based on the link you provided for WOODS, we identified that the Human Activity Recognition (HHAR) dataset aligns with our experimental setup to some extent. Therefore, we incorporated the HHAR dataset in our experiments. Please refer to the global rebuttal G1 for more details.

---

Q2. Table 2 demonstrates that fNIRS has relatively clear boundaries, as TAS models perform well on this dataset even without considering label consistency. Therefore, to more effectively highlight Con4m’s advantages in handling inconsistent labels, we decided not to include the fNIRS dataset in the ablation studies.

---

Q3. We believe that due to the significant differences in architecture and domain among various baselines, directly comparing computational complexity would be unfair. For example, models based on the Transformer architecture have a significantly higher time complexity than those based on CNNs. Therefore, we only compare the time complexity of Con4m with that of a standard Transformer.

The time complexity of Con4m can be divided into two main parts. The primary computational cost is on the same order as vanilla Transformer. Assuming the number of consecutive input time segments is $L$, the hidden representation dimension is $D$, the number of classes in the classification task is $C$, and the local iteration count of the function fitting module is $I$.

• Con-Transformer: The time complexity of vanilla self-attention is $\mathcal{O}(LD^2+L^2D)$, and the time complexity of the Gaussian kernel branch is $\mathcal{O}(LD^2+L^2)$.
• Coherent class prediction: The time complexity of Neighbor Class Consistency Discrimination is $\mathcal{O}(LDC)$, and the time complexity of the Tanh function fitting is $\mathcal{O}(ICL)$.

Therefore, the computational cost and bottlenecks of Con4m are similar to vanilla Transformer.

---

Q5. Since label inconsistency is difficult to precisely define, it is challenging to determine an optimal percentage of label modifications. However, during training, one of our criteria for stopping is when the model no longer changes additional labels. You can find this implementation in our code. Generally, datasets with higher noise levels tend to have a higher percentage of label changes. For example, the fNIRS-0 dataset has a change rate of 3%, while SEEG has a change rate of 13%. Similarly, for the Sleep dataset with disturbance ratios of 0%, 20%, and 40%, the change rates are 10%, 12%, and 18%, respectively.

---

Q6. The Tanh function is particularly well-suited to encapsulate the four behaviors depicted in Figure 3. As a widely used activation function, it is easier to understand and manipulate. In contrast, functions like arctangent and arcsine are more complex and harder to control. Appendix B demonstrates Tanh’s ease of optimization and its precise fitting capabilities.

W1&2.

1. Our setup differs significantly from segmentation models (FLOSS[1], ESPRESSO[2], ClaSP[3]) in that they are able to identify change points but are unable to determine the specific classes before and after these points, particularly in multi-class tasks.
2. Our public datasets are multivariate, which presents a challenge for the performance of most segmentation models (ClaSP[3]), which are designed to handle univariate time series.
3. Our use case may be impractical and complex due to the necessity of setting/learning parameters, including the number of segments and the length of subsequences, in segmentation models (FLOSS[1], ESPRESSO[2]).

Nevertheless, we conduct an exploratory analysis of ClaSP on the SEEG dataset (the only univariant dataset) and present the results with two metrics for reference. These results are solely intended for illustration purposes, as unsupervised segmentation models are not well-suited to our scenario and setup.

[1] Gharghabi, Shaghayegh, et al. "Domain agnostic online semantic segmentation for multi-dimensional time series." (2019).

[2] Deldari, Shohreh, et al. "Espresso: Entropy and shape aware time-series segmentation for processing heterogeneous sensor data." (2020).

[3] Ermshaus, Arik, Patrick Schäfer, and Ulf Leser. "ClaSP: parameter-free time series segmentation." (2023).

---

W3. Thank you for your valuable questions and suggestions. We will reorder Figures 4(a) and 4(b) and attempt to combine Figures 2 and 3 into a single illustration. However, we would like to emphasize that Con4m was not designed specifically for segmentation. Our modeling is consistently based on Problem Definition 3.1, which involves making independent predictions for each segment and then integrating them into coherent predictions. Our work aims to highlight a point often overlooked by current mainstream supervised TSC approaches: the temporal dependency and coherence of contextually classified samples. Experiment 4.3 further illustrates the contributions of our work.

On the other hand, all the baselines related to temporal action segmentation (TAS) are equipped with segmentation capabilities, as they are specifically designed for segmentation tasks. Since Con4m is a supervised learning model, we selected representative supervised learning models in TAS for comparison. Therefore, the selection and comparison of baselines are fair. Nevertheless, we have considered your recommendation to incorporate unsupervised segmentation models for comparison in W1&2.

---

Q3. Our experimental setup is consistently aligned with Problem Definition 3.1 (lines 136-140). All models are trained on the disturbed data and tested on the original noise-free data. We introduce disturbances at the timestamp level of the original data (lines 236-243), then sample time intervals based on these disturbances and segment them. All models treat each time segment as an instance for prediction and evaluation, which aligns with the TSC task setup. The evaluation setup itself is independent of whether a model can detect boundaries. Con4m does not explicitly perform boundary detection, whereas TAS models are specifically designed for boundary segmentation.

---

Q4. Section 4.3 effectively demonstrates Con4m’s prediction consistency. Moreover, Fig.5 and Fig.8 use color coding to visually highlight the alignment between the model’s predictions and the actual labels. It is evident that, unlike Con4m, other models fail to provide predictions that are both coherent and accurate. In practical applications, Con4m offers substantial improvements in usability, significantly outperforming other models in overall classification metrics. Dispersed and inconsistent predictions can hinder doctors from quickly identifying seizure regions without examining large amounts of raw data.

Thank you very much for recognizing our work.

W1. We will integrate Figures 2 and 3 into a single illustration to improve clarity.

---

W2. We will explicitly clarify the meanings of p^ and p~ in the main text: p^ represents the model's independent prediction for a sample, while p~ denotes the context-aware prediction that incorporates the results from neighboring samples.

---

W3. The time complexity of Con4m can be divided into two main components. The primary computational cost is on the same order as vanilla Transformer. Assuming the number of consecutive input time segments is $L$, the hidden representation dimension is $D$, the number of classes in the classification task is $C$, and the local iteration count of the function fitting module is $I$.

• Con-Transformer: The time complexity of vanilla self-attention is $\mathcal{O}(LD^2+L^2D)$, and the time complexity of the Gaussian kernel branch is $\mathcal{O}(LD^2+L^2)$.
• Coherent class prediction: The time complexity of Neighbor Class Consistency Discrimination is $\mathcal{O}(LDC)$, and the time complexity of the Tanh function fitting is $\mathcal{O}(ICL)$.

Therefore, the computational cost and bottlenecks of Con4m are similar to vanilla Transformer.

---

W4&Q3. Thank you for the insightful question. Con4m is primarily designed for classification tasks, focusing on extracting structures and patterns within time segments. When dealing with datasets that have periodic patterns, we believe Con4m can still effectively capture these periodicities, provided that a complete cycle is included within a single classification instance. In this context, the Gaussian kernel acts as a one-dimensional Gaussian smoothing filter. We believe that classification tasks are typically less sensitive to periodic patterns compared to forecasting tasks. Furthermore, we have included a human activity recognition dataset HHAR, where sensor data naturally exhibit periodic changes as people walk or run. Con4m performs well on the six-class HHAR dataset, demonstrating its ability to capture periodic patterns.

---

W5. The execution time of Con4m on SEEG data is as follows: (1) Offline training (124,000 training samples + 62,000 validation samples): approximately 30s/epoch; (2) Batch inference (batch size of 64 with 62,000 test samples): approximately 14s. These times fully meet the practical requirements for inference.

---

Q1. We employ learnable absolute positional encoding. The absolute positional encoding is equivalent to the order index of time series segments, as the fundamental unit of our model is a time segment. This clarification will be addressed in the primary text of the paper.

---

Q2. Con4m is capable of handling multivariate time series data. As shown in Table 1, the fNIRS and Sleep datasets have 8 and 2 features, respectively. We employ a multi-channel CNN to jointly extract information from these features (see Figure 2), as the primary focus of this paper is not on multi-channel modeling. However, Con4m can be seamlessly integrated with existing multi-channel modeling approaches by using their output representations as inputs to Con4m.

---

Q4. Summation is the most direct and straightforward approach to achieving smoother representations, as concatenation and the Hadamard product do not explicitly provide this effect. Additionally, we enable the model to adaptively learn the necessary degree of smoothness, as sample representations in transition regions should not be continuous with those from different classes.