Con4m: Unleashing the Power of Consistency and Context in Classification for Blurred-Segmented Time Series

Dear Reviewer JdCs,

Thanks for your questions and we carefully address your concerns as follows:

Weakness 1:

Solving the state transition problem holds significant practical significance.

• Harmonizing inconsistent boundary labels is an important means of reducing annotation costs. As you mentioned, SleepEDF, being a widely used publicly available dataset, requires highly accurate labels. Achieving this level of precision necessitates the labor-intensive process of expert annotation, which is time-consuming. In the era of large-scale models, data and labels have become the core fuel for model performance. Unlike the field of NLP, medical annotations are precious and scarce. To reduce annotation costs and enable multiple institutions to collaborate on building the foundational model, harmonizing the annotations of different doctors is an essential step.
• Inconsistent boundary labels can also lead to a decrease in model performance. Samples near the boundaries also appear throughout the entire time series. For instance, in SEEG data, there exists waveform overlap between the seizure waves and the interictal waves. As demonstrated in the case study presented in the paper, the repeated occurrence of samples near the boundaries confuses the model during training. Learning label consistency with contextual information can reduce the interference caused by inconsistent labels, allowing the model to converge more consistently on training samples.
• Accurately identifying boundaries is necessary for certain application scenarios. For example, in the seizure detection task based on SEEG data, doctors need to determine the lesion areas by considering the temporal order of seizure waves generated by different channels (corresponding to different brain tissues). This is because the seizure waves will propagate from the lesion areas to surrounding brain tissues. The spread occurs rapidly, making it crucial to finely identify the boundaries of seizure onsets. Inconsistent labels prevent the model from providing precise and consistent predictions at the boundaries.

Weakness 2:

• The core insight of our theory lies in the selection of valuable contextual information. While people often focus solely on contextual information at the data level, our theoretical framework allows for the incorporation of contextual information at the label level into the model's design. In Section 3.2, lines 134-144, we mention that the theoretical variable $x_{\mathbb{A}\_t}$ can be replaced by $y_{\mathbb{A}\_t}$, benefiting from the analytical framework of information theory.
• Based on theoretical analysis, the design in Section 3.2 revolves around incorporating $y_{\mathbb{A}_t}$ into the model's predictions. Introducing the neighbor class consistency discrimination task is the most intuitive approach. We explicitly discriminate whether contextual samples belong to the same category through supervised learning. Based on the output weights of the discriminator, we directly incorporate the label information of contextual samples into the final predictions through a linear weighted sum. This design is straightforward, concise, and effective.

Weakness 3:

• As you mentioned, publicly available datasets like SleepEDF require highly accurate labels. The absence of BST datasets is due to previous works ignoring the issue of label inconsistency. It is precisely because of this absence that we propose a novel label disturbance method for time series data (lines 219-226). Our experiments (lines 266-272) demonstrate that the label disturbance method we propose is more essential and challenging for time series data. We have also included the codes in the attachment on how to disturb the publicly available datasets, which can be used by others.
• To validate the model's performance on more realistic datasets with inconsistent labels, we conducted experiments on SEEG. Apart from the data of the subjects in the test group, the data of other subjects were not manually harmonized (lines 239-242). These experimental results can more realistically reflect the impact of label inconsistency on the model. Due to strict ethical review, SEEG is highly sensitive medical data that cannot be publicly released at the moment. However, we are actively working towards collaborations with hospitals and subjects, hoping to share this data in a safer and restricted manner.

Question 1:

• In the expressions related to mutual information and probability, $x_t$ represents the random variable of the input raw data. We have adopted the \displaystyle \r command as suggested in the ICLR template.
• In Figure 1, $x \in \mathbb{R}^{L \times W \times d}=\{ x_1, \dots, x_L \}$, where $L$ represents the number of consecutive time segments, $W$ represents the window size (number of time points) in each segment, and $d$ represents the dimensionality of the input features.
• All of $x_1,\dots,x_L$ are realizations of $x_t$.

Question 2:

• Function fitting does introduce additional computational overhead. However, since we are fitting a Tanh function and the gradients do not backpropagate to the parameters of the encoder, such overhead is acceptable within the entire model.
• Let's assume the number of consecutive time segments in the input is $L$, the hidden representation dimension is $D$, and the number of classification categories is $C$. The Tanh function fitting involves $I$ internal iterative steps. The function fitting module fits the predictions for each category separately, where the prediction for each category in each time segment is a constant. Therefore, the time complexity of the function fitting module is $\mathcal{O}(ICL)$. If we only consider the final prediction linear layer in the encoder, which maps the hidden representation to the prediction logits, its complexity is $\mathcal{O}(DCL)$.
• Hence, we suggest controlling $I<D$ in practical usage to effectively manage the time complexity of the function fitting module. In our experiments, we set the default parameters to $I=100$ and $D=128$.

Question 3:

As depicted on the left side of Figure 2, both $\ell_1$ and $\ell_2$ jointly guide the learning process of the entire encoder.

Weakness 3 & Question 1-2:

Thank you for your valuable inquiry. We apologize for the oversight in this section. We have reorganized Theorem 1 as follows:

Theorem 1. Introducing a contextual sample set that maximizes the predictive ability of labels yields the maximum information gain.

Proof. Expanding $\mathbb{I}(y_t; x_{A_t}|x_t)$, we have:

$ \mathbb{I}(y_t; x_{A_t}|x_t) &= \sum_{x_t} p(x_t) \sum_{x_{A_t}} \sum_{y_t} p(y_t,x_{A_t}|x_t) \log{\frac{p(y_t,x_{A_t}|x_t)}{p(y_t|x_t) p(x_{A_t}|x_t)}} \\\\ &= \sum_{x_t}p(x_t) \sum_{x_{A_t}} \sum_{y_t} p(y_t|x_t,x_{A_t}) p(x_{A_t}|x_t) \log{\frac{p(y_t|x_t,x_{A_t})}{p(y_t|x_t)}} \\\\ &= \sum_{x_t} p(x_t) \sum_{x_{A_t}} p(x_{A_t}|x_t) D_{KL} (p(y_t|x_t,x_{A_t}) \Vert p(y_t|x_t)). $

Given a fixed classification sample $x_t$ and the inherent distribution $p(y_t|x_t)$ of the dataset, the KL divergence is a convex function that attains its minimum at $p(y_t|x_t,x_{A_{t}})=p(y_t|x_t)$. As $p(y_t|x_t,x_{A_{t}})$ approaches the boundary of the probability space, indicating a stronger predictive ability for $y_t$, the value of KL divergence increases. Due to the convexity of KL divergence, there exists a contextual sample set in the dataset that maximizes $D_{KL}(p(y_t|x_t,x_{A_{t}}) \\| p(y_t|x_t))$. We denote this sample set as $A_{t}^{\*}$ and the maximum KL divergence value as $D_{t}^{\*}$. Additionally, we note that $\sum_{x_{A_{t}}} p(x_{A_{t}}|x_t)=1$. Hence, we can obtain the upper bound for the information gain $\mathbb{I}(y_t;x_{A_{t}}|x_t) \le \sum_{x_t} p(x_t) \sum_{x_{A_{t}}} p(x_{A_{t}}|x_t) D_{t}^{\*} \le \sum_{x_t} p(x_t) D_{t}^{\*}$. To achieve this upper bound, the model needs to introduce a contextual sample set $A_{t}^{\*}$ for each sample that maximally enhances its label's predictive ability. Moreover, the model needs to reach an optimal selection strategy distribution $p(x_{A_{t}^{\*}}|x_t)=1, p(x_{A_{t}}|x_t)=0$ (for $A_t \neq A_{t}^{\*}$).

---

According to Theorem 1, the model needs to find the optimal contextual sample set that enhances the predictive ability of each sample's label. In this paper, we utilize learnable weights to allow the model to adaptively select potential contextual sample sets. After aggregating contextual sample information, we train the model using sample labels. Through explicit supervised learning, the model can enhance its predictive ability for samples end-to-end while optimizing the selected contextual sample set. On the other hand, benefiting from an information-theoretic perspective, $x_{A_{t}}$ in Theorem 1 not only includes the raw data of contextual samples but also incorporates their label information ($y_{A_{t}}$). Therefore, we can introduce contextual information at both the data level (Section 3.1) and the label level (Section 3.2) to enhance the model's predictive ability.

In the specific implementation, to balance computational costs, we limit the selection of contextual samples within a continuous and finite time range. At the data level, we introduce a learnable Gaussian kernel function to adaptively aggregate neighboring samples with a Gaussian prior, obtaining a smoother representation over time. At the label level, we explicitly train a neighbor class consistency discriminator. We aggregate the predicted results of contextual samples using the output probabilities of this discriminator as weights.

Dear Reviewer zaSc,

Thank you sincerely for your thorough and valuable comments. We have carefully addressed your concerns and confusion by making relevant clarifications and modifications in the following response. We will also update the corresponding parts in the original manuscript and submit a revised version shortly.

Weakness 1 & Question 3:

Apologies for any confusion caused. Allow us to provide the following clarifications:

• Blurred-Segmented Time Series (BST) refers to a type of time series data that exhibits distinct characteristics different from traditional Time Series Classification (TSC) data. In practical applications. BST data often arises, and it is characterized by two main aspects—Blurred Transitions and Prolonged State Durations. We establish this definition in lines 23-29 within the main context.

• We divide a time interval into several consecutive segments and mainly focus on classification for each individual segment. We appreciate your feedback and we've incorporated a formal problem definition at the beginning of Section 3 as follows:

  Definition 1. Given a time interval comprising of $T$ consecutive time points, denoted as $s=\\{ s_1,s_2,\dots,s_T \\}$, a sliding window length of $w$ and slide length $r$ is employed for segmentation. The time interval is partitioned into $L$ time segments, represented as $x=\\{ x_i=\\{ s_{(i-1) \times r + 1},\dots,s_{(i-1) \times r + w} \\} | i=1,\dots,L \\}$. The model is tasked with predicting labels for each time segment $x_i$.

• In our paper, we aim to employ 'segment' and 'state' when referring to specific scenarios, and in the context of machine learning, we prefer the terms 'sample' and 'label'. According to the problem definition above, our model focuses on predicting labels for each time segment. Therefore, 'sample' is equivalent to 'segment'. Also, the terms 'state' and 'label' are synonymous in our context, with each state having a corresponding label. Thank you for your recommendation and we will standardize and streamline the use of these terms in our paper.

• According to the problem definition, each time segment maintains a fixed length in the same dataset, as indicated in Table 1.

Weakness 2:

• The nature of BST data inherently lacks a clear definition of boundaries. As described in lines 30-37, due to the absence of standardized quantification criteria, different doctors exhibit experiential variations in annotating boundaries. Moreover, this variation itself is ambiguous and lacks a precise definition. Therefore, the removal of boundary samples is not feasible in practice.
• Employing a fixed length to remove boundary data is a crude and uncontrollable approach. Likewise, eliminating uncertain boundary samples through expert consensus would be prohibitively costly and unsustainable.
• Some scenarios require precise boundary labels. For instance, in the seizure detection task based on SEEG data, doctors need to consider the temporal sequence of seizure occurrences in different brain regions to devise subsequent surgical strategies. If the model does not account for boundary labels during the training phase, it will struggle to accurately identify the class of samples at the boundaries during the inference phase.

Weakness 4:

• As shown in the leftmost part of Figure 1, we implement the two-branch Con-Attention. $Q$, $K$ and $V$ vectors represent the query, key and value of the self-attention mechanism respectively. To distinguish between different computational branches, we use $s/S$ to represent the branch based on Gaussian prior, and $t/T$ to represent the branch based on vanilla self-attention. $\sigma$ is the learnable scale parameter of the Gaussian kernel function. $T^l$ and $S^l$ are the aggregation coefficients of the two branches in the $l$-th layer.
• For the neighbor class consistency discrimination part, our framework allows for the incorporation of contextual information at the label level into the model's design. According to Theorem 1, we aim to identify a set of contextual samples that maximizes the model's predictive capability at the label level. Since directly optimizing the aggregation at the label level is challenging, we adopt the approach of aggregating predictions of samples belonging to the same class to enhance the model's predictive capability. This approach is inspired by the observation that for graph neural networks based on the homophily assumption, aggregating neighbor information belonging to the same class can improve predictive performance [1,2]. Therefore, we explicitly train a discriminator to determine whether two samples belong to the same class. The model selects a contextual sample set based on the discriminator's output probabilities and weights their predictions based on the probability of being classified as the same class.

[1] McPherson, Miller, Lynn Smith-Lovin, and James M. Cook. "Birds of a feather: Homophily in social networks." Annual review of sociology 27.1 (2001): 415-444.

[2] Zhu, Jiong, et al. "Beyond homophily in graph neural networks: Current limitations and effective designs." Advances in neural information processing systems 33 (2020): 7793-7804.

Question 4:

Thank you for seeking clarification. And your insight aligns exactly with what we mean. We will incorporate this clarification into our paper.

Question 5:

Each group is of the same significance in our analysis. To obtain the cross-validation results, we randomly partition the subjects within the dataset into non-overlapping subsets for both training and testing. For fNIRS, we first divide the data into 4 groups by subjects and follow the 2 training - 1 validation - 1 testing (2-1-1) setting to conduct cross-validation experiments. Therefore, there are $C^2_4 \times C^1_2=12$ experiments in total. Similarly, we divide the Sleep data into 3 groups and follow the 1-1-1 experimental setting. Therefore, we carry out $C^1_3 \times C^1_2=6$ experiments. The full contents are also elaborated in lines 679-685 of Appendix G.

Question 6:

• The reference coordinates "2s 4s ..." in Section 4.5 serve as a guide, while the actual window length (segment length) of SEEG data is 1s with a slide length of 0.5s, as detailed in Table 1.

• During the labeling of time points from time segments, to reduce the error, we adopt a simple strategy for the time points at the intersection of two consecutive time segments. Specifically, the labels of the time points in the first half of the overlapping data align with the preceding time segment, while the latter half aligns with the subsequent time segment. For example, if the segment 0 consists of time points "1 2 3 4" with label "a", and the segment 1 comprises "3 4 5 6" with label "b". Then, time point 3 is assigned label "a", and time point 4 is assigned label "b".

  Label	a	a	a	b	b	b
  Segment 0	1	2	3	4
  Segment 1			3	4	5	6

Question 7:

Detailed implementation of the boundary disturbance is shown in lines 219-226 under the heading "Label disturbance". There exists a fundamental distinction between boundary disturbance and random noise. The boundary disturbance is employed to simulate scenarios where labels are inconsistent. The disturbance is introduced by disturbing the boundary points between different states. It is noted that only the results depicted in Figure 3(b) involve in random noise, which shows that boundary disturbance poses a more intrinsic and challenging problem compared to random noise for time series data. All of the other experiments are based on boundary disturbance.

Question 3:

The time complexity of Con4m can be divided into two main parts. The primary computational cost is on the same order as vanilla Transformer. Assuming the number of consecutive input time segments is $L$, the hidden representation dimension is $D$, the number of classes in the classification task is $C$, and the local iteration count of the function fitting module is $I$.

• Con-Transformer: The time complexity of vanilla self-attention is $\mathcal{O}(LD^2+L^2D)$, and the time complexity of the Gaussian kernel branch is $\mathcal{O}(LD^2+L^2)$.
• Coherent class prediction: The time complexity of Neighbor Class Consistency Discrimination is $\mathcal{O}(LDC)$, and the time complexity of the Tanh function fitting is $\mathcal{O}(ICL)$.

Therefore, the computational cost and bottlenecks of Con4m are similar to vanilla Transformer. Large sequence lengths (numbers of time segments) and large hidden representation dimensions both increase the model's computational complexity. In contrast to the natural language processing field, where sequences are typically discrete, time series data consists of continuous numerical values. This allows for controlling the sequence length $L$ by adjusting the length of each time segment (patch). Thus, Con4m can be scaled to longer time series by appropriately tuning the patch length, the number of CNN layers, and the size of the convolutional kernels.

Dear Reviewer 5NtK,

Thank you for acknowledging our work. In the following response, we provide some clarifications and explanations:

Weakness 1 & Question 1:

• Specifically, mutual information measures the intrinsic properties of a dataset. For example, $\mathbb{I}(y_t;x_t)$ quantifies the correlation between sample and label distributions in the dataset from a probabilistic perspective. $\mathbb{I}(y_t;x_t,x_{A_{t}})$, on the other hand, describes the correlation between sample and label distributions when contextual samples are introduced. The theoretical framework is built upon the assumption that the model can perfectly capture these correlations (line 79). In other words, we seek the theoretical upper bound of the model's performance.
• A larger value of mutual information indicates stronger correlation between variables. For instance, in the inequality $\mathbb{I}(y_t;x_t,x_{A_{t}}) \ge \mathbb{I}(y_t;x_t)$, introducing contextual samples $x_{A_{t}}$ enhances the correlation between the target sample $x_t$ and its label. In classification tasks, a higher correlation between input samples and labels implies that input samples are more easily distinguishable by their labels. In an ideal scenario, the model also possesses a higher upper bound on its performance to discriminate between samples.

Weakness 2 & Question 2:

Based on the previous answer, the increase in $I(y_t;x_{A_t}|x_t)$ determines the extent to which the upper bound of the model's performance improves. The purpose of Theorem 1 is to elucidate the specific contextual sample set that can maximize the correlation between the target sample and its label (i.e., $I(y_t;x_{A_t}|x_t)$). The following explanation is based on our latest version of Theorem 1 (Please see https://openreview.net/forum?id=EvBx5whpzJ&noteId=ZprzVTtrvl). In our proof, predictive capability is equivalent to the uncertainty of the conditional probability distribution of the label given the input sample. As the distribution $p(y_t|x_t,x_{A_{t}})$ approaches the boundary of the probability space, meaning the predicted probability for a certain class approaches 1 (and the probabilities for other classes approach 0), the uncertainty decreases. According to the proof, lower uncertainty corresponds to larger values of KL divergence. The upper bound of $I(y_t;x_{A_t}|x_t)$ is achieved when the KL divergence is maximized, indicating the lowest uncertainty. Therefore, we conclude that introducing contextual samples that maximally enhance predictive capability for the label results in the maximum information gain.

Weakness 3 & Question 4:

• According to Theorem 1, the model needs to find the optimal contextual sample set that enhances the predictive ability of each sample's label. In this paper, we utilize learnable weights to allow the model to adaptively select potential contextual sample sets. After aggregating contextual sample information, we train the model using sample labels. Through explicit supervised learning, the model can enhance its predictive ability for samples end-to-end while optimizing the selected contextual sample set.
• The Gaussian kernel function serves as our prior knowledge for selecting the contextual sample set. BST data exhibits temporal persistence for each state (line 38). By paying closer attention to and aggregating neighboring samples, the model can acquire temporally smoother representations of time segments. Smoother representations lead to smoother predictive probabilities. This benefits not only the prediction of consecutive time segments belonging to the same category with the same label but also aligns with the gradual nature of class transitions (lines 110-114). According to the conclusions of the latest version of Theorem 1, the use of the Gaussian kernel function allows for a more targeted selection of the contextual sample set, thereby enhancing the model's predictive capability. We will revise these statements in the new version accordingly.

Dear Reviewer 2FMY,

Thank you for your comments and we address your concerns as follows:

Weakness 1:

Thank you for your valuable suggestion.

• With domain knowledge, given the prior visual window size of manual annotation, we recommend selecting a slightly larger window size as the time interval length for two reasons:

  • Typically, the annotation window does not contain data spanning more than two classes.
  • Opting for a slightly larger window allows the model to leverage more contexts.

  For instance, in the case of SEEG data, the doctors tend to use a 15-seconds visual window to identify the seizure waves. And based on empirical evidence, the duration of seizures is generally not shorter than 15 seconds. Hence, we chose a 16-second time interval to try to avoid the occurrence of multiple label transitions.

• We are currently conducting experiments with time intervals scaled at 0.5, 1.5, and 2 times the initially selected length. Upon completion of the experiments, we will promptly provide the results and analysis.

  To ensure the comparison to existing experimental parameters as fair as possible:

  • We maintain identical experiment groups.

  • We sample the same number of time intervals based on three different time interval lengths respectively. We proportionally scale both the window length and slide length, ensuring consistency in the number of time segments across all experiments.

  • The hyperparameters of the main model and baselines remain consistent.

Weakness 2:

In Figure 3(b), we compare the performance of Con4m under the settings of random disturbance and boundary disturbance. The fNIRS and Sleep data are publicly available with accurate labels, allowing us to experiment with different disturbance strategies. However, for the SEEG data, we only have precise annotations for the test subjects, while the training data includes inconsistent annotations from different doctors (lines 238-242). Therefore, we cannot apply the random disturbance strategy to the SEEG training data. In the ablation experiments, we opted for the experimental settings with more significant boundary disturbance to clearly validate the effectiveness of each component of the model. Additionally, due to the constraints of the cognitive experimental setup, the transitions between different classes in the fNIRS data are more explicit, making them less susceptible to boundary disturbance. Hence, we selected Sleep-20%/40% and SEEG as the experimental groups for the ablation experiments.

Weakness 3:

We have conducted relevant experimental analyses. In the ablation experiments, - Con-T refers to replacing the Con-Transformer module with vanilla Transformer, thereby removing the prior information at the data level. - Coh-P indicates the simultaneous removal of the Neighbor Class Consistency Discrimination module and the function fitting module, meaning the elimination of the contextual information at the label level. - Fit represents the removal of only the function fitting module, implying the exclusion of prior trend information at the label level. Conversely, + Con-T and + Coh-P denote the retention of data-level and label-level contextual information, respectively.

We conduct the experiment about time interval length by selecting well-performing baselines and comparing them with Con4m. The results of the $F_1$ scores for each model are presented below:

Based on the experimental results, the following observations can be made:

1. It is not recommended to choose time intervals shorter than the manually annotated visualization window. For both fNIRS and Sleep data, all models perform the worst when the time interval was set at 0.5 times the window size, with a significant decrease in performance. This may be due to insufficient information captured by the smaller window size.
2. Within the range from the visualization annotation window to the average durations of the shortest class, various time intervals can be chosen. For both fNIRS and Sleep data, models show an improvement in performance as the time interval increases. This improvement is particularly significant in the Sleep data, which may be attributed to the labeling scheme assigning a label to each 30-second time window.
3. It is recommended to consider both the characteristics of the data itself and the requirements of practical applications. For the higher sampling rate and complexity of SEEG data, MiniRocket demonstrates more stable performance. However, Sel-CL and SREA show a decrease in performance as the window size increases. This can be attributed to the non-stationary nature of SEEG, which involves more diverse waveform variations with larger window sizes. Con4m, by considering contextual sample information, maintains relatively stable performance and even achieves higher performance. Additionally, to ensure comparability, we do not adjust the model parameters. When dealing with larger time intervals, it is necessary to consider increasing the model's parameter size appropriately.
4. In summary, Con4m consistently outperforms other models and exhibits a consistent performance improvement trend across all data. This indicates the rationality and superiority of Con4m in addressing the classification problem in BST data.