Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approach

We greatly appreciate your supportive remarks and constructive feedback. We are delighted to see that you found the studied problem is important and methods are novel, and that the paper is presented and explained very clearly. We provide detailed responses to your questions below. We hope that our responses sufficiently address your concerns.

1. Identification may not be accurate at early stages

We prevent this problem by setting warm-up epochs. During the initial warm-up epochs, the model performs regular training and there is no downweighting for samples, ensuring stable model training. The downweighting is only activated after the initial warm-up epochs, thus minimizing the risk of error propagation due to imperfect identification at early stages.

2. How to avoid potential misidentification of hard samples?

Avoiding potential misidentification of hard positive pairs as faulty pairs is an important point and we indeed take this challenge into account when designing DBPM. As discussed in Sec 4.2, hard positive pairs that are generated by strong augmentation while retaining their original identities can indeed benefit contrastive learning.

Unlike faulty positive pairs, hard positive pairs are learnable, meaning that the model can still extract representative information shared between them. Therefore, hard positive pairs usually have relatively large losses at certain training iterations rather than suffering high losses over the entire training process. Pairs that constantly exhibit large contrastive losses during the entire training are more likely to be faulty positive pairs that are difficult for the model to learn.

DBPM identifies bad positive pairs based on their historical training behaviors in a dynamic manner, which allows more reliable identification of potential bad positive pairs and reduces the chance of misidentification.

3. Rationale behind design of Equation 3

The memory module is a look-up table that records the loss value of each positive pair as soon as it is being trained at each epoch. When computing the global statistic ($\mathcal{M}_e$) for identifying potential bad positive pairs at each epoch, it is essential to have the loss value of all samples to accurately estimate the loss distribution that describes historical training behaviors.

Calculating the mean and standard deviation of the loss before the current epoch ensures that we are using a complete set of data. This helps DBPM be more reliable in identifying potential bad positive pairs based on their historical training behavior. During the ongoing epoch, some samples may not have been trained yet, so their losses would not be available. If we compute the mean loss using incomplete data from the current epoch, it could introduce bias or inaccuracies into our global statistics.

4. False negative pairs

This is quite an insightful suggestion. We believe that DBPM can also be extended to find false negative samples by modifying the memory module to record loss/similarity scores between negative samples. In this paper, we focus on bad positive pairs and not yet the false negative pair problem.

Dear Reviewer DxUx,

Thank you for your review and constructive comments on our paper. We have provided corresponding responses, which we believe have covered your concerns. As the discussion period is close to its end, we would appreciate your feedback on whether or not our responses sufficiently resolve your concerns. Please let us know if you still have any concerns that we can address.

Best regards,

Authors

Thank you very much for your encouraging comments on our paper, especially for acknowledging the significance of the problem we studied and the value of our proposed methods. We believe that our work will inspire more future time series contrastive learning studies to tackle this practical yet challenging issue. The following are our responses to your questions. We hope that our responses not only clarify our work but also provide insight into future explorations, and lead to a favorable increase of the score.

1. Could more principled statistical methods be explored to automatically determine optimal threshold?

Thank you for your insightful comment. We totally agree that the principled statistical method in determining thresholds is important and offer a promising avenue for further investigations. At the current stage, DBPM is a simple and effective solution that aligns with our observations. More importantly, it provides a clear analysis of the newly recognized bad positive pair problem. By choosing proper thresholds, we have achieved meaningful improvements in the performance of contrastive learning models.

Therefore, we plan to study more principled statistical methods in our future study. Besides statistical methods, another direction for future exploration is more sophisticated decision-making models (Reinforcement Learning) for automatically determining thresholds.

We believe our work provides a strong baseline in the newly identified bad positive pair problem, and we anticipate that it will inspire more further research and explorations.

2. Does this approach generally work for other losses (e.g., SupCon)?

We would like to clarify that our experiments included evaluations on SimSiam, which employs the cosine similarity of positive pairs as its optimization objective, rather than InfoNCE loss. The results, as demonstrated in Tables 1 and 2, consistently indicate that our proposed DBPM enhances SimSiam's performance across various tasks.

In terms of supervised contrastive learning (SupCon), we did not include it in this study since the focus of the research is on self-supervised contrastive learning, where supervisory signals are derived from only contrasting views. The supervisory signals in supervised contrastive loss are derived from task-related labels, which may result in different observations and solutions than in self-supervised learning.

3. Did the authors conduct ablation experiments to distinguish individual impacts?

We thank the reviewer for the insightful comment. In Appendix A.8 Tables 7, 8, and 9, we provide an ablation analysis that includes the assessment of individual effects of specific types of bad positive pairs. The terms "w/o $t_{np}$" and "w/o $t_{fp}$" refer to models that exclude the handling of noisy positive pairs and faulty positive pairs, respectively.

It is possible that specific datasets are predominantly influenced by one over the other. For instance, in Table 7, the performance appears to be marginally affected by the choice of $\beta_{fp} \in [2,3]$ in scenarios where we do not address noisy positive pairs (w/o $t_{np}$). Conversely, when faulty positive pairs are not managed (w/o $t_{fp}$), a smaller $\beta_{np}$ is associated with improved performance, potentially indicating a higher prevalence of noise within the dataset.

Our DBPM can be used to determine such unbalanced influences by selectively isolating one type of bad positive pair, thereby enabling an insightful analysis of their respective influences on the dataset.

4. Is it possible to isolate certain features and then forecast based on them?

Isolating certain features and forecasts if a dataset is predominantly influenced by a specific type of bad positive pair is an insightful avenue for exploration.

Addressing this problem requires recognizing some universal features that are exclusive to noisy positive pairs and faulty positive pairs, respectively. However, contrastive models can learn different features in different datasets, making it somewhat challenging to determine the universal features. For example, “certain features” learned from ECG data may not be usable for identifying bad positive pairs in financial time series data.

In addition, identification of bad positive pairs is still a prerequisite for understanding their characteristic features. We can only analyze their features in detail when these pairs are identified.

While we currently lack a definitive solution to this challenge, your question has indeed opened up an interesting avenue for future exploration. We appreciate your insightful question and look forward to delving into these possibilities in our subsequent works.

Dear Reviewer Cugc,

Thank you for your review and valuable feedback on our paper. We have provided corresponding responses, which we believe have covered your concerns. As the discussion period is close to its end, we would appreciate your feedback on whether or not our responses sufficiently address your concerns. Please let us know if you still have any concerns that we can address.

Best regards,

Authors

Thank you very much for your constructive comments! We are delighted to see that you found the studied problem is important and underexplored, and that the paper is presented and articulated very clearly. We provide detailed responses to your concerns as follows. We hope the additional information will further clarify our work and lead to a favorable increase of the score.

1. Further clarify the uniqueness of the problem

We thank the reviewer for your constructive suggestions. Here we would like to further clarify the uniqueness of the bad positive pair problem.

In image contrastive learning, the prior work [1] only identified the faulty view problem, as it is rare to see images containing overwhelming noise. Our findings reveal the presence of not only faulty positive pairs but also noisy positive pairs in time series contrastive learning. This can be attributed to the sensitive nature of time series data. In real-world applications, noise, data collection errors, and various distortions can lead to bad positive pairs. For example, in collecting electroencephalography (EEG) data, noise can occur due to ocular, muscular and cardiac activities [2].

Our paper presents both theoretical analysis and empirical evidence demonstrating that these types of pairs can negatively affect the representations learned from time series contrastive learning.

The particular challenge of addressing two different types of bad positive pairs is unique to time series data and distinguishes them from image data. Our results reveal that methods directly adapted from image data, such as RINCE[1], fail to effectively tackle the issues these pairs present in the time series context. This underscores the necessity of understanding the unique characteristics of these pairs within time series data, highlighting a distinct challenge in this domain.

Following your suggestions, we will improve the clarity on the uniqueness of this problem by involving the above discussion in our revised manuscript.

[1] Chuang, Ching-Yao, et al. "Robust contrastive learning against noisy views." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Gandhi, Sunil, et al. "Denoising time series data using asymmetric generative adversarial networks." Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part III 22. Springer International Publishing, 2018.

2. Noisy alignment assumption might be overly stringent

We would like to clarify that, as we described in Sec 1 (Page 2), we assume that noisy pairs and faulty pairs are extreme cases that should represent a small portion of a normal dataset. However, the existence of such extreme cases can impair the quality of representations learned from the model. Our DBPM is designed to deal with extreme cases that adversely affect model performance in real-world datasets.

3. Methodology does not seem to offer substantial technical innovation

We would like to emphasize that our DBPM is the first design in addressing a novel problem of bad positive pairs in self-supervised time series contrastive learning. Though the idea of finding and penalizing training instances of bad quality may have been studied in other settings such as supervised learning, our DBPM differs from them in two aspects.

• We focused on a novel bad positive pair problem. Addressing this problem requires understanding the training behaviors of samples in the context of self-supervised time series contrastive learning, a challenge previously unexplored. For example, a small contrastive loss does not always indicate that the model has learned useful patterns, which is quite different from the situation in supervised learning.

• The design of DBPM aligns well with our empirical observations and innovatively incorporates a dynamic method for the identification of bad positive pairs, enhancing the algorithm's reliability. The transformation module offers flexibility in weight estimation, allowing it to accommodate various scenarios effectively. In addition, DBPM is designed as a lightweight plug-in that does not require any trainable parameters, thereby simplifying the training procedure and rendering it adaptable to a broad spectrum of methods.

In terms of performance, we would like to argue that as a lightweight plug-in, DBPM almost always brings benefits to existing methods, especially in scenarios where bad positive pairs are more pronounced. For example, in Figure 4, we show that DBPM markedly improved prediction performance in certain time series categories.

Our study and proposed method provide a strong baseline in the newly identified bad positive pair problem, and we anticipate that it will inspire more further research and explorations.

Dear Reviewer iore,

Thank you for your effort in reviewing our paper and offering valuable comments. We have provided corresponding responses, which we believe have covered your concerns. As the discussion period is close to its end, we would appreciate your feedback on whether or not our responses adequately address your concerns. Please let us know if you still have any concerns that we can resolve.

Best regards,

Authors

We greatly appreciate your recognition of the innovation and contributions in our work. We are glad that you found the paper novel, well-structured and clearly articulated. We address your concerns point by point as follows. We hope the additional information will further enhance our paper and lead to a favorable increase of the score.

1. Concerns on ablation study

We thank the reviewer for the valuable suggestions. Following your suggestions, we conducted an additional ablation study on PTB-XL dataset in the following way:

• w/o M: we directly identify bad positive pairs at each epoch without using the historical information.

• w/o T: we remove the transformation module and set the weight of identified bad pairs to zero.

The results are shown below:

We can see from the results that both the memory module M and transformation module T are important for ensuring consistent improvements of model performance.

M plays a crucial role in reliably identifying potential bad positive pairs. For instance, if we do not use historical information in the Rhythm classification task, performance can be adversely affected. While T ensures a smooth weight reduction which contributes to a more stable training process. This can be seen from the suboptimal model performance when removing T.

We hope the results provide a more comprehensive understanding of the unique effects of each component on overall performance and address your concerns.

Following your suggestion, we will add these new results to our revised manuscript.

2. Detailed exploration of algorithmic complexity

Thank you for your valuable suggestions. We conducted a detailed analysis on algorithmic complexity. Suppose we have $N$ instance:

The parts of the algorithm that cause extra runtime:

a. Calculation of the mean and standard deviation:

• For each pair, its mean $m_{i,e}$ is updated based on the previous means, which is an $O(1)$ operation per instance since it is based on pre-computed values.

• The overall mean $\mu_e$ across all pairs at each epoch requires summing up $N$ mean values and dividing by $N$, which is an $O(N)$ operation.

• Once the overall mean $\mu_e$ is calculated, the $\sigma_e$ requires a pass over the $N$ mean values to calculate the sum of the squared difference from the $\mu_e$, which is an $O(N)$ operation. Then the calculation of actual $\sigma_e$ involves dividing by $N$ and taking the square root, both are $O(1)$ operations.

• Therefore, calculating the mean and standard deviation will result in a runtime complexity of $O(N)$.

b. Condition check:

• Each comparison is an $O(1)$ operation, so for all $N$ instances, the condition check is $O(N)$ per epoch.

c. Calculation of weights:

• Calculating the weights for each instance is a constant-time operation $O(1)$. Therefore, for all $N$ instances, the calculation of weights is $O(N)$.

In summary, the runtime complexity of DBPM is O(N).

The parts of the algorithm that cause extra memory:

In our algorithm, memory module M serves as a look-up table for storing the training loss for each sample along the training epochs, therefore original M is of the size $N \times E$. Here we show the M can be optimized to size $N \times 1$ in practical implementation:

For sample $i$ at $e$-th epoch, its training behavior is calculated by averaging the training losses before $e$-th epoch:

$$m_{(i,e)} = \frac{1}{e-1} \cdot \sum_{e'=1}^{e-1} L_{(i,e')}$$

The above equation can be rewritten as:

$$m_{(i,e)} = \\{[ \frac{1}{e-2} \cdot \sum_{e'=1}^{e-2} L_{(i,e')}] \cdot (e-2) + L_{(i,e)}\\} \cdot \frac{1}{e-1}$$ $$m_{(i,e)} = [m_{(i,e-1)} \cdot (e-2) + L_{(i,e)}] \cdot \frac{1}{e-1}$$

This means that, for calculating the mean training loss of sample $i$ at $e$-th epoch, we only need to store the mean training loss of sample $i$ at $(e-1)$-th epoch. Therefore, the memory size can be reduced to size $N \times 1$ in the practical implementation. Assuming a 16-bit floating-point number for each loss, the total extra memory required would be $2×N$ bytes.

We will add these analysis to our revised manuscript.

3. Concerns on parameter sensitivity analysis

The sensitivity analysis can be found in Appendix A.8 Tables 7, 8, and 9. The use of a manual threshold in our current design is a simple and effective solution that aligns with our observations. By choosing proper thresholds, we have achieved meaningful improvements in the performance of the contrastive learning models.

We agree that relying on a manual threshold might not be the optimal solution for all datasets or models. Therefore, as we mentioned in Sec.5, we plan to explore more sophisticated techniques for adaptive thresholding in our future work. This could involve learning the threshold directly from the data or using decision-making methods (e.g., Reinforcement Learning) to adjust the threshold dynamically during training.

We believe our work provides a strong baseline in the newly identified bad positive pair problem, and we anticipate that it will inspire more further research and explorations.

4. What are other potential applications?

To provide a clearer analysis of the newly recognized bad positive pair problem, this study mainly focused on time series classification tasks. As we mentioned in Sec.5, we will apply DBPM to more applications such as forecasting and anomaly detection in our future works.