REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning

Thank you for your kind words and discussion points. We have revised our text to capture our discussion on the different performance across datasets and the presence of false negatives. We have addressed each of your specific points below (with W = weakness and Q = question).

"Figure 4 shows that the diagonal pattern is worse on the PPG and ECG data compared with that on the HAR data. Some explanations need to be provided here" (W1):

• This is a good point. The different classes within HAR are arguably more distinctive than the classes for ECG and PPG, which can be seen in each dataset's visualization in Fig. 10-12, and we added this detail in the Fig. 5 [formerly Fig. 4] caption. Appendix A.3 further details each of the distinctive domains that each dataset is capturing.

False Negatives (Q2):

• The idea that false negatives potentially being among the candidates is a great discussion point, and it is a fundamental issue within contrastive learning more broadly, rather than being specific to our REBAR method. For example, in augmentation-based approaches like SimCLR [1], they sample negative items from the batch, and the batch may contain items from the same downstream classification category as the anchor. However, they are able to still achieve strong performance. Similarly, our approach is still able to achieve the best representation learning performance across our baselines. In regards to the proposed thresholding method, we believe that this method would be ineffective. A threshold should not be designed to be a hyperparameter as it would be dependent on how each specific anchor's specific motifs compare to other general instances within the same class. However, learning a threshold as a function of the anchor would be non-obvious to do due to the lack of class label ground truth during contrastive learning. Addressing false negatives is an ongoing research area, and we are exploring other strategies as a part of future work. We add this discussion into the Appendix A.6.

Typo (Q3):

• Thank you for pointing this out, and we have fixed it in the text.

Citations:

[1] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

Thank you for the discussion points and suggestions. Based upon our discussion, we have significantly revised our paper to clarify the scope of our paper to be focused on time-series that can be specifically defined as a series of class-labeled subsequences, adding a new t-SNE visualization in Fig. 6 and more analysis to further expand experimental results, as well as adding a new Tbl. 3 that highlights an ablation study and hyperparameter sensitivity analysis of the REBAR model. We have addressed each of your specific points below (with W = weakness and Q = question).

Formatting (W1):

• Thank you. We have added numbers to all of our equations, clarified the notation, and added definitions throughout the Sec. 3 Methods and the rest of the paper. Our Sec. 3 has been completely revised and reorganized to have a better flow to improve our REBAR method's understandability.

Experiment Scale (W2):

• Thank you for this question. We have revised the paper to clarify the scope of our contribution: Our method targets longer (extended) time series that are themselves composed of class-labeled subsequences, such a single recording of HAR data composed of the different types of physical activities, as illustrated in the new Fig. 1. There are relatively few datasets that support this task. For example, TNC only included HAR, ECG, and a small simulated dataset. We note that our scope is well-aligned with many mobile health, embedded sensing, and IoT applications which involve continuous and passive data collection and produce very long recordings with few to no labels. This is an area where the datasets that are available for research purposes have lagged behind the driving applications. We note that datasets such as UEA and UCR contain pre-segmented class-labeled time series that are not suitable for evaluating our approach (our datasets have time-series that are 8,325,000 / 334,080 / 15,000 time points long, compared to the median length of 621 and 301 for UEA and UCR respectively). Please see revisions to Sec. 1. Introduction and Sec. 6 Conclusions.
  • Using this type of long time-series data in our problem also allows us to benchmark sampling-based contrastive learning methods, such as TNC and our own REBAR method. These sampling-based methods exploit the time-series nature of our task. Because each subsequence is part of a longer time-series that is changing labels over time, we can sample positive and negative examples directly from the longer time-series, rather than attempting to create positive examples from augmentations.
• With this being said, we have expanded upon our results section to increase our experimental scale to add a new Fully Supervised baseline in Tbl. 1, a new Fig. 6 t-SNE visualization, as well as a visualization of the various positive/negative pairs identified by REBAR for a given time-series in the new Fig. 7.
  • The linear probe trained on our REBAR representation consistently achieved the strongest results, even beating the fully supervised model in PPG and HAR, achieving the same accuracies, but higher AUROC and AUPRC. This demonstrates our REBAR's methods strength in learning a representation that is better at handling class imbalance than the fully-supervised model.
  • Our t-SNE visualizations in Fig. 6 further validate our findings regarding our REBAR method's clustering ability. In HAR, all methods except for REBAR group walk, walk up, and walk down together. In PPG, all of the methods have poor clustering, but REBAR seems to perform the best with clearer separation between classes and has fewer discontiguous regions for the same class. In ECG, REBAR continues to have the best intra-class clustering.
  • Our positive/negative pair visualization in Fig. 7 visualizes the positive pairing from the list of candidates for a given anchor that was identified by our REBAR method, and an additional gallery of 15 more of this visualization is included in Appendix A.5. We see that even when there is no exact match of the anchor within the candidates, REBAR's motif-comparison retrieval and reconstruction is able to identify a positive example from the candidates that shares the same class as the anchor.

Thank you for your feedback to our response. We would like to emphasize that with our clarified scope on “time-series that are defined as a series of class labeled subsequences”, there are very few datasets that support this task. TNC only included HAR, ECG, as well a small simulated dataset designed specifically to assess their model's stationarity test feature [1]. Our work adds the PPG dataset for stress classification, to introduce an additional benchmark dataset for this type of time-series contrastive learning task. If you have any suggestions on additional datasets to include, we would be happy to include them in the next revision, but we believe that the three datasets we have included are very distinctive and allow us to assess the strength of each of the contrastive learning methods.

HAR, PPG, and ECG each demonstrate distinctively different characteristics. measuring acceleration, electrical activity of the heart, and blood volume change. The differences between the classes for a given dataset are all captured very differently. The activity labeled subsequences in HAR are primarily distinguished by amplitude, as seen in Fig. 10. Using PPG to identify stress is an ongoing research topic, but recent work has found that stress labeled subsequences in PPG are captured by wave shape [2]. The atrial fibrillation labeled subsequences in ECG are distinguished by irregular peak frequency. Therefore, although we only contain three datasets, their distinctive nature demonstrates the strength of our method.

[1] Tonekaboni, Sana, Danny Eytan, and Anna Goldenberg. "Unsupervised representation learning for time series with temporal neighborhood coding." arXiv preprint arXiv:2106.00750 (2021).

[2] Celka, Patrick, et al. "Influence of mental stress on the pulse wave features of photoplethysmograms." Healthcare technology letters 7.1 (2020): 7-12.

(edited to add TNC citation)

Thank you for your great discussion points. Based your points, we have added 3 new figures (conv cross-attn in Fig. 3 to better explain our approach and masking procedure; t-SNE in Fig. 6 and pos pair examples in Fig. 7 to better explain our results) and a new ablation study+hyperparameter table into the main text, as well as substantially revising our paper to add more explanation of our method with a time-series motif framing, further model analysis and explanation to the results, and clarifying the scope of our paper to be focused on time-series that can be defined as series of class-labeled subsequences. We have addressed each of your specific points below (with W = weakness and Q = question).

Difference in Masking (W1, Q1):

• We replace the word "evaluation" with "application" here and in our text, so that it is more clear we are using the intermittent mask explicitly during application of REBAR in contrastive learning to identify positive/negative pairs. Our REBAR cross-attention model compares the motifs within a specific receptive field around a masked time-point in the query sequence with the motifs in the key sequence. Then, it retrieves the best matching motif from the key to be used for reconstruction of that specific masked time-point in the query, and this idea is illustrated in the new Fig. 1 and Fig. 3.
• At training time, a contiguous mask is used so that the model learns to compare specific, potentially class-discriminative, motifs in the query with specific, potentially class-discriminative, motifs in the key, rather than comparing minor, transient, non-unique motifs. See Fig. 4a) and b) for attention weight visualizations.
• At application time, a contiguous mask could be used, however, in doing so, only the motifs near the contiguously masked out region would be compared to the key. This is because when reconstructing a given masked time-point and another point contiguous to it, the receptive fields to be used to identify motifs to be compared to the key are heavily overlapping.
  • An intermittent mask allows for many different motifs in the query, each of them captured in a receptive field around the many masked time-points dispersed throughout the signal, to be compared to the key during reconstruction. This allows for a higher coverage of the query in conducting such motif-similarity comparisons with the key. Therefore, during application, when we are testing each candidate as the key for a given anchor as the query, we would be able to identify the candidate which is most similar to the anchor as a whole.
• We empirically justify this intuition. Table 4 in Appendix A.1.2 shows this combination of using a contiguous mask for training and an intermittent mask for application achieves the best performance in our validation experiment. Additionally, Figure 4b) demonstrates that the cross-attention model trained with a contiguous mask still maintains a sparse attention when evaluated on a transient mask. Therefore, the model is still reconstructing the query based on specific retrieved motifs from the key at each of the transiently masked out time-points.

"The paper does not specify the exact number of samples within the datasets, which seem to be on the smaller side … UCR … UEA" (Q2):

• Thank you for the suggestion. We have added the exact number of samples within the datasets into our paper in Appendix A.2. Once we normalize our datasets by segmenting them into the subsequences that are used to train each of our self-supervised learning models, we have a total number of 76,590 / 6,914 / 1,305 sequences for ECG, HAR, and PPG, respectively. This is much larger than the datasets found in UCR and UEA: the median total dataset size in UCR is 687 and in UEA it is 621. In fact, if we look specifically at the 29 dataset subset of UEA that TS2Vec primarily presents in their paper, then this UEA subset only has a median dataset size of 412.5. Note that:
  • 90 out of the 128 datasets in UCR and 123 out of the 183 datasets in UEA have 1,000 or less sequences.
  • 8 out of the 128 total datasets in UCR and 14 out of the 183 total datasets in UEA have 100 or less sequences.

Thank you for your great suggestions. We have added 5 new figures (intuition in Fig. 1; conv cross-attn in Fig. 3; t-SNE in Fig. 6; pos pair in Fig. 7) and a new ablation study+hyperparameter table into the main text, as well as substantially revising our paper to capture your comments. We have addressed each of your points below (with W = weakness and Q = question).

Lack of Ablation Studies + Hyperparameter Analysis (W1, W3, Q1):

• Thank you for this suggestion: we have added a new Table 3, which includes the results of our ablation study and a hyperparameter sensitivity analysis for our REBAR model.
• Each of the model components and modifications contributes to REBAR's strong contrastive learning performance, especially the dilated convolutions (e.g. .061 drop in accuracy and a .263 drop in NMI). The exclusion of our reversible instance norm leads to a .008 drop in accuracy and a .127 drop in NMI, and the addition of an explicit positional embedding leads to a .006 drop in accuracy and .120 drop in NMI. As noted in Section 3.1, we intentionally keep the design of REBAR attention simple to emphasize the motif comparison within the cross-attention mechanism.
• The REBAR model is fairly robust to hyperparameter tuning. When we modify the size of contiguous masks, the size of the dilated conv's receptive field, or the number of epochs to train the REBAR cross-attention, the downstream linear probe and clusterability performance remains consistent.
  • Increasing the mask size from 15 to 25 increases accuracy by .006 and increases NMI by .017. Decreasing the mask size from 15 to 5 decreases accuracy by .004 and decreases NMI by .017.
  • Increasing the receptive field from 43 to 55 decreases accuracy by .016 and increases NMI by .037. Decreasing the receptive field from 43 to 31 decreases accuracy by .003 and increases NMI by .007
  • Decreasing trained REBAR cross-attention epochs from 300 to 200 increases accuracy by .006 and increases NMI by .070. Decreasing trained REBAR cross-attention epochs from 300 to 100 increases accuracy by .006 and decreases NMI by .013.
  • We note that these results should not imply that the REBAR model should not be trained. Tbl. 5 in the Appendix shows that if REBAR is only trained for 1 epoch, then accuracy drops by 0.010 and NMI drops by 0.251.

More Visualizations of Learned Embedding + Positive/Negative pairs. (W2, Q2)

• Thank you for this idea. In the Sec. 5 Results section, we have added t-SNE visualizations of each model's encoding in the new Fig. 6, and a visualization of the various positive/negative pairs identified by REBAR for a given time-series in new Fig. 7, as well as an additional gallery of 15 more of this positive/negative pair visualization within Appendix A.5.
• Our t-SNE visualizations in Fig. 6 further validate our findings regarding REBAR's ability to learn embeddings with superior class-discriminative properties. In HAR, all methods, except for REBAR, group "walk", "walk up", and "walk down" together. In PPG, all of the methods have poor clustering, but REBAR seems to perform the best with clearer separation between classes and fewer non-contiguous regions within the same class. In ECG, REBAR exhibits the best intra-class clustering.
• In Fig. 7 positive/negative pair visualization, we illustrate the positive example identified by our REBAR method from the list of candidates, for a given anchor in each class. Even when there is no exact match for the anchor within the candidates, REBAR is still able to select an example that has the same class as the anchor.

"Without comparison with baselines, Figure 4 [confusion matrices] …" (W4) :

• We would like to clarify that the confusion matrices (now in Fig. 5) are designed to assess how REBAR's pseudo-distance function between two subsequences predicts mutual class membership between the two. This validates REBAR's effectiveness in learning a discriminative distance function, before contrastive learning takes place. Since the other baselines from the literature do not construct a pseudo-distance function, they can't be included in this analysis. We have clarified the text to make this more clear.

Notation (W5):

• Thank you: we have added notation definitions at the start of the Sec. 3 Methods section and clarified the notation further in 3.1 with more details.

"intuition why different masks are used in the training and evaluation [i.e. application]" (Q3)

• We replace the word "evaluation" with "application" here and in our text, so that it is more clear we are using the intermittent mask explicitly during application of REBAR in contrastive learning to identify positive/negative pairs. Our REBAR cross-attention model compares the motifs within a specific receptive field around a masked time-point in the query sequence with the motifs in the key sequence. Then, it retrieves the best matching motif from the key to be used for reconstruction of that specific masked time-point in the query, and this idea is illustrated in the new Fig. 1 and Fig. 3.
• At training time, a contiguous mask is used so that the model learns to compare specific, potentially class-discriminative, motifs in the query with specific, potentially class-discriminative, motifs in the key, rather than comparing minor, transient, non-unique motifs. See Fig. 4a) and b) for attention weight visualizations.
• At application time, a contiguous mask could be used, however, in doing so, only the motifs near the contiguously masked out region would be compared to the key. This is because when reconstructing a given masked time-point and another point contiguous to it, the receptive fields to be used to identify motifs to be compared to the key are heavily overlapping.
  • An intermittent mask allows for many different motifs in the query, each of them captured in a receptive field around the many masked time-points dispersed throughout the signal, to be compared to the key during reconstruction. This allows for a higher coverage of the query in conducting such motif-similarity comparisons with the key. Therefore, during application, when we are testing each candidate as the key for a given anchor as the query, we would be able to identify the candidate which is most similar to the anchor as a whole.
• We empirically justify this intuition. Table 4 in Appendix A.1.2 shows this combination of using a contiguous mask for training and an intermittent mask for application achieves the best performance in our validation experiment. Additionally, Figure 4b) demonstrates that the cross-attention model trained with a contiguous mask still maintains a sparse attention when evaluated on a transient mask. Therefore, the model is still reconstructing the query based on specific retrieved motifs from the key at each of the transiently masked out time-points.

REBAR performance on Distinctive Time-series Domains (Q4)

• We test our approach in three different time series domains which consist of signals with qualitatively very different properties, as illustrated in the new Fig. 10-12. The different classes within HAR are arguably more distinctive than the classes for ECG and PPG, although the different HAR categories of "walk", "walk up", and "walk down" require a more subtle distinction. This is reflected in our results, with our method performing the best on HAR. The fact that we achieve SOTA results in three very different data domains speaks to the effectiveness of our method. Regarding more systematic variations, we note that our datasets also contain substantial variation in signal lengths. The subsequence lengths are adapted to each of the signal domains and vary significantly in length, and the source signals that they are sampled from vary significantly as well. HAR uses 2.56s subsequences sampled from a 5 min time-series. PPG uses 60s subsequences sampled from an 87 min time-series. ECG uses 10s subsequences sampled from a 9.25 hour time-series.

More explanation of REBAR Convolutional Cross-Attention (Q5):

• Thank you for this comment. In Sec. 1 Introduction, we have added the new time-series motif language to help better explain the intuition of our idea, and a new Fig. 1 has been added to illustrate the intuition. Sec. 3 Methods has been completely revised and re-organized to incorporate the motif language. The start of Sec. 3 has been modified to introduce the idea of using REBAR's reconstruction error as a distance metric earlier, motivating our approach. Sec. 3.1 now ties the mathematical formulation of cross-attention with our new Fig. 3 to explicitly demonstrate how convolutions aid the cross-attention in retrieving and comparing motifs. Training REBAR is now its own section, Sec. 3.2, with Applying REBAR is Sec. 3.3. In these two sections, we are able to ground the masking motivation more clearly with the motif language. With the ablation study and hyperparameter sensitivity results in Tbl. 3, we can clearly understand how each of the REBAR cross-attention components contribute to downstream performance, as well as how to train the model. Finally, in the Appendix A.1, we include extra details on modeling decisions, as well as a new Fig. 8 that illustrates the exact dilated convolution module used in REBAR.