FreRA: A Frequency-Refined Augmentation for Contrastive Learning on Time Series Classification

We deeply appreciate the valuable feedback and constructive comments from the reviewer. We would like to address your questions below.

Q1. Compared to existing works, this work does not exhibit a notable advantage in classification performance.

A1. We would like to highlight the overall performance improvement of our FreRA compared to advanced SOTA methods on the 5 benchmarks in Table 1, as presented in the table below.

Although the performance gain of FreRA over SoftCLT on some datasets is relatively modest, this is primarily attributed to the use of the soft InfoNCE loss within its TS2Vec framework. For a fair comparison, we present the performance gains of our FreRA and the soft InfoNCE loss on the same TS2Vec framework in the table below (rows 2-3). We observe that FreRA brings larger improvements to the contrastive learning framework.

Moreover, FreRA and the soft InfoNCE loss represent independent improvements to the contrastive learning framework. Due to the plug-and-play design of FreRA, these enhancements can be seamlessly integrated. The results in rows 3-4 demonstrate that incorporating FreRA into SoftCLT further introduces additional gains to its performance.

Moreover, in the evaluations of transfer learning and anomaly detection tasks, our FreRA achieves significant improvements. This is evidence that FreRA can capture the inherent semantics of the time series and generalize to unseen data distributions and different downstream tasks.

Lastly, we highlight the significance and contributions of our work as follows:

• A novel perspective of designing automatic augmentation from the frequency domain. We provide a novel perspective to a practical and less-explored problem, automatic augmentation for time series contrastive learning, from the frequency domain. We identify three properties: global, independent, and compact, which advance the view generation and are beneficial for the time series classification task.

• A simple yet effective frequency-domain automatic augmentation. We develop FreRA, a lightweight and unified automatic augmentation method for contrastive representation learning in time series classification tasks. FreRA can be applied in a plug-and-play manner and is jointly optimized with the contrastive learning model.

Q2. Why was SHAR data selected instead of one of the datasets listed in Table 1 (e.g., UCIHAR) to evaluate the transfer capability of the algorithm?

A2. We selected the SHAR dataset because it presents a larger domain gap, as discussed in [1]. This larger domain gap makes the task more difficult and allows us to better assess the generalization and transferability of our method. It also makes our conclusion that FreRA leads to transferable representations more convincing.

Q3. What might be the reasons for the performance difference with the reference work (Qian et al., 2022)?

A3. The performance difference could be due to the different choices of the encoder and different hyper-parameter search scopes. The experiments in [1] aim to compare performances of various contrastive learning frameworks while our focus is mainly on the performance difference among different augmentations. It is worth noting that, within our work, we apply the same encoder and hyper-parameters search space, which constitute a fair comparison with baselines.

[1] Qian, Hangwei, Tian Tian, and Chunyan Miao. "What makes good contrastive learning on small-scale wearable-based tasks?." Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. 2022.

Q4. What did the transfer learning experiment aim to prove?

A4. The transfer learning experiment aimed to evaluate whether representation learning with FreRA could effectively transfer useful knowledge to data from unseen domains that have large domain gaps with the training data. The result demonstrates that FreRA achieves stronger transferability compared to other baselines. This is attributed to FreRA's ability to infuse variance into the augmented view while leaving critical semantics intact. The encoder thus learns representations that capture the inherent semantics and disregard environment noise. FreRA therefore helps to reduce the domain gap through enhanced augmentation. It further verifies the effectiveness of FreRA.

Q5. If all three modules were removed, what would be the resulting performance?

A5. If all three modules were removed, the model would downgrade to a vanilla contrastive learning framework where both views are the original input. We present the results averaged over 30 datasets from the UEA archive in the table below. The significant performance drop when all three modules are removed (the fourth column) indicates removing the augmentation leads to collapsed representation learning. To remove all three modules and avoid collapsed training, a simple way is to apply basic augmentations such as pre-defined time-domain and frequency-domain augmentations. The results are presented in the last two columns

Q6. If FreRA were integrated into softCLT, would there be any gain in performance?

A6. We thank the reviewer for the valuable question. We integrate FreRA into SoftCLT and observe a gain in performances on the three large HAR datasets, as shown in the table below. These results further validate FreRA’s flexibility and effectiveness when applied to other contrastive learning frameworks.

We deeply thank the reviewer for raising important questions. We sincerely hope our clarifications above have addressed your concerns and can improve your opinion of our work.

Dear Reviewer 7F2V,

We would like to express our gratitude for your constructive comments and questions. We look forward to further discussions if you have any remaining questions.

Best,

Paper 6561 Authors

We deeply appreciate the valuable feedback from the reviewer. We would like to address your concerns below.

Q1. At the beginning of the paper, the authors make strong assumptions that existing predefined augmentation methods are primarily adopted from vision and are not specific to time series data. There are already several methods, especially frequency-based augmentation, e.g., TF-C, method design for time series contrastive learning.

A1. We thank the reviewer for pointing out this important point. We are aware that existing pre-defined augmentations include time-domain and frequency-domain augmentations. Time-domain augmentations are mostly adopted from the vision domain. They often introduce mismatched patterns into the data because they do not account for the intrinsic characteristic of time series. On the other hand, frequency-based augmentations, such as high-pass and low-pass filters, require prior knowledge of the dataset to determine the selection of appropriate augmentation functions. Moreover, other stochastic frequency-domain augmentations, such as the frequency-based augmentation in TF-C, introduce random noise that can interfere with the critical information.

The challenges of existing predefined augmentations can be summarized as follows:

• They often fail to consider the intrinsic characteristics of time series data, resulting in mismatched patterns due to stochastic perturbations.
• Certain augmentations require prior knowledge of the dataset, which is not always accessible in the contrastive learning paradigm.
• The wide range of possible augmentation functions requires extensive trials and errors to select the optimal one, making the augmentation process costly and less practical.

The sentence referenced by the reviewer was intended to highlight the problems with existing predefined time-domain augmentations and introduce the novel perspective from the frequency domain. We have revised the sentence to make it more precise and accurate. Moreover, we have included a detailed discussion of the frequency-based augmentations and their problems in Appendix A.2.

Q2. Since the paper mainly provides the frequency-based augmentation, the motivation study, such as Figure.1 probably should highlight more about whether current frequency-based method can capture the semantics, rather than only focus on the time-domain,

A2. We thank the reviewer for the valuable suggestion. We add the pre-defined frequency-domain augmentation 'amplitude-and-phase-perturbation' denoted as $\mathcal{T}_f(\textsf{x})$ in Figure 1 with explanations updated in our manuscript. It still fails to preserve semantic integrity in the augmented view as we observe from its low mutual information (MI) values. This further explains the limitation of existing augmentations in capturing semantics and highlights the importance of our FreRA.

We deeply thank the reviewer for raising insightful comments. We sincerely hope our clarifications above have addressed your concerns and can improve your opinion of our work.

Dear Reviewer GiHe,

We would like to thank you for your valuable suggestions on enhancing our work. We have provided a response to address your concerns and revised our paper accordingly. We are happy to have further discussions if you have any outstanding concerns or questions.

Best,

Paper 6561 Authors

We thank the reviewer for the time to review our work and offering valuable comments. We address your questions below.

Q1. clarify important terms

A1. We clarify the important terms as follows.

• "Semantic integrity" refers to the preservation of the meaningful features within the time series that are essential for the downstream classification tasks, as we explained in lines 65-69 in the revised paper. It is opposed to 'semantic degradation'. In math, it is defined as the state of augmented view $\mathsf{v^\ast}$ when $\text{MI}(\mathsf{v^\ast},\mathsf{y}) = \text{MI}(\mathsf{x},\mathsf{y})$, where $\mathsf{x}$ and $\mathsf{y}$ are the random variables denoting time series sample and the label, and $\text{MI}$ represents mutual information.

• "Critical frequency components" refers to the frequency components that contain substantial information related to the downstream classification task, such as those containing key recurring patterns in the time series. Conversely, "unimportant frequency components" are those that contribute minimally to classification tasks, such as noise from the environment.

Q2. Why can we measure semantic integrity using mutual information? Is this consistent with humans’ understanding of semantics?

A2. From the definition of mutual information, the value of $\text{MI}(\mathsf{v^\ast},\mathsf{y})$ quantifies the amount of information the augmented view $\mathsf{v^\ast}$ can provide about the label. Therefore, it measures the extent to which critical semantics relevant to $\mathsf{y}$ are preserved, which is the definition of semantic integrity as we explained above.

Unlike images and natural language, the semantics of time series are not intuitively recognizable for human understanding. In other words, given a time series signal, humans’ understanding of its semantics is ambiguous and vague. Alternatively, by quantitatively measuring the mutual dependency between the time series and its semantics, the mutual information is superior to humans' understanding in reflecting the completeness of unintuitive semantics. Moreover, extensive works [1-3] have applied mutual information to measure the amount of task-relevant semantics information, which supports its usage in our work.

[1] Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).

[2] Tian, Yonglong, et al. "What makes for good views for contrastive learning?." Advances in neural information processing systems 33 (2020): 6827-6839.

[3] Luo, Dongsheng, et al. "Time series contrastive learning with information-aware augmentations." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 4. 2023.

A3. How can we measure the importance of frequency components?

A3. Similar to how we quantify the important information in the time domain, we use mutual information to measure the importance of frequency components. Since the frequency components are complex numbers, we calculate MI using only their real parts for compatibility. In Figure 6 of our revised manuscript, we visualize the MI of 3 datasets (Libras, ArticularyWordRecognition, and Epilepsy) by blue-grey bar plots. The important components (those with high MI values with labels) are distributed in low frequencies, middle frequencies, and across multiple frequencies in these datasets, respectively, showing diverse distribution. Despite the diversity, the learned vector $\mathbf{s}$, plotted in orange lines, consistently captures the inherent critical information by learning to assign higher values to the most semantically relevant frequency components. This further verifies the effectiveness and the generalization of FreRA across diverse distributions.

Q4. What are these critical components critical for?

A4. Intuitively, the critical components are critical for the downstream tasks as they contain task-related information. In practice, preserving the critical components in the augmented view allows the learned representations to have good performance in the downstream classification tasks.

Q7. How FreRA achieves both semantic-preserving information and a considerable amount of variance and which designs correspond to these two sides respectively.

A7. Semantic-preserving information and a considerable amount of variance correspond to $\mathbf{w}\_{\text{crit}} \odot x_f$ and $\mathbf{w}\_\text{dist} \odot x_f$ in the Eq. (2) of our manuscript, respectively. They are achieved through the corresponding modification vectors $\mathbf{w}\_{\text{crit}}$ and $\mathbf{w}_\text{dist}$, which are derived from the identity modification module and self-adaptive modification module, respectively. The details are presented below.

1. Semantic-preserving information is achieved by the identity modification module on critical frequency components. Specifically, we learn a lightweight trainable parameter vector $\mathbf{s}$ to capture the inherent semantic distribution in the frequency domain. The semantic-preserving information is preserved by the critical frequency components identified by $\mathbf{s}$.

2. A considerable amount of variance is achieved by the self-adaptive modification module on unimportant frequency components. In the vector $\mathbf{s}$, the value of each element $s_i$ indicates the importance of the $i$-th frequency component $x_f^i$ for the global semantics, as defined in lines 321-322 in the revised paper. Therefore, frequency components with smaller values of $s_i$ are considered unimportant. To infuse variance, we apply perturbation to those unimportant components. Disturbing them does not affect the semantics of the time series because they are independent of the critical frequency components. The selection of unimportant components and the strength of perturbation on each one of them are designed to be adaptive to the input time series dataset. This ensures the amount of variance infused is appropriate.

Q8. The self-adaptive modification seems simple and tricky. It only uses the vector s and threshold to select and scale unimportant frequency components. The motivation for scaling these components is unclear.

A8. The motivation for scaling these components is to infuse variance into the augmented view. Previous studies [3-4] have shown that sufficient variance or task-irrelevant noise can improve the performance of contrastive learning models.

However, they achieve this by adding random noise in the entire time domain, which introduces variance while inevitably interfering with the critical information. In contrast, FreRA avoids this issue by selectively adding variance to the well-separated unimportant frequency components. This ensures the critical semantics are well preserved. Moreover, the global property of the frequency components ensures that all timestamps are altered with distortion applied only to unimportant components.

The self-adaptive modification is simple to implement in practice. However, it is deliberately designed to adaptively add variance to the augmented view. The adaptive nature of the self-adaptive modification is reflected in two aspects:

1. Adaptive unimportant component selection: Instead of handpicking a threshold value to separate the unimportant components from the rest, we determine the value with statistical information of the vector $\mathbf{s}$. Specifically, we use the mean value of $\mathbf{s}$ as the threshold and $D = \{i | s_i < \min(0, \frac{1}{F} \sum_{i=1}^{F}{s_i})\}$ to denote the set of unimportant components' indices, as explained in lines 344-350 in the revised paper.
2. Adaptive strength of modification of each unimportant component: To adjust the degree of distortion according to the irrelevance of each frequency component, a scaling factor $\delta_s = \frac{1}{\lvert D \rvert} \sum_{i=1}^{F} \mathbb{1}_{\{i \in D\}} \lvert s_i \rvert$ is applied to the unimportant components. This ensures the least important frequency component gets amplified mostly in the distortion step, as explained in lines 351-355 in the revised paper. Its effectiveness has been empirically verified, as detailed in our response to Q10.

Despite the simple but meticulous design, the effectiveness of the self-adaptive modification module has been verified by the result of the ablation study presented in Table 3. It improves the average accuracy of the UEA archive from 0.695 to 0.754, with 0.059 absolute improvement.

[3] Luo, Dongsheng, et al. "Time series contrastive learning with information-aware augmentations." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 4. 2023.

[4] Zheng, Xu, et al. "Parametric Augmentation for Time Series Contrastive Learning." arXiv preprint arXiv:2402.10434 (2024).

Q10. The ablation study is coarse, and some important variants are missing. For example, modifying all (or randomly selected) frequency components and modifying unimportant components randomly.

A10. We thank the valuable advice from the reviewer. We have included a more comprehensive ablation study considering three more variants of infusing variance into the frequency domain:

• Modifying all frequency components with stochastic perturbation.
• Modifying randomly selected frequency components with stochastic perturbation.
• Modifying unimportant frequency components with stochastic perturbation.

The results averaged over 30 datasets from the UEA archive are presented in the table above. The performance drop from the third setting highlights the effectiveness of self-adaptive modification applied to unimportant components, as compared to stochastic perturbation. However, randomly disrupting the unimportant components is still better than directly removing them from the generated view ("w/o modification on noise components" in Table 3). The first two settings demonstrate the importance of isolating critical components when introducing noise. Modifying all or random frequency components inevitably interferes with the critical components, which damages the semantic information in the generated views and leads to degraded performance. Randomly perturbing all frequency components results in larger performance drops.

Q11. As this paper focuses on time series classification, why is it also evaluated in anomaly detection?

A11. In this work, we treat anomaly detection as a 3-class classification problem. Notably, the class distribution is highly imbalanced (the class distribution follows 1240:6200:6200) in the anomaly detection dataset. The evaluation results in the anomaly detection tasks further verify the effectiveness of our method in a more difficult problem and demonstrate its generalizability across different tasks beyond standard time series classification.

Q12. Figure 2 is hard to read. The different colors for vector blocks in FreRA are confusing.

A12. We thank the reviewer for sharing the feedback on Figure 2. We have made the following amendments in our revised manuscript to make the figure more clear and readable.

1. Update the legend to clarify the use of colors for the vector blocks.
2. Add explanation to the colored blocks in $\mathbf{w}_\text{dist}$ in the caption.

The use of different colors for the vector $\mathbf{s}$ is intended to indicate that $\mathbf{s}$ learns to assign different levels of importance to the frequency component. The colored blocks in $\mathbf{w}\_\text{dist}$ are to illustrate adaptive distortion strength on unimportant frequency components ($\mathbf{w}\_\text{dist}$ has matching colors with $\mathbf{s}$ on the positions of unimportant components).

We deeply thank the reviewer for raising important questions. We sincerely hope our clarifications above have addressed your questions and concerns and can improve your opinion of our work.

Q9. Compared with some SOTA baselines, such as SoftCLT and InfoTS, the advantage of FreRA is not clear, especially on UEA and UCR datasets.

A9. We would like to highlight the overall performance improvement of our FreRA compared to advanced SOTA methods on the 5 benchmarks in Table 1, as presented in the table below.

Although the performance gain of FreRA over SoftCLT and InfoTS on the UEA and UCR archives is relatively modest, this is primarily attributed to the superior TS2Vec framework they both utilize. To illustrate, we present the baseline performances of TS2Vec and SimCLR in row 1 and row 4 of the table below. We notice that SimCLR is a weaker framework on the UEA and UCR archives. However, integrating FreRA to SimCLR results in significant improvements to the contrastive learning framework, which exceed the performance gains achieved by the augmentation in InfoTS and the soft InfoNCE loss in SoftCLT. This improvement eliminates the performance gap (0.018 for the UEA archive and 0.101 for the UCR archive) caused by the inferior SimCLR framework as compared to TS2Vec.

Moreover, FreRA and the soft InfoNCE loss in SoftCLT represent independent improvements to the contrastive learning framework. Due to the plug-and-play design of FreRA, these enhancements can be seamlessly integrated. The results in rows 3-4 demonstrate that incorporating FreRA into SoftCLT further introduces additional gains to its performance.

Notably, in the evaluations of transfer learning and anomaly detection tasks, our FreRA achieves significant improvements. This is evidence that FreRA can capture the inherent semantics of the time series and generalize to unseen data distributions and different downstream tasks.

Dear Reviewer Qixi,

We would like to express our gratitude for your valuable questions and suggestions. We have provided point-by-point replies to your concerns. We are wondering if our response has properly addressed your concerns. We look forward to discussions with you if you have any outstanding concerns or questions.

Best,

Paper 6561 Authors

Thank you for the rebuttal. Some of my concerns have been addressed.

I think the technical novelty of the method, which is mainly based on the trainable vector S is still a little limited. Although the vector S can be automatically trained, it still needs to control the trade-off between the contrastive objective and the regularization term to balance huge semantic changes and trivial solutions of no changes. Considering this, the idea of this method seems similar to simply controlling the strengths when using augmentations. Currently, the critical components are a little abstract to me. It may need more analysis to show their properties and why they are critical. Furthermore, using the contrastive objective to learn data augmentation is not reasonable enough for me, as this objective does not clearly encourage meaningful changes in data from the augmentation. It is also unclear why S can be used to adaptively modify unimportant components, considering that it is not directly trained to do this. Besides, this method may need more significant improvements to show its effectiveness. It is not very convincing to claim that FreRA does not outperform SoftCLT clearly because SoftCLT uses TS2Vec.

We thank the reviewer for the constructive comments and valuable feedback. We address your concerns below.

Q1. The importance distinction of this method is mostly for the entire time series, and it could be better to compare it with other methods and analyze the theoretical computational complexity.

A1. We have included comparisons with key baseline methods in the current version, including (1) 11 commonly used handcrafted time-domain augmentations (2) 5 handcrafted frequency-domain augmentations (3) 3 SOTA automatic augmentation for contrastive learning, and (4) 5 SOTA time series contrastive learning frameworks. These baselines also operate on the entire time series and are strong benchmarks for time series classification tasks. The uniqueness of our FreRA lies in its superior ability to preserve global semantics from the entire time series, as illustrated in Figure 1 and the results.

The overall computational complexity involves three components as explained below:

1. The transformation involves two parts: - The frequency-domain augmentation involves a Fourier Transform and an inverse Fourier Transform. Implemented by the Fast Fourier Transform (FFT), the computational complexity is $\mathcal{O}(L \log L)$, where $L$ is the sequence length. - Based on the learned $\mathbf{s}$, the identity modification module and the self-adaptive modification module conduct element-wise operations and introduce computation complexity of $\mathcal{O}(F)$.
2. Update of trainable parameters. The augmentation function is parameterized by a lightweight vector $\mathbf{s} \in \mathbb{R}^{F}$, where $F=\lfloor{L/2}\rfloor+1$ is the number of frequency components. The computational complexity of updating $\mathbf{s}$ is $\mathcal{O}(F)$.
3. The auxiliary loss introduced by the augmentation. $\mathbf{w}_\text{crit}$ is regularized by the L1-norm. The computational complexity introduced by the loss term is $\mathcal{O}(F)$.

The overall computational complexity is dominated by the Fourier Transform and its inverse. Hence, the overall complexity for FreRA can be approximated as $\mathcal{O}(L \log L)$.

In addition, we analyze the computational complexities of the other three SOTA automatic augmentations are present them in the table below. The analysis accounts for per-instance complexity. $B$ and $d$ denote the batch size and feature dimension respectively.

The overall computational complexity of InfoMin$^+$ clearly dominates FreRA. The efficiency of the other three methods depends heavily on the setting of hyper-parameters $B$ and $d$. FreRA consistently achieves competitive performance without imposing significant computational burden, making it a superior option for practical applications.

Q4. The authors claim that FreRA can be benefited by any contrastive learning framework, but only show the results of InfoNCE. What about other CL paradigms, such as SimCLR, etc.? It could be better to present more sufficient ablation.

A4. We thank the reviewer for the insightful comment. In our submission, we apply the architecture of the widely used SimCLR with InfoNCE as the contrastive learning framework. The reason we use InfoNCE instead of NT-Xent, as originally applied in SimCLR, is the better empirical performance, as shown in the table (rows 7-12) below. The same usage has been deployed in [1-2] as well. Moreover, we provide an ablation study evaluating FreRA on alternative contrastive learning frameworks, including TS2Vec, TS-TCC and BYOL, in the Appendix.

Following the suggestions from the reviewer, we have further expanded our evaluation to include additional contrastive learning frameworks: the SimCLR architecture with NT-Xent as contrastive loss functions, as well as an advanced contrastive learning framework SoftCLT. Our current evaluation covers 5 contrastive learning frameworks and 3 types of contrastive loss functions. It is worth noting that the contrastive losses used in TS-TCC, TS2Vec, and SoftCLT are different variants of InfoNCE, each with its unique formulation. The results presented below consistently demonstrate that FreRA is a plug-and-play method that consistently and effectively enhances existing contrastive learning frameworks.

[1] Yeh, Chun-Hsiao, et al. "Decoupled contrastive learning." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[2] Wu, Junkang, et al. "Understanding contrastive learning via distributionally robust optimization." Advances in Neural Information Processing Systems 36 (2024).

Q5. The experimental results are selected from the highest performances among 11 time-domain augmentations and 5 frequency-domain augmentations. Is this fair enough? There seems to be randomness with such selection strategy.

A5. The highest performances we present are the upper limits of pre-defined augmentations. Such a selection strategy, listing only the highest results, also intends to ensure a simple and concise tabular presentation without randomness.

The selection of the optimal pre-defined augmentation is data-specific and there is no unified pre-defined augmentation that works well on all the datasets. Even if we report the best empirical results of the optimal augmentations selected from exhaustive trials and errors on each dataset, they are still worse than the results of our FreRA. In this context, the comparison is fair enough, as the selection strategy aims to find out the augmentations that empirically best suit the given dataset. This comparison again highlights the effectiveness of FreRA.

Q6. The results of the impacts of hyper-parameters could be moved to the main paper for a better organization.

A6. We thank the reviewer for the suggestion. We have included the ablation study on the impacts of hyper-parameters in the main paper accordingly. Due to the space limit, the plot and the detailed analysis remain in the Appendix, but the main conclusions of this ablation study are accessible from the main paper.

We deeply thank the reviewer for raising insightful comments. We sincerely hope our clarifications above have addressed your concerns and can improve your opinion of our work.

Dear Reviewer cXW7,

We would like to thank you again for providing valuable feedback and constructive suggestions. Please kindly let us know if our response has addressed your questions. We look forward to more discussions if you have further comments.

Best,

Paper 6561 Authors