Invariant Convolutional Layers for Time Series

• [Reply to Q2 - Learning Invariant Features in the Frequency Domain:] As the reviewer pointed out, learning invariance in the frequency domain is also a suitable approach for time series. Models operating in the frequency domain typically decompose a signal into its Fourier basis, apply a filter via point-wise multiplication, and then transform the signal back to the temporal domain. However, applying such filters directly to the decomposed signal inherently leads to the learning of global invariance rather than the local invariance we address with our framework. To mitigate this issue, methods like TSLANet [1] segment the signal and apply filters in the frequency domain to each segment. While this limits the filters to specific frequencies due to the discrete Fourier transform (DFT), it raises the challenge of capturing local invariant features in the temporal domain while maintaining sufficient model discriminativeness. This trade-off can be understood through the uncertainty principle, which governs the resolution trade-off between time and frequency domains. Our approach addresses this challenge by operating entirely in the temporal domain. This allows us to focus on extracting local features invariant to deformations, which require a broad spectrum of frequencies to be encoded. As demonstrated in our experiments (Section 4), this distinction leads to both performance and computational advantages. Our lightweight network (200K parameters for an example dataset in Figure 3) achieves top performance (0.72 accuracy on the UEA dataset, as shown in Table 1 and Figure 3) by encoding invariance to real-world deformations such as amplitude, offset, and trend. In contrast, TSLANet [1], a significantly larger network (3M parameters), achieves lower performance (around $69\%$ average accuracy on UEA).

• [Reply to Q3 - Relevant Works on Invariant CNNs and Shift Invariance:] We would like to refer the reviewer to our General Comments section and specifically [Comment 1], where we provide justifications about the positioning of the work with respect to works on invariant CNNs. In the original draft, we had originally added this discussion in the introductory Section 1, page 2, in the second paragraph. We are willing to change this paragraph if the updated discussion addresses the expressed concerns. Regarding the referenced papers [2, 3], the first one [2] deals with shift invariance in image processing. While convolutions are inherently equivariant to shifts, achieving shift invariance typically involves ending convolution layers with an average pooling layer, ignoring boundary effects. However, most current architectures insert downsampling layers between convolution layers, which introduces sensitivity to shifts. In [2], the authors address this issue by proposing a downsampling method that maintains shift invariance. Similarly, our proposed invariant convolutions are also equivariant to time shifts, and networks using these convolutions can achieve time-shift invariance depending on their structure. For example, in our experiments, all networks are time-shift invariant because we conclude the convolution layers with average pooling without incorporating downsampling layers. The second paper [3] addresses data imputation in traffic time series. Importantly, time series have a meaningful sequential structure where the order of observations is critical. However, traditional methods for data imputation based on low-rank matrices are invariant to permutations of rows and columns, which leads to poor performance on time series data. To address this limitation, the authors proposed a novel imputation method that leverages convolution to ensure local dependence on time ordering, thereby counteracting permutation invariance. This approach differs significantly from ours, as their work focuses on overcoming irrelevant invariance in existing imputation methods by exploiting time series structures. In contrast, our work focuses on encoding meaningful invariance for time series within convolutions.

[1] Eldele, E., Ragab, M., Chen, Z., Wu, M., & Li, X. (2024). Tslanet: Rethinking transformers for time series representation learning. arXiv preprint arXiv:2404.08472.

[2] Jacques, B. G., Tiganj, Z., Sarkar, A., Howard, M., & Sederberg, P. (2022, June). A deep convolutional neural network that is invariant to time rescaling. In International conference on machine learning (pp. 9729-9738). PMLR.

[3] Chen, X., Cheng, Z., Cai, H., Saunier, N., & Sun, L. (2024). Laplacian convolutional representation for traffic time series imputation. IEEE Transactions on Knowledge and Data Engineering.

We would like to sincerely thank the reviewer for acknowledging the good level of our contribution in terms of methodological soundness, mathematical presentation, and significant performance improvements. We are pleased to address the concerns raised by the reviewer by carefully replying to the provided comments and questions below.

• [Reply to Weakness - Comparison with Linear and Frequency-Domain Models:] MLP-based models and models that extract frequency-domain features have indeed recently proven quite successful in time series modeling, with several relevant architectures that we thoroughly present in the provided related work section (please see the first paragraph of Section 2). In terms of comparisons for both classification and anomaly detection, we leverage 2 state-of-the-art frequency-based deep learning architectures, namely TimesNet and TSLANet, that capitalize on Fast Fourier Transform to extract frequency-domain representations to capture multiple periodicities and achieve robustness to noise, respectively. Both frequency-based TimesNet and TSLANet methods are experimentally motivated as general-purpose convolutional time series modeling methods which explains our choice to include them as baselines for both classification and reconstruction-based anomaly detection. On the contrary, the FITS method proposed by the reviewer, while built upon frequency-domain features, is originally designed and motivated for time series regression tasks, namely forecasting, and anomaly detection. In this direction, we have already considered in our study several successful models in time series regression tasks applied to time series classification (we refer to TimesNet, TSLANet, DLinear, PatchTST, Crossformer baselines) and even more in number for time series anomaly detection (we refer to TimesNet, TSLANet, DLinear, LightTS, PatchTST, ETSformer, FEDformer, Autoformer, Pyraformer, Informer, Reformer). With respect to the MLP-based baseline DLinear that is suggested by the reviewer, we would like to clarify here that it is already included as a baseline in both classification and anomaly detection experiments and significantly outperformed the proposed method, and several baselines, especially for the CNN-dominated task of time series classification. Based on the above justifications, we believe that the considered baselines include well-established and widely cited approaches, reflecting a broad spectrum of effective building blocks (such as convolutional, attention, and linear but also frequency-based ones) from the relevant literature.

• [Reply to Q1 - Layer Design for Variant and Invariant Features:] We think there is a potential misunderstanding in this comment. Our proposed convolutional layer is not built solely upon invariant filters. In the methods section 3, paragraph pool of convolutions, we indeed provide mathematical formulations for the design of convolution layers, including filters of increasing order of invariance complexity. In our experiments, we considered a pool of convolution layers concatenating three types of filters, which are (i) normal ones (similar to vanilla convolutional filters), (ii) filters invariant to offset shift and amplitude scaling, and (iii) filters invariant to offset shift, linear trend and amplitude scaling. Based on this, our proposed convolutional layer is built upon both variant and invariant filters in order to capture both variant and invariant features of the time series.

• [Reply to Q1 - Effect of Kernel Size in Capturing Invariances:] The kernel size of the proposed convolutional layer is indeed an interesting aspect for further discussion. More specifically, we can intuitively assume and experimentally confirm that a kernel, in order to have sufficient information to extract the designed amplitude-related invariances, needs points that are at least equal to a minimum value between 50 and and half of the length of the time series (which is also mentioned in Appendix A.4 in hyperparameter selection). This choice of the kernel length, while rather heuristic, seems to apply with success to the wide range of the tested time series datasets. Additionally, training larger kernels in our layers is achieved by the Fast Fourier Transform, as mentioned in Section 3.2, page 6. On the contrary, leveraging very small kernels, e.g., of length 4 or 8, that are commonly employed in standard convolutions could not be sufficient for learning robust representations incorporating invariances with respect to amplitude scaling. Finally, like standard convolutions, very large kernels can be inefficient in training.

• [Reply to Q2 - Computational Efficiency of FFT-based over Standard Convolution:] As thoroughly explained in the methods Section 3.2, page 6, paragraph on the fast computation of our proposed convolutional layers, we leverage the Fast Fourier Transform (FFT) for performing the proposed convolutions [1] to enable the use of larger kernel sizes (unlike standard CNN-based models that typically use small kernel sizes of 2 or 4 elements in the convolution operation). Based on this fast implementation, the computation is not affected in terms of time cost by the kernel size. While several implementations can be publicly found inside the repository [2] from which we derived the code for the FFT computation of the convolution, there is a time complexity comparison for the execution time with increasing kernel sizes for standard (in blue) and FFT-based convolution (in red). This plot confirms the time efficiency of FFT-based convolution for increasing kernel sizes. For $N$ output channels, the complexity of the operations using standard convolutions (non-FFT-based) is $\mathcal{O}(BNCWL)$, where $B$ is the batch size, $L$ is the length of the series, $C$ the channels and $W$ the width of the kernel size, whereas for the proposed convolutions (FFT-based) as described above and in the relevant section in the manuscript, the complexity of the computations is $\mathcal{O}(BNCL\log (L))$. Therefore, for $\log(L) < W$, the proposed convolutional layer is more efficient than standard 1D-CNN.

• [Reply to Q3 - Invariant Convolution for Time Series Compared to Normalization Techniques:] The z-normalization discussed in the example on page 5 is a well-established technique in time series data mining [4]. It produces an embedding that is invariant to amplitude scaling and offset shifts, a property that is naturally derived within our mathematical framework. In the deep learning community, z-normalization is commonly associated with the batch normalization layer, which is designed to accelerate and stabilize the learning process.

However, there is a fundamental distinction between batch normalization and our approach. Batch normalization operates at a global scale, computing invariance for the entire time series. In contrast, the proposed invariant convolution computes invariance at a local scale, corresponding to the size of the convolutional kernel. This means that our method captures features that are locally invariant to well-designed time series deformations.

[1] Mathieu, M., Henaff, M., & LeCun, Y. (2013). Fast training of convolutional networks through ffts. arXiv preprint arXiv:1312.5851.

[2] Odom, F. III. (2023). fkodom/fft-conv-pytorch [Software]. Plainsight. Retrieved from https://github.com/fkodom/fft-conv-pytorch

[3] Tang, W., Long, G., Liu, L., Zhou, T., Blumenstein, M., & Jiang, J. (2020). Omni-scale cnns: a simple and effective kernel size configuration for time series classification. arXiv preprint arXiv:2002.10061.

[4] Paparrizos, J., Liu, C., Elmore, A. J., & Franklin, M. J. (2020, June). Debunking four long-standing misconceptions of time-series distance measures. In Proceedings of the 2020 ACM SIGMOD international conference on management of data (pp. 1887-1905).

• [Reply to W2 - Choice of Baselines/Inclusion of CNN methods:] There is potentially a misunderstanding for the reviewer with respect to the type of neural network architectures used as baselines for time series classification. More specifically, as shown in Table 1, we use 9 baselines in total, among which 6 are convolutional-based (namely TimesNet, TSLANet, Inception, ResNet, CNN, and Rocket), 2 are built upon attention-forming transformer-like architectures (i.e., PatchTST and Crossformer), and 1 is a purely linear architecture (DLinear). Additional transformer-based architectures were used as baselines for the task of reconstruction-based anomaly detection, following the recent trends in time series regression tasks, but definitely this is not the case for the experimental design we followed in classification. Additionally, among the two self-supervised contrastive learning methods evaluated in the Transfer Learning experiment, TS-TCC is built upon transformer-like modules, while TS2Vec is built upon stacked dilated convolutions interleaved with max-pooling operations. Based on the above, the vast majority of the considered baselines for the classification tasks are indeed convolutional, yet a wide variety of building modules has been tested for the different setups, always following the trends of relevant papers in this field.

• [Reply to W3 - Details of Hyperparameters in InvConvNet Design:] We would like to ask the reviewer about the exact part that could potentially benefit from more details on the hyperparameter selection for the employed Invariant Convolutional Layers inside the example InvConvNet architectures. Explicit details about the ranges in which hyperparameters, such as the kernel size and the number of hidden dimensions, were chosen are explicitly stated in Appendix A.4, 3rd paragraph, page 19, lines 995-1006. Specific combinations for each dataset are provided in scripts inside the code of the supplementary material for InvConvNet and the baselines used. The example InvConvNet architectures we leveraged to showcase the representational capabilities of the proposed Invariant Convolutional Layers are thoroughly explained in Appendix A.2.1 and Figures 4 and 5. Among the three different tested variants, two are single layer (in terms of depth - depicted in Figure 4(Left)), the so-called standard InvConvNet (1 kernel size), and the inception-like InvConvNet (3 distinct kernel sizes operated in parallel). Both single-layer variants excel in the datasets used for classification, as mentioned in Appendix A.4. Finally, one InvConvNet variant, the so-called multi-scale (depicted in Figure 4(Right)), is a 2-layer to capture finer granularity, which is crucial for the reconstruction-based anomaly detection task. The first layer is the proposed convolutional layer built upon a pool of convolutions (of kernels that are (i) variant, (ii) invariant to amplitude scaling and offset shift, and (iii) invariant amplitude scaling and linear trend invariant) and the second layer is a standard 1D-CNN (details are given in the appendix).

We really appreciate the time devoted by the reviewer to provide very interesting comments on our contribution and would like to thank him for highlighting the clear presentation, mathematical soundness, and efficient implementation of our proposed Invariant Convolutional Layers. In response to the issues raised by the reviewer, we address each point in detail.

• [Reply to W1 - Pool of Convolutions:] In the methods section 3, we provide the mathematical formulations for the design of a novel convolutional layer that can be invariant to a pool of predefined deformations. In our experiments, the pool of convolutions corresponds to the concatenation of three types of filters in our proposed convolutional layer, which are (i) normal ones (similar to vanilla convolutional filter), (ii) filters invariant to offset shift and amplitude scaling, and (iii) filters invariant to linear trend and amplitude scaling. The formalization for the pool of convolutions is provided in the relevant paragraph on page 6, and its components are visualized in Figure 2 (Right). The term "pool of convolutions" in our case, deviates from the one in the paper provided by the reviewer [3], in the fact that we refer to the operation of combining different types of filters with respect to different invariances. Combining the output of vanilla (non-invariant) convolutional filters (e.g., by concatenation), which is performed in [3], is rather a very common practice for extracting features corresponding to different receptive fields based on the characteristics of the filter (e.g., kernel size, dilation). For instance, this is also common in the classical Inception architecture, which we use as a baseline, among several others.

• [Reply to W1 - Components Contribution in Invariant Convolutional Layer:] We already provide ablation studies for the component's contribution to our proposed convolutional layers, with respect to the potential influence of the types of invariances in multivariate classification for all considered datasets (26 datasets of UEA, UCIHAR, Sleep-EDF, Epilepsy) in Table 2 and additional results and for the synthetic experiment on the 4 larger univariate UCR datasets in Table 3 (see also [Comment 2] of the General Comments section for [Additional Results] provided for a stronger synthetic deformation). For the ablation studies on the components performance, we consider filters that correspond to (i) mixed types of filters (model named InvConvNet), (ii) solely normal filters (no invariance, model named InvConvNet-N), (iii) filters invariant to offset shift and amplitude scaling (model named InvConvNet-O), and (iv) filters invariant to linear trend and amplitude scaling (model named InvConvNet-T). Our proposed layer is built upon mixing the three types of filters in terms of invariance (denoted as the pool of convolutions). In the ablation study, we consider the same optimal total number of hidden dimensions, which is derived by tuning a hyperparameter denoted as $d$. For the mixed types of invariances, each type of convolution receives almost one-third of the $d$ dimension, with adjustment to sum up to $d$ (more details are explained in the hyperparameter selection paragraph of Section Appendix A.4).

• [Reply to W1 - Comparisons with Multi-Scale CNNs:] As mentioned by the reviewer, capturing information at different scales with CNN is a common approach for time series, which has proved to be relevant in numerous cases. Typically, this is achieved either by increasing the network's depth or by combining kernels of different sizes, as in Omni-scale [3]. Among the considered baselines, several are built upon stacked convolutions or different kernel sizes to capture multiple scales, such as TSLANet, Inception, ResNet and Rocket. In our manuscript, we formally introduce the invariant convolution for arbitrary kernel sizes. In the experiments, we incorporate kernels of three different sizes to remain competitive with other methods, which also employ multi-scale strategies. It is worth mentioning that we did not focus on fine-tuning the kernel sizes, as they were determined heuristically (see Appendix A.4). The critical question of whether combining invariance with CNNs provides improvements independent of multi-scale strategies is addressed in our ablation study (Section 4, page 8). On the UEA datasets, we observed a $3\%$ increase in accuracy when incorporating invariant convolutions of various types into CNNs. This result demonstrates that adding invariance to convolutions enhances the performance of CNNs for classification tasks.

[1] Kvinge, H., Emerson, T., Jorgenson, G., Vasquez, S., Doster, T., & Lew, J. (2022). In what ways are deep neural networks invariant and how should we measure this?. Advances in Neural Information Processing Systems, 35, 32816-32829.

[2] Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C. K., Li, X., & Guan, C. (2021). Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112.

[3] Lessmeier, C., Kimotho, J. K., Zimmer, D., & Sextro, W. (2016, July). Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference (Vol. 3, No. 1).

[4] Ragab, M., Chen, Z., Wu, M., Li, H., Kwoh, C. K., Yan, R., & Li, X. (2020). Adversarial multiple-target domain adaptation for fault classification. IEEE Transactions on Instrumentation and Measurement, 70, 1-11.

[5] Jacques, B. G., Tiganj, Z., Sarkar, A., Howard, M., & Sederberg, P. (2022, June). A deep convolutional neural network that is invariant to time rescaling. In International conference on machine learning (pp. 9729-9738). PMLR.

[6] Chaman, A., & Dokmanic, I. (2021). Truly shift-invariant convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3773-3783).

• [Reply to W4 - Practical Value of Invariant Convolutions:] The practical value of the proposed Invariant Convolutions is the formalization of convolutional layers that are invariant to group actions specifically designed for time series data, which are incorporated explicitly in the layer design but offering flexibility into learning these properties. Standard neural network architectures, when applied to noisy and non-stationary time series data, often exhibit generalization issues and can be prone to overfitting. This is clearly showcased in the example of the ECG data in Figure 1, where a standard convolution fails to capture relevant features of the signal. Incorporating knowledge on learning invariant features for time series is often done by applying strong augmentations external to the model architecture (for instance, self-supervised contrastive learning forces positive and negative views by employing targeted loss function in training). Yet, the way augmentations are produced increases the computational complexity and makes strong assumptions about the data distribution (mentioned in several contrastive learning time series works). For instance, our method, while not depending on unsupervised pertaining yet, excels when evaluated on different target data (please see Table 4). To the best of our knowledge, this is the first work in the field of time series that mathematically formalizes layers that are invariant to specific group actions with respect to amplitude scaling while providing theoretical guarantees. The proposed layer is incorporated in lightweight (see Figure 3) and shallow modules (see Figure 4), forming the example InvConvNet architecture, which is yet competitive in performance against very complex and deep architectures for time series modeling. The proposed Invariant Convolutions can act as a solid mathematical tool and strong feature extraction module in future works that aim to study robust models for time series analysis.

• [Reply to Q1 - Details on the Transfer Learning Experiment:] We refer the reviewer for [Comment 3] of the General Comments section for additional details/results on the Transfer Learning Experiment. The self-supervised methods are pre-trained and fine-tuned on each source domain and evaluated on the unseen target domains. The supervised method InvConvNet is trained on the source domain (in a supervised way) and evaluated on the unseen target domains. Interestingly, even with the absence of general-purpose representations agnostic to the task, our proposed invariant convolutions are able to learn features transferable to the unseen target domains, which can be attributed to the successful extraction of invariant features through our specifically designed layers.

• [Reply to Q2 - Discussion on Shift Invariance in CNNs:] As noted in [Comment 1] of the General Comments Section, Reviewer WiU6 invited us to compare our contribution with other works [5, 6], which addresses time scale and shift-invariance in deep learning, respectively. Below, we outline the specific differences between these works and our approach. The first paper [5] addresses invariance to the group of time scalings defined as $( t \mapsto at \ | \ a \in ]0, +\infty[ )$. Our work diverges in several ways. First, we address spatiotemporal invariance rather than solely time invariance. Secondly, the greedy approach in [6] approximates time scaling invariance by applying convolutions at multiple logarithmic scales and selecting the optimal scale through max pooling. This results in approximate invariance. In contrast, our mathematical framework provides an exact formulation for invariant convolutions, which can be computed efficiently using the FFT. The second paper [6] deals with shift invariance in image processing. While, as you mentioned, convolutions are inherently equivariant to shifts, achieving shift invariance typically involves ending convolution layers with an average pooling layer, ignoring boundary effects. However, most current architectures insert downsampling layers between convolution layers, which introduces sensitivity to shifts. In [5], the authors address this issue by proposing a downsampling method that maintains shift invariance. Similarly, our proposed invariant convolutions are also equivariant to time shifts, and networks using these convolutions can achieve time-shift invariance depending on their structure. For example, in our experiments, all networks are time-shift invariant because we conclude the convolution layers with average pooling without incorporating downsampling layers.

We truly thank the reviewer for the time devoted to assessing our work and providing interesting comments. Additionally, we are pleased to hear that the reviewer confirms the novelty of the proposed architectural design and the significance of the paper's overall research direction, as well as the presentation of key components of the model architecture and the presented experimental setups. Following the provided comments, we next aim to address any weaknesses and questions raised by the reviewer.

• [Reply to W1 - Relevant Works on Invariant CNNs:] We would like to refer the reviewer to our General Comments section and specifically [Comment 1], where we provide justifications about the positioning of the work with respect to works on invariance in deep learning. In the original draft, we had originally added this discussion in the introductory Section 1, page 2, in the second paragraph. We are willing to change this paragraph if the updated discussion addresses the expressed concerns.

• [Reply to W2 - Math Details vs Architectural Details in Methods Section:] We would like to thank the reviewer for their thorough suggestions and the time invested in providing comments relevant to the structure and presentation of the paper. However, we would like to highlight that the central contribution of our work (also reflected in the title) is the formalism and mathematical design of fast convolutional layers that are invariant to specific group actions targeted to the relatively underexplored field of time series invariance in deep learning. We, therefore, believe that the mathematical details, as also confirmed by many of the other reviewers, remain a significant part of the presented work in terms of potential impact and positioning against the vast abundance of explicit architectural designs for time series tasks. Contrary to this, our layers are general and could be adapted to a time series model as feature extractors to leverage invariances. For this reason, we chose to present examples of architectures that are shallow (see Figure 4) and simple yet achieve robust performances against complex baselines. Due to the page limit, we kept the task-specific architectural details in the appendix yet tried to present key aspects in the last paragraph of the methods (section 3) and the experimental setups described in the main paper. We are pleased to hear the reviewer's opinion on the above arguments about the chosen structure of the methodological section and potentially address specific parts that could potentially enhance the coherence of the architectural design in the paper.

• [Reply to W3 - Significance of Overall Results] We kindly disagree with the reviewer on the point made for the significance of the overall results achieved by the proposed invariant convolutions. This is clearly shown in the classification experiments on UEA and additional datasets (UCIHAR, Epilepsy, Sleep-EDF) with accuracy improvements up to $0.7\%$ compared to the best competitor out of several quite complex and computationally demanding baselines (an example is shown in Figure 3). When adding realistic deformations such as offset and trend to simple UCR datasets, the drop in classification performances of the proposed InvConvNet with mixed convolution types (including invariant ones) remains relatively low compared to standard convolutions (please see Table 3 and [Comment 2] of the General Comments section for additional comments/results). It demonstrates that standard CNNs have difficulties to correctly learn invariant features which corroborates with results observed in [1] for image data. In the transfer learning experiment (please see [Comment 3] of the General Comments section for additional comments/results), when evaluating an unseen target domains, the proposed InvConvnet offers average improvements of around $4\%$ compared to contrastive learning methods that leverage general-purpose self-supervised pretraining. Finally, on the more demanding reconstruction-based task of anomaly detection where the need for scale invariances can be questioned, the proposed InvConvNet architecture remains relatively competitive.

• [Reply to W3 - Inconsistency in the UCR Ablation Study of Table 3:] We refer the reviewer to our answer in [Comment 2] of the General Comments section for comments/additional results. The slightly higher accuracy achieved by InvConvnet-N (built on only normal convolutions) can be attributed to the inherent z-normalization of the employed UCR datasets and the simplicity of the classification. In contrast, when synthetic deformations with respect to offset and linear trend are added, our proposed and mixed InvConvNet achieves better performances than the variants built solely upon one type of convolution.

I would like to thank the authors for their detailed responses and clarifications. While I will avoid reopening a lengthy discussion, I have a few additional remarks to share:

• UCR datasets. I remain unconvinced by the explanation regarding the UCR dataset results, particularly the performance gap between InvConvNet and InvConvNet-N (-normal) in the setting without deformations. Z-normalization is a widely recognized standard technique when applying neural networks to time series classification. Therefore, I would expect the proposed InvConvNet to match the performance of InvConvNet-N (-normal) on the UCR dataset if it is to be considered competitive.
• Transfer learning experiments. If I understand correctly, both contrastive representation methods, TS2Vec and TS-TCC, are trained in an unsupervised manner on the source datasets, and their classification head (an SVM in the case of the original TS2Vec paper) is also trained on the source datasets. The comparison then evaluates the zero-shot performance without retraining the classification head on the target dataset. This approach seems unusual, as it disregards the common practice of training the classification head on the target dataset for transfer learning tasks. This is especially important when the classification head functions as a geometric separator. Typically, in transfer learning experiments for classification (e.g., the T-Loss paper), the unsupervised architecture is trained on a source dataset and used as a universal feature extractor for target datasets, where the classification head is subsequently trained on the target representations.
• Mathematical framework. As noted in my initial review, the general mathematical framework presented in Section 3.1 provides a helpful unification of certain invariant properties that may be desirable in specific time series modeling tasks. However, I remain unconvinced about the general usefulness of hard invariances such as linear trend invariance or amplitude scaling invariance in most time series classification scenarios.
• Discussion of invariance properties in Deep Learning. Thank you for addressing the topic of invariance in deep learning and referencing related works like the "Shift Invariance in CNNs" paper in Computer Vision. In my opinion, this broader discussion provides a valuable context for positioning the paper.

I appreciate the effort in your rebuttal, but for the reasons stated above, I will keep my score the same.

• [Reply to W3 - Novelty Justifications and Relevant Works on Invariant Architectures:] We would like to refer the reviewer to our General Comments section and specifically [Comment 1], where we provide justifications about the positioning of the work with respect to works on invariance in deep learning. In the original draft, we had originally added this discussion in the introductory Section 1, page 2, in the second paragraph. We are willing to change this paragraph if the updated discussion addresses the expressed concerns. Regarding the referenced papers [5, 6], the first one [5] addresses invariance to the group of time scalings defined as $( t \mapsto at \ | \ a \in ]0, +\infty[ )$. Our work diverges in several ways. First, we address spatiotemporal invariance rather than solely time invariance. Secondly, the greedy approach in [6] approximates time scaling invariance by applying convolutions at multiple logarithmic scales and selecting the optimal scale through max pooling. This results in approximate invariance. In contrast, our mathematical framework provides an exact formulation for invariant convolutions, which can be computed efficiently using the FFT. The second paper [6] deals with shift invariance in image processing. While convolutions are inherently equivariant to shifts, achieving shift invariance typically involves ending convolution layers with an average pooling layer, ignoring boundary effects. However, most current architectures insert downsampling layers between convolution layers, which introduces sensitivity to shifts. In [5], the authors address this issue by proposing a downsampling method that maintains shift invariance. Similarly, our proposed invariant convolutions are also equivariant to time shifts, and networks using these convolutions can achieve time-shift invariance depending on their structure. For example, in our experiments, all networks are time-shift invariant because we conclude the convolution layers with average pooling without incorporating downsampling layers.

• [Reply to W4 - Results on the UCR Ablation Study in Table 3:] We thank the reviewer for pointing out this inconsistency, which could indeed benefit from additional justifications. We refer the reviewer to our answer in [Comment 2] of the General Comments section. The slightly higher accuracy achieved by InvConvnet-N (built on only normal convolutions) can be attributed to the inherent z-normalization of the employed UCR datasets [2] and the simplicity of the classification. In contrast, when synthetic deformations with respect to offset and linear trend are added, our proposed mixed InvConvNet achieves better performances than the variant built solely upon normal convolutions.

[1] Kvinge, H., Emerson, T., Jorgenson, G., Vasquez, S., Doster, T., & Lew, J. (2022). In what ways are deep neural networks invariant and how should we measure this?. Advances in Neural Information Processing Systems, 35, 32816-32829.

[2] Dau, H. A., Bagnall, A., Kamgar, K., Yeh, C. C. M., Zhu, Y., Gharghabi, S., ... & Keogh, E. (2019). The UCR time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6), 1293-1305.

[3] Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C. K., Li, X., & Guan, C. (2021). Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112.

[4] Yue, Z., Wang, Y., Duan, J., Yang, T., Huang, C., Tong, Y., & Xu, B. (2022, June). Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 8, pp. 8980-8987).

[5] Jacques, B. G., Tiganj, Z., Sarkar, A., Howard, M., & Sederberg, P. (2022, June). A deep convolutional neural network that is invariant to time rescaling. In International conference on machine learning (pp. 9729-9738). PMLR.

[6] Chaman, A., & Dokmanic, I. (2021). Truly shift-invariant convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3773-3783).

We would like to thank the reviewer for their positive comments concerning the soundness of the mathematical details of the proposed Invariant Convolution, along with the definitions of time series deformations and group actions. We are also pleased to know that the reviewer recognizes the potential significant impact of our contribution to practical scenarios involving time series data, such as the presented ECG example in Figure 1. In the following responses, we aim to address each of the highlighted weaknesses individually and in detail to further support our contribution.

• [Reply to W1 - Fixed vs Learnable Invariant Kernel:] In the introductory ECG example (Figure 1), we deliberately fixed the kernel to a single heartbeat and evaluated its correlation along an ECG signal affected by a trend. This example illustrates that a standard convolution fails to capture the high correlation between heartbeats, whereas a convolution that is locally invariant to linear trends successfully identifies these correlations. As noted in your review, this example effectively clarifies and motivates our work. For the sake of simplicity, we did not explore whether standard CNN layers can learn such invariance in the introduction. However, this question is explicitly addressed in the synthetic experiment (pages 8-9), where we analyze the decline in classification performance as baseline datasets with no distortions are progressively deformed with realistic synthetically generated deformations. The results demonstrate that standard CNNs struggle to learn invariances compared to invariant convolutions, aligning with findings in [1] for image data. A detailed description of the synthetic experiment is provided in [Comment 2] of the General Comments section. In the revised manuscript, we will explicitly acknowledge the limitations of the introductory example and indicate where this question is thoroughly addressed.

• [Reply to W2 - Comparisons to Augmentation-based Contrastive Learning Methods:] We would like to grab the reviewer's attention to the Transfer Learning experiment presented in the relevant paragraph on page 9 (for performance scores, please see Table 4). In this experiment, we directly compare our proposed Invariant Convolutions (InvConvNet method) to two common self-supervised contrastive learning techniques that have shown great success in time series modeling, specifically TS-TCC [3] and TS2Vec [4]. This experiment involves training on a source dataset (left side of the arrow) and testing on a held-out target dataset (right side of the arrow). Self-supervised contrastive learning leverages self-supervised pre-training on data without labels, followed by supervised fine-tuning, which can give a significant advantage to the generalization properties of the backbone architectures. In our work, we carefully designed time series-specific invariant convolutions that can be incorporated into any supervised or unsupervised framework. Our proposed lightweight (see Figure 3) supervised InvConvNet acts as a simple proof of concept for the generalization capabilities of invariant convolutions. To be fair in comparison, since our example InvConvNet is supervised, we leveraged supervised baseline methods for classification and anomaly detection (Tables 1, 2, 3, 5). We applied the self-supervised contrastive methods to the Transfer Learning experiment, where they can show a natural advantage. More specifically, TS-TCC is indeed a transformation-based contrastive method that considers both weak and strong augmentations of the input. Weak augmentations involve adding random variations to the signal and scaling up its magnitude. In contrast, strong ones involve splitting the signal into a random number of segments and shuffling them. TS2Vec method criticizes the commonly used transformation-based consistency (scaling, jittering, permutations) in previous works for making strong assumptions on the data distribution. To overcome this, TS2Vec proposes contextual consistency, which treats representations at the same timestamp in two augmented contexts as positive and considers augmentations that do not change the magnitude of the data, such as timestamp masking and random cropping. Interestingly, those augmentations excel in classification performance in the original TS2Vec paper over common scale transformations, which even deteriorate performance when added to the method. We, therefore, chose two representative contrastive methods built upon different types of augmentations, including scale-level and timestamp-level ones. Our experimental results for the experiment in Table 4 showcase the excellent performance ($4\%$ avg. improvement) of invariant convolutions when transferred to unseen data over self-supervised techniques. We would also like to refer the reviewer for [Comment 3] of the General Comments section for additional details/results on the Transfer Learning Experiment.