Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching

We thank the reviewer for the feedback. We are slightly surprised by some of their comments, which we believe are already addressed in the manuscript. We hope that this response will help to clarify any misunderstandings. In line with the reviewer's comments, we have carried out an extensive set of new experiments which we believe address all of the reviewer's concerns. We hope the reviewer will take this into account and consider raising the score accordingly.

[W1, W4] Empirical evaluations are limited / Missing comparisons with recent models.

In line with the reviewer's feedback, we have added comparisons to LRU, S5, and MAMBA on 5 long temporal modeling new datasets, where we find RFormer to perform very competitively. These results are included in Table 5 of the attached PDF to this rebuttal.

The reason we focus on these tasks is because our method is tailored to time-series processing, and other image or text-based benchmark datasets (such as LRA/sMNIST, etc), which typically test model memory, are not as suitable in this setting. We will add a comment along these lines in the paper as well.

In terms of performance on random dropping, we have added experiments on 15 new datasets, comparing against SoTA models for continuous time series processing. We report the performance and average rank of our model in Table 4 (PDF), where our model consistently outperforms the baselines. We find that our model performs very competitively, as well as being orders of magnitude faster than the baselines.

In terms of the small empirical evaluation, we would like to stress that we have now tested our model on a wide variety of datasets. We have also run additional experiments whose results can be found in Tables 1, 2, and 3 of the PDF comparing RFormer with CRU [1]. Additionally, Table 4 (PDF) reports the performance of 8 extra baselines on 15 new datasets with different random dropping rates. We consider this extensive evaluation very much in line with the experiments carried out by other sequence modeling architectures (e.g. NCDE, NRDE, coRNN, noisy RNN). In terms of the tasks considered, we also find that they are consistent with the experiments in these papers, which focused on classification and regression tasks. Nonetheless, we have extended our results into the forecasting scenario through a step-ahead forecasting task, see response to [W2,W3] and the General Response.

[W5] Missing ablation studies on the components of the proposed architecture, particularly the role of the global and local components in the multi-view signature, truncation level $n$, etc.

We would like to clarify that these results are already in the paper. In particular, they can be found in Appendix E: Ablation Studies. We have also carried out a set of experiments on the sensitivity of the model to hyperparameters, which can be found in Figure 2 of the associated PDF document.

[W6] Missing related work.

We would like to clarify that we have a dedicated related work section in Appendix B, where we have already included citations to many of the works that the reviewer mentions. In particular, we include a discussion of Unitary RNNs, Orthogonal RNNs, expRNNs, chronoLSTM, antisymmetric RNNs, Lipschitz RNNs, coRNNs, unicoRNNs, LEMs, waveRNN, Linear Recurrent Units, and Structured State Space Models (S4 and Mamba) on lines L678 - L690. Furthermore, we discuss efficient attention variants such as Sparse Transformer, Longformer, Linear Transformers, BigBird, Performer, and Diffuser in lines L691 - L702. We will add the remaining citations the reviewer suggests.

[W2, W3] The tasks considered are rather simple / Missing evaluations on time series forecasting tasks.

We would like to echo our previous statement on how similar models (NCDE, NRDE, coRNN, noisy RNN) have tested exclusively in classification and regression tasks and how these models have become widely accepted by the community.

In terms of forecasting, we would like to highlight that time series forecasting pipelines using Transformers typically train by masking the input representation iteratively to predict the next time step. In our setting, since we are compressing the input representation, these pipelines cannot be straightforwardly implemented for our model. We intend to work towards a model that uses these principles just for time series forecasting, but note this would require a completely new training pipeline as well as heavy tuning. That said, we reiterate that the focus of this work is to showcase the benefits of using Rough Path Signatures within the attention mechanism for efficient and continuous time-series processing, as demonstrated through the widely accepted experimental settings of time-series classification and regression.

We have, however, included a step-ahead forecasting task on the Apple Limit Order Book volatility to extend our results into te forecasting scenario. The results can be found in the General Response.

[Q2] While the proposed method can improve the modified transformer model in sequence modeling, why would the method be attractive for practitioners when they could use more sophisticated models like SSMs for sequence modeling?

We thank the reviewer for raising this point.

We believe that showing the significant benefits of RFormer when compared to the traditional Transformer is beneficial to the community. There seems to be a wider adoption of hybrid models (SSM + Transformer) by the community given their superior performance [2]. In tasks that require temporal processing, the use of RFormer could potentially make these models have improved inductive biases, allow for continuous-time processing, and increase efficiency and inference speed.

[1] Schirmer, Mona, et al. "Modeling irregular time series with continuous recurrent units." International conference on machine learning. 2022.

[2] Lieber, Opher, et al. "Jamba: A hybrid transformer-mamba language model." arXiv preprint (2024).

We thank the Reviewer for the engaging review and valuable feedback. We believe we have incorporated the reviewer's suggestions into our manuscript and hope our changes warrant an increase in the score.

[W1] Need for stronger empirical analysis.

We have added 19 new datasets, 8 extra baselines, and new synthetic experiments (Tables 4-5/1-3 in the PDF). We hope these additions address the reviewer's concerns and provide a stronger empirical analysis.

[W2.1] Investigation of continuous-time and irregularly-sampled data limited.

Most tasks are indeed regularly sampled, except for the Limit Order Book data. However, we performed random drop experiments in Section 4.3 and Appendix F, expanding Section 4.1 to irregularly sampled settings.

We conducted additional experiments to enhance our model's empirical trustworthiness. For regularly sampled data, our model was tested against SOTA models (S5, Mamba, LRU) across 5 datasets, showing very competitive performance (Table 5 PDF).

For irregularly sampled settings, we compared our model against 4 additional baselines, including [1,3] as suggested by the reviewer, across 15 datasets from the UCR time-series archive. Table 4 (PDF) shows our model's performance and average rank (due to space constraints), consistently outperforming the baselines.

Further, Table 2 (PDF) compares CRU and RFormer in an irregularly sampled synthetic data setting (with shorter sinusoids and a smaller number of classes than the experiments in Section 4.1), supported by CRU's hyperparameters validation in Table 1 and training time analysis in Table 3. These experiments demonstrate that recurrent models perform well with short sequences. Note that despite its superior performance, our model is significantly faster than other continuous-time models, as shown in Appendix F.2 and Table 3 (PDF).

[W2.2] OOM results and efficiency plots.

We thank the reviewer for raising this point. Regarding OOM errors for ContiFormer, Appendix E of the ContiFormer paper demonstrates its memory requirements scale exponentially with sequence length, making it impossible to run on long time-series datasets considered in our paper. We felt this is worth noting because we think ContiFormer is most similar in spirit to RFormer in that it augments the attention mechanism in a continuous-time fashion. We hope you find that with the additional baselines, this point is made clearer.

We want to highlight that efficiency experiments can be already found in the paper in Figure 4 (Section 4.2. Training efficiency), showing seconds per epoch of all models. When a line stops, it indicates an OOM error. Appendix F.2 contains more information on computational times, showing RFormer can be used on context lengths of 250k time steps. We apologize if this was not clear and will modify the text accordingly.

[W2.3] More experiments, particularly for forecasting.

We have significantly increased the number of experiments during this rebuttal, adding 19 new datasets and 8 new baselines (see responses to [W1,2.1]) and new synthetic experiments.

Regarding forecasting, we should remark that forecasting pipelines using Transformers typically involve iteratively masking the input representation to predict the next time step. In our setting, since we compress the input representation, these pipelines cannot be directly applied and would require extensive modifications, which is not feasible during the rebuttal timeframe. However, we include a step-ahead forecasting of the Apple Limit Order Book volatility (see General Response). Further, we point the reviewer to the HR task, which contains classical temporal dynamics (e.g. cyclicality).

We would like to highlight that the main points of our paper on efficiency, inductive bias, and continuous processing should hold with our experiments. Furthermore, we would like to point out that other models in the area (such as [1,3]) have also been applied in the same tasks and have not been tested for forecasting, which requires extensive tuning.

[W2.4] Hyperparameter tuning for baselines.

In short, yes, we did. We thoroughly validated the step and depth parameters used to compute the log-signatures for the NRDE model (Appendix D). We did this as these are the only hyperparameters for which we performed tuning in our model.

For the ODE-RNN and NCDE/NRDE models, we validated the architectural choices. Due to occasional sub-optimal performance in our replications, we included the original results for shared datasets (see Tables 1,3). For ContiFormer, we used hyperparameters from the official repository, testing with both 1 and 4 heads (our model uses 1 head), as detailed in Appendix F.3. For the new datasets, we used optimal hyperparameters from manuscripts/repositories, except for CRU, which we validated ourselves.

Regarding the sensitivity of the model to hyperparameters, we direct the reviewer to Appendix E and have included new sensitivity experiments in Figure 2 of the PDF.

[W3] Improving related work section.

For the final version, we will move the discussion on related work to the main text as suggested and we will contrast our work with other models, highlighting (i) the recurrent vs. attention-based structure, (ii) computational efficiency, and (iii) the treatment of the signature with local vs. global windows in the case of NRDEs.

[Q1] Clarifications.

What we mean here is that the input data should be regularly sampled (to retain decent performance), and each input sequence must be sampled the same number of times. In many practical scenarios (e.g., medical data), time-series may vary wildly in their length. For Transformers, which have a fixed context window, this poses a non-trivial challenge of how to standardize the input data such that it may be encoded by the model. As suggested by reviewer wNkv, this can be improved through padding. We will add some discussion along these lines and tone down some of our wording.

We thank the reviewer for the feedback, as it has been very relevant in updating our manuscript (especially the comment on oversmoothing). We hope that the reviewer will be satisfied with the additional experiments carried out and will be inclined to raise the score.

[W1.1] Motivation of synthetic frequency classification.

The idea of this experiment was to have the relevant information for the classification experiment at the beginning of the time series, as it would have to be propagated forward in order to correctly perform the classification. This is the type of task where we expect traditional recurrent models to fail due to vanishing gradients if the sequence is relatively long (2000 points in this case). This is likely the reason for the poor performance of Neural CDEs, which are a continuous-time analog of RNNs. However, we highlight that this was useful to test if RFormer was able to retain the long-range capabilities of the Transformer despite operating on a very compressed representation. This was further evaluated in a number of very long datasets (most notably EigenWorms), where RFormer was not only the fastest but also the most performant.

If the reviewer would suggest an alternative that might be more suitable in their view, we would be more than happy to run additional experiments for this task.

[W1.2] Clarification on RFormer’s sample efficiency on Sinusoidal vs. Long Sinusoidal datasets.

Upon inspection, we found a bug in our code for this particular experiment, where we were not adding the time information to the dataset before passing it to RFormer (this was only the case in this experiment, due to an incorrect local version of the code). After re-running the experiment, we see similar sample efficiency gains from RFormer in this task for the same hyperparameters. Furthermore, we found that some additional tuning of the signature level in this new setting led to even greater sample efficiency gains (Thank you for this!). The results can be found in Figure 2 in the associated PDF. We have also included an experiment investigating the effect of signature hyperparameters in Figure 2 of the PDF as well.

[W1.3] RFormer's robustness to sampling frequency changes vs. Neural-CDE/RDE.

The theoretical reason behind why RFormer can process irregularly sampled sequences is shown in Proposition 3.1 (L175). In terms of the performance of Neural CDE and Neural RDE, we note that they experience a similar drop for this task as in the other tasks considered. To better understand how our model compares to NRDEs and NCDEs in settings in which these baselines perform well, we ran an additional random dropping on 15 experiments, which are shown in Table 4 of the PDF. We find that RFormer is not only better performing, but also several orders of magnitude faster than the rest of the SoTA benchmarks considered.

[W2] Missing analysis on interpolation methods.

We agree that investigating the impact of different interpolation methods in the multi-view signature computation is interesting. Our choice to use linear interpolation of data was due to the efficient signature computation of paths of this form. This efficiency is due to 1) linear interpolation being a local operation and 2) as noted in the Appendix (L644), the signature of piecewise-linear functions can be computed explicitly in terms of the data points. Continuous interpolations such as splines are non-local, meaning the computational burden is high for long time series. Deriving a convenient, computationally efficient method for computing the signature is also non-trivial. As such, we could consider this approach to be "pareto-optimal", as the reviewer suggests.

Since we focused our experiments on long time series, we decided against investigating this in our paper. However, based on your review, we will add a small section detailing this choice.

[Q1] Large overlaps in representation space.

We thank the reviewer for this excellent question. We have investigated this matter further, and this has led to some very interesting new insights that we feel should be brought to the attention of all the reviewers. For this reason, we have included our response to this question in our General Response.

Additionally, we would also note that the multi-view signature transform is only an instance of a broader idea of presenting the model with information at different scales, which we found to improve performance (see Ablation in Appendix E.1). There are many other ways of providing this multi-scale information (such as restricting the global view to a few previous points rather than all previous points, as suggested by the reviewer), but this would be dataset and task-dependent. In the limit, we hope that these parameters can be tuned with the downstream task loss, but we leave this for future work.

We have included a discussion of our hypothesis of why RFormer achieves substantial efficiency gains in our manuscript, as well as a related citation [1]. We thank the reviewer again for the very helpful comment on oversmoothing.

[Q2] Discussion on input length of MLP and Transformer.

Thank you for bringing this point to our attention. We agree with the reviewer that padding can be used as an ad-hoc solution to guarantee the same sequence length in the batch, even though it has been found that other models (recurrent and convolutional) sometimes struggle with this solution, see [2].

We wanted to highlight this as a weakness of the Transformer model when comparing it to RFormer, which will always give the same representation to the model due to the flexibility of the signature. However, we will tone down our wording in accordance with the reviewer's suggestion.

[1] Rusch, T. Konstantin, et al. "Graph-coupled oscillator networks." International Conference on Machine Learning. PMLR, 2022.

[2] Dwarampudi, Mahidhar et al. "Effects of padding on LSTMs and CNNs." arXiv preprint arXiv:1903.07288 (2019).