Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors

We are glad the reviewer found our work insightful and inline with common practices when training large models, which was part of the motivating factor for our work, and in general appreciate the detailed response and insightful comments and suggestions. The reviewer raised several questions which we address in order:

1. The paper does not seem to take the cost of SPT vs training directly on the downstream task into account, or at least does not make this axis of comparison clear.: This question was raised by all reviewers - please see our unified response above.
2. It is stated in Section 3.2 that sub-par LRA performance is often cited as a prime motivating factor for new methods, but it seems efficiency is often just as important a reason new methods are proposed: We agree that SSMs and transformer variants augmented with SSMs are typically more efficient in terms of compute compared to vanilla transformers and have stated this point on Page 5, para 4.
3. ...it still seems the structured biases are helping and the traditional procedure is somewhat predictive of the ordering. Or is this just an artifact of the experiments? A potential discussion on this point could be useful.: We agree with the reviewer that those indeed help and SPT does not render them redundant. Our goal in section 3.3 was to address this exact issue in a controlled setting, evaluating the structured biases in S4, since a comparison directly with Transformers is difficult due to the large differences between models. In the revised version, we included possible implications of different inductive bias between S4 and Transformers as a possible explanation for the remaining gap, mentioned in the beginning of section 3.3, 1st paragraph of page 6.
4. Table 1 lists many efficient attention methods that were originally evaluated on LRA. It would have been interesting to see a couple of these methods also trained with SPT to confirm empirically that they also do not perform as poorly when using SPT...: The purpose of our work was to re-examine the common evaluation pipeline for popular long range benchmarks & therefore we chose to focus on widely adopted models and how performance gaps between them change when SPT is incorporated. While we agree this is an interesting point, efficient attention methods are not as widely-used as vanilla attention and hence we leave their evaluation as future work.
5. The tasks considered in this paper are standard, but nonetheless the addition of less synthetic or larger scale tasks and experiments would also make the paper more compelling to a broader audience: We agree that to prove the efficiency of SPT in general (rather than only on long sequence tasks) a larger study is in order. In the introduction, 1st paragraph page 2, we cite Krishna et al [1], El-Nouby et al [2] that provide such an analysis on standard NLP and Vision tasks respectively, with further details in the related work section. Along with our work, a large number of tasks and modalities demonstrate the efficacy of SPT. Since we observed training from scratch is widespread when evaluating long range sequence models, we focused on these types of tasks and provide a thorough investigation of SPT and its benefits in that context, and how it leads to a more modern evaluation scheme.
6. No code appears to be included ...: We have added the link to the codebase made available through the anonymized link at the end of the introduction in page 3.
7. In Table 2, was the Transformer + Rotary embeddings trained without SPT? This seems unlikely. Perhaps there is a typo in the description of the Transformer methods in this table?: In Table 2 the first line (Transformer + Rotary) is indeed trained from scratch, the difference between this model and in Table 1 is the use of Rotary embeddings, as well as the larger model size for several tasks.
8. In Table 2, the X "denotes computationally infeasible or unreported results. These are 2 drastically different things and 2 separate symbols should probably be used.: As per your suggestion we have fixed this in the revised version.
9. The methods evaluated for Figure 2 still seem to use complex valued parameterizations even though they are randomly initialized. Is this still necessary when using SPT? Since complex values can be problematic when scaling large scale systems, it would be interesting if SPT also removed the need for this in SSMs/linear RNNs.: We agree with the reviewer this is an interesting experiment, but leave it for future work.

[1] Kundan Krishna, Saurabh Garg, Jeffrey Bigham, and Zachary Lipton. “Downstream datasets make surprisingly good pretraining corpora”.
[2] Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Herv ́e Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training?

We thank the reviewer for the supportive review and address the raised weakness and questions as follows:

1. SPT computational costs relative to initializations of state space models: This question was raised by all reviewers - please see our unified response above.
2. ... on what scale of data SPT ends up resulting in poor initializations that the ones used in state space models : We clarify that the plots in Figure 3 show relative gains offered by SPT over trained from scratch baseline and for performing SPT standard S4 initialization was used. At extremely low data regimes (0.5% of the original data), the absolute model performance after finetuning on the downstream task is low and hence the relative gain offered by SPT is not significant - however it is not negative, i.e., not inferior to the trained from-scratch baseline. 0.5% of the original data amounts to 225 samples for the Image task and 125 samples for the Text task. As a general guidance, when performance on the pretraining task itself approaches random accuracy on the validation set, we would expect SPT not to be beneficial, but even for 125 samples in Text task, pre-training accuracy is ~ 50% which is significantly higher than chance performance for character-level language modeling. We have added the original dataset sizes to the caption of Figure 3.

We are encouraged to hear that the reviewer found our work to be thorough and our method effective. The reviewer raises several important questions that we address in order:

1. how to best combine SPT with supervised fine-tuning: This question was raised by all reviewers - please see our unified response above.
2. results of Transformers and S4 on PathX-256: The question about PathX-256 results from a misuse of notation. We could not fit a single sample on PathX-256 (input length 65K) on our 24GB GPU with Transformer with chunked attention and therefore could not evaluate them at all. We have modified the notation in Table 2 to clarify the difference between setups that are computationally infeasible and unreported results.
3. What is the point of the experiment with Pythia?...: We thank the reviewer for pointing to the lack of clarity in the Pythia section. It is common practice in various areas of ML to take LLMs pretrained on text and directly adapt them to other modalities such as molecular data, tabular data, code and speech - researchers have reported significant benefits from this approach [1, 2, 3]. In the Pythia experiments, we wanted to examine if pretraining on a large text corpus would result in large gains on LRA or whether maintaining the modality between PT and FT tasks is important. We agree that comparing the Pythia results to the available Transformer results makes comparison difficult due to different architecture and model sizes. Therefore, in the revised version, we have added an additional baseline of Pythia with random initialization, denoted as Pythia 70M (Rand Init) and indeed observe benefits from pretraining on text, yet not on all tasks, which shows the importance of SPT leading to to better performance across the entire suite.

[1] Igor Melnyk, Vijil Chenthamarakshan, Pin-Yu Chen, Payel Das, Amit Dhurandhar, Inkit Padhi, Devleena Das, Reprogramming Pretrained Language Models for Antibody Sequence Infilling.
[2] Herzig et al., TAPAS: Weakly Supervised Table Parsing via Pre-training.
[3] Ao et al., SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing.

We appreciate your response on finding our work thorough and informative, specifically the remark on our analysis for conv. kernels between HiPPO and SPT models. The reviewer raises several important questions that we address in order:

1. Self-training may be effective not only across the data scale but also the model size: This observation indeed aligns with our reported results. Our experiments show that original model sizes used in LRA visual tasks (Image, Pathfinder, PathX) are too small for high performance (Table 1) which led us to scale up the model sizes (Table 2), as mentioned in the 1st paragraph of page 5, section 3.2. To further investigate this, we plan on adding a scaling experiment as suggested in the next revision, comparing performance across model sizes for SPT and trained from scratch models.
2. Would SPADE+SPT or MEGA+SPT further improve the performance? Is there a reason this comparison is not included in the paper?: The reported results for hybrid models indeed outperform SPT variants, in our work we focus on transformers and S4 as they seem to be the most widely-used. We note that we did try training MEGA from scratch but were unable to reproduce the reported results.
3. The current results are meaningless since the downstream task datasets are very different from Pythia 70M. I'm not sure we are able to find the right dataset to cover all the tasks for Long Range Arena. What about separating this experiment from Long Range Arena and showing the comparison for another downstream task on the language domain?: The purpose of Pythia experiment is to see if pretraining on text is helpful on LRA, as has been observed on various modalities such as molecular data, tabular data, code and speech [1,2,3]. The issue of finding a unifying dataset that can be used for pretraining across modalities seen in LRA is indeed difficult, as mentioned by the reviewer, and we view SPT as a remedy for it. For text-based tasks specifically, the question of evaluating the effectiveness of SPT vs. pretraining on large text corpora is discussed thoroughly in Krishna et. al. [4], showing that in many cases SPT rivals PT on large corpora as mentioned both in the 1st paragraph of page 2, and the related work section. Due to their comprehensive analysis, we focused on SPT in the context of evaluating models on long-range benchmarks. Nonetheless, for a more reliable comparison we have added an additional baseline of Pythia with random initialization in the revision, denoted as Pythia 70M (Rand Init), and expanded the discussion in section 3.5 following these addition. The randomly initialized baseline from which clearly shows the benefits of text pretraining are more easily observed. Yet in some cases, the pretrained model fails to significantly outperform the randomly initialized model, unlike SPT, which provides benefits across the board.
4. Are S4/Transformers trained with the same number of epochs as S4+SPT/Transformers+SPT: This question was raised by all reviewers - please see our unified response above.

[1] Igor Melnyk, Vijil Chenthamarakshan, Pin-Yu Chen, Payel Das, Amit Dhurandhar, Inkit Padhi, Devleena Das, Reprogramming Pretrained Language Models for Antibody Sequence Infilling.
[2] Herzig et al., TAPAS: Weakly Supervised Table Parsing via Pre-training.
[3] Ao et al., SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing.
[4] Kundan Krishna, Saurabh Garg, Jeffrey Bigham, and Zachary Lipton. “Downstream datasets make surprisingly good pretraining corpora”.