ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale

We thank the reviewer for recognizing the novelty and performance of our method, as well as the clarity of our presentation. We appreciate the insightful feedback and address the reviewer's concerns below.

• [W1] Performance improvement of ScaLearn: We would like to emphasize that the performance of ScaLearn is considerably better than the baselines at a much lower or similar parameter count. While the performance on high-resource tasks (QQP, MNLI) is similar to some other methods (since the source adapter can learn the task well), this is considerably different for many other tasks: This is especially the case on SuperGLUE, as shown in Table 3 - our ScaLearn beats the best baseline by ~2 percentage points (!), and this is especially visible on the low-resource tasks - notably CB (90.89 vs. 87.68 for the best baseline, Compacter++) and RTE (78.88 vs. 76.53 for the best baseline, Adapter-m). In addition, on the very different multilingual benchmark HumSet, ScaLearn is the only (!) method that clearly beats any of the single-task learning methods. We attribute this to the simplicity and power of scaling output representations. Furthermore, we also show considerably improved few-shot learning abilities (Section 6.3).
• [W2] Significance of parameter-efficiency: We agree that our method provides strong parameter-efficiency and that the original backbone model typically has more parameters. However, especially since models tend to get larger and larger, it is important to be able to efficiently adapt the model to novel domains, tasks, and languages and mitigate catastrophic interference. The importance of our problem setup and efficiency is also highlighted by reviewer YAiw. This strongly motivates a modular design of such models. In this paradigm, the parameter-efficiency of PEFT (parameter-efficient fine-tuning) has become paramount, especially when scaling up the number of tasks and, thus, modules and corresponding parameter counts [1]. Therefore, this is a crucial aspect of any method within this paradigm. This is especially the case in two-stage MTL methods - and in this regime, ScaLearn, and especially its variations, is highly parameter-efficient while also clearly outperforming the baselines. For an overview of the parameter-efficiency of ScaLearn and all baselines, we refer the reviewer to Figure 1 on page 1 and the parameter-efficiency analysis (Section 6.1).
• [W3] Experiments beyond NLP tasks: We understand the reviewer's interest in applying our method to other contexts beyond NLP. In this work, our focus is on showcasing its strong performance and efficiency in the context of NLP. In doing that, we follow numerous other papers that equally focus on NLP tasks [2, 3, 4]. We have shown the performance of our method on a diverse number of NLP tasks, domains, and languages. Yet, we thank the reviewer for the idea, and it comprises an interesting direction for future research.
• [Q1] Variability in ScaLearn* performance: Overall, the performance of the different variations of ScaLearn is rather similar. Moreover, the performance of ScaLearn and ScaLearn++ tends to be highly similar on most tasks (cf. Table 2, 3, and 4), with both performing the best overall. The performance of the variants relying on uniform scaling—ScaLearnUniform and ScaLearnUniform++—also tends to be highly competitive on most tasks and is generally a bit lower than the non-uniform variants, but still very strong. We attribute this to the difference in scaling the output representations element-wise or not (i.e., non-uniform or uniform scaling) and the corresponding difference in learning capacity. For the best performance, we advise using the vanilla ScaLearn or ScaLearn++, and if the parameter count is the main constraint, we advise using ScaLearnUniform++.

[1] Modular Deep Learning. arxiv preprint, 2023.

[2] Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. NeurIPS, 2022.

[3] Adapter-Fusion: Non-destructive task composition for transfer learning. EACL, 2021.

[4] Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data. ICLR, 2023.

We thank the reviewer for their thoughtful feedback and for recognizing the technical soundness and thoroughness of the experiments. We appreciate the constructive feedback and address each of the concerns raised in the review below.

• [W1] Focus on adapter-based methods: We acknowledge the reviewer's observation regarding the focus on adapter-based methods. This focus was intentional, following Pfeiffer et al. [1]. Our method is strongly motivated by our analyses of scaling the output representations of adapters (Section 3). Furthermore, we chose to concentrate on adapters due to the wide availability of pre-trained adapters via AdapterHub [2], which facilitates accessibility and reproducibility in the research community. To the best of our knowledge, such sharing platforms are currently not available for other PEFT architectures. However, these are crucial for two-stage MTL methods since they rely on readily available modules (in our case, adapters). Moreover, we justify the use of adapter modules to enable a fair comparison with the other two-stage MTL baseline - AdapterFusion. Additionally, as suggested by Reviewer YAiw, we added AdapterSoup as an additional adapter-based two-stage MTL baseline. In addition, we understand that other PEFT architectures like LoRA and prompt tuning operate differently, and, hence, these methods are not directly compatible with our method, which scales the output representations of the source modules. While this is a valid limitation of our study, we leave the exploration of expanding our method to other PEFT techniques for future work.
• [W3] Use of different benchmarks such as CrossFit [5]: We agree with the importance of diversity in target and source tasks as well as large-scale analyses. To address diversity, we included SuperGLUE and HumSet in addition to the commonly used GLUE benchmark. SuperGLUE presents considerably more challenging tasks than GLUE, which are also more varied in terms of tasks and corpora sizes. We relied on GLUE and SuperGLUE as we wanted to use benchmarks that are most commonly used within the research community. To include a more distinct benchmark, we added additional experiments on the multilingual, multi-class classification benchmark HumSet, which spans the domain of human crisis response, making it very different from both GLUE and SuperGLUE. Using a different backbone model, XLM-R, the similarly strong results on HumSet showcase our method's applicability across different models, languages, and task formulations In addition, we evaluate our method on the combination of all GLUE and SuperGLUE tasks and show that it also outperforms the baselines at a fraction of the parameter cost when considering a higher number of source tasks. This shows that our method also works across more dissimilar tasks and when scaling up the number of source tasks, too. These challenging and different settings demonstrate the robustness and adaptability across different tasks, domains, and languages of our method. Still, we recognize the potential of including large-scale benchmarks like CrossFit in future work to further validate our method under even more different conditions.
• [W4] Comparison to MoEs and overfitting: We appreciate the insightful comment and acknowledge the conceptual similarity between the scaling in our method and Mixture of Experts (MoEs) models. A key difference between our method and MoEs is that, in our setup, the backbone language model remains frozen during both stages of training. This inherently limits the learning capacity compared to fully fine-tuning the entire model, thereby reducing the risk of overfitting—a common issue with more complex models like MoEs. This effect is also observed by Lian et al. [6] when proposing SSF (Scaling & Shifting your Features): "Such improvements in robustness and OOD [out-of-domain] datasets might come from the fact that SSF freezes most of the pre-trained parameters, which maximally preserves the knowledge learned from the large-scale dataset and thus maintains a better generalization ability." Overall, we recognize the conceptual parallels with MoEs. Our approach avoids this phenomenon by keeping the backbone model frozen and, thus, sidestepping issues like overfitting and collapse, ensuring robust and generalizable performance across a variety of tasks.

[1] AdapterFusion: Non-destructive task composition for transfer learning. EACL, 2021.

[2] https://adapterhub.ml/

[3] Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. NeurIPS, 2022.

[4] https://adapterhub.ml/blog/2022/09/updates-in-adapter-transformers-v3-1/#ia3

[5] CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP. EMNLP, 2021

[6] Scaling & shifting your features: A new baseline for efficienmodel tuning. NeurIPS, 2022.

We thank the reviewer for acknowledging the importance, simplicity, and merits of our method. We appreciate the insightful feedback and address each of the concerns below.

• [W1] Experimental section and baselines: We acknowledge the reviewer's emphasis on the importance of a diverse evaluation to demonstrate the merit of our method. We have already included a broad range of strong single-task learning (STL) and multi-task learning (MTL) baselines, and we appreciate the reviewer's perspective on this matter. In response, we have included the PEFT method (IA)^3 [4] and AdapterSoup [2] (adapted to the two-stage MTL framework) in our experiments.

RTable 1: Performance on GLUE using RoBERTa-base.

RTable 2: Performance on SuperGLUE.

RTable 3: Performance on HumSet using XLM-R-base.

RTable 1, 2, and 3 demonstrate that our ScaLearn model outperforms the newly introduced baselines.

Regarding (IA)^3, it performs similarly but slightly worse than other STL methods in our setup. We attribute this difference to our different experimental setup, including tasks, dataset sizes, and PLMs, which vary meaningfully from the original (IA)^3 paper [4]. Aside from the original study, hardly any comparisons of (IA)^3 with other PEFT techniques exist; the closest is [5], which is aligned with our findings.

Regarding AdapterSoup, we adapted it to the two-stage MTL framework by adding the target task corpus and, hence, target adapter to the similarity calculation, and also train a new task head on every target task. Still, its comparatively worse performance suggests that calculating weights using sentence similarity is not the best fit for our specific problem setup.

We incorporated the results of both as an additional baseline in the revised draft. This addition, we believe, further strengthens our evaluation and addresses the reviewer's main concerns, and reflects our commitment to a thorough and inclusive comparative analysis.

[1] Combining parameter-efficient modules for task-level generalisation. EACL, 2023.

[2] AdapterSoup: Weight averaging to improve generalization of pretrained language models. EACL, 2023.

[3] Multi-head adapter routing for data-efficient fine-tuning. arxiv preprint, 2023.

[4] Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. NeurIPS, 2022.

[5] https://adapterhub.ml/blog/2022/09/updates-in-adapter-transformers-v3-1/#ia3

We thank Reviewer Exzr for their positive feedback on our method and for acknowledging the clarity of the paper. We appreciate the reviewer’s positive feedback and constructive suggestions. Below, we address each of the comments and concerns in detail.

• [W1] Selection of source and target tasks: We agree with the importance of diversity in target and source tasks. We included SuperGLUE for this very purpose, as it presents considerably more challenging tasks than GLUE, which are also more varied in terms of tasks and corpora sizes. We relied on GLUE and SuperGLUE as we wanted to use benchmarks that are most commonly used within the research community. To include a more distinct benchmark, we added additional experiments on the multilingual, multi-class classification benchmark HumSet, which spans the domain of human crisis response, making it very different from both GLUE and SuperGLUE. Using a different backbone model, XLM-R, the similarly strong results on HumSet showcase our method’s applicability across different models, languages, and task formulations. In addition, we evaluate our method on the combination of all GLUE and SuperGLUE tasks and show that it equally outperforms the baselines at a fraction of the parameter cost when considering a higher number of source tasks as well. These challenging and different settings demonstrate the robustness and adaptability across different tasks, domains, and languages of our method.
• [W2] Paragraph length and clarity: We thank the reviewer for highlighting the importance of clarity in our paper. In response to the reviewer's valuable feedback, we have thoroughly revised the "Models and baselines" section on page 6 to enhance its readability. Additionally, we have reviewed the entire manuscript to ensure clarity in all sections. We appreciate any further specific feedback on sections that could benefit from increased clarity.
• [Q1] Exploration of other parametrizations: We acknowledge the idea of a broader study encompassing a larger range of parametrizations; however, it would meaningfully deviate from the core contribution of the paper. Our experiments on scaling the output representations of adapters have provided a strong motivation for learning the scaling coefficients to combine different adapter output representations. This has led to our proposed method, ScaLearn, and its variations, showing improved performance and efficiency compared to previous methods.