Seeded LoRA: Collaborative Fine-Tuning Through Seed Initialization of Adapters

Thank you so much for taking the time to review or work and suggesting improvements. We appreciate it and will try to incorporate as many of them as possible in our final submission. Below, we add responses to some of the questions raised. We hope this additional information provides a complete picture of our work and contributions.

Q1: discuss how CoFT relates to and differs from multi-task/continuous learning, which also aims for improved model performance

A1: CoFT is an interesting middle-point between federated learning and multi-task learning. In multi-task learning one often has access to all the data. In the case of CoFT, you only have the weights of existing models and a single new task/dataset. The inherent difficulty is to do multi-task learning and (1) you only have one dataset + weight, (2) you do not know what the weights have been trained on exactly. The difference to continuous learning is that CoFT is that continuous learning gradually adapts, while in CoFT the main artifacts that are shared are trained models after certain checkpoints. We will add clarification and more context comparing CoFT/our method with continual/multi-task learning approaches.

Q2: In Dataset-Specific Expert fine-tuning, does the seeded LoRA recognize the dataset during training (thereby updating domain-specific LoRA)?

A2: It does not need to recognize the dataset because all LoRA adapters are trained independently. So (1) a seed expert is trained on a specific dataset, (2) at test time all LoRA adapters are merged (including the new dataset-specific expert), (3) this approach yields high quality domain-specific performance without knowing the input domain (no routing required). This stands in contrast to almost all routing work, where the incoming data needs to be recognized for domain-specialization.

Q3: The authors' approach of using the same subspace for initializing task-specific LoRAs opposes O-LoRA's method of using orthogonal subspaces and merging them after continued learning. More discussion and comparison with O-LoRA are needed.

A3: Geometrically, one can have merging without interference if (1) LoRAs are orthogonal, (2) LoRA start from the same weight space and only are optimized sparsely. O-LoRA does (1), Seeded LoRA does (2): Once the optimization landscape has settled (seed training; picture going down an optimization canyon), then most weights do not move anymore while only some move heavily. Depending on the data used, different dimensions move, while most dimensions remain steady. One does not have this property without a seed LoRA, since at the beginning of LoRA finetuning all weights move until they have settled into the local optimization landscape.

Thank you for your clarification.

Regarding Q2, a clearer process is needed.

1. When you said, "all LoRA adapters are trained independently," does it mean that after a seed expert is initialized, for each dataset, a new LoRA model initialized with that seed expert is trained? For instance, for 10 datasets, there are 10 LoRAs trained independently?

2. For the comparison methods, including lora-mixture and MoLoRA-mixture, does it mean that the datasets are merged and fine-tuned on a multi-task dataset?

As the discussion period is nearing its end, we wanted to ask Reviewer c3Qw if there are any follow-up points we can clarify. We have responded to all concerns raised. If there are no further points of clarification regarding the manuscript, experiments, our fine-tuning procedure, and details about the optimization subspace that makes our method possible, we would kindly ask that reviewer c3Qw to consider increasing their score to reflect this.

Thank you so much for taking the time to review or work and suggesting improvements. We appreciate it and will try to incorporate as many of them as possible in our final submission. Below, we add responses to some of the questions raised. We hope this additional information provides a complete picture of our work and contributions.

Q1: Testing across 1-2 additional open-source models that support fine-tuning or instruction tuning could provide a broader and more relevant context for the findings.

A1:We acknowledge that more experiments could add to the robustness of our results, but we lack the computational resources to complete more training runs. Note that we carefully control for data and different methods for many different datasets/setups. Overall, this gives us a large sample size which makes our results already quite robust. We run experiments on a setup that extends the experiments in the MoLoRA work, which was accepted to ICLR2024, so our experiments have a larger diversity than previously established work. The main result is that for the first time we are able to merge models without losing significant performance if the target tasks are diverse.

Q2: When fine-tuning with LoRA (mixture), are you using a single adapter, or does the number of adapters match LoRA (individual)?

A2: For LoRA (mixture) we use a single adapter trained on all data with a parameter budget equivalent to the one used in the other methods. In this case, we’re comparing performance across tasks for a fixed parameter budget, and show that LoRA (mixture) as well as MoLoRA, suffer from task inference.

Q3: I suggest a variant of LoRA (individual) in which all LoRAs are applied simultaneously during fine-tuning on the mixture dataset.

A3: Thanks for the suggestion, we will update table 1 to better showcase results. The difficulty of combining multiple LoRA adapters that are not seeded is what motivated this work in the first place. We experimented with applying all LoRAs simultaneously during fine-tuning. This setting is equivalent to SeededLoRA without a seed expert, and results for it can be found in Appendix H, table 8. From that table we can observe that task inference still happens. The average score for this setting is 50.52, lower than SeededLoRA’s 52.47 average score when using the seed expert,

Q4: isn’t a trained model already positioned within an optimized subspace? How, then, would additional seeding impact performance? Furthermore, wouldn't a new seed be necessary whenever a new dataset or task is added to the mixture?

A4: Previous work shows that the spectrum of the Hessian is spanned by a number of large eigenvectors that approximately equivalent to the number of classes in the training set. However, finetuning sets often have very different data. For example, instruction tuning data is very different from pretraining data. This new data leads to optimizations in parameter space where some parameters that were “dormant” become “active” and as such, the new optimization shifts slightly, but significantly, therefore requiring the seeded approach to let the optimization landscape settle in its new local environment.

Q5: A crucial ablation study would be to evaluate performance on out-of-distribution datasets. [...] This would clarify whether re-seeding is essential for a new dataset of the same task. [..] For a completely new task, however, I would certainly expect that re-seeding would be required, but proving that with an ablation would make the experiment section stronger.

A5: We also asked ourselves this question, but since we are computationally limited, we trusted in the result from Li et al.[1], who showed that training a seed model on javascript still achieved local mode connectivity across diverse, unrelated domains such as medical, law, arxiv, or C4. If the same holds for adapters, our method should work even if all expert-specific datasets are not available at the initial training phase. [1]: https://arxiv.org/abs/2208.03306

Q6: I am uncertain about the purpose of Section 7, as it seems to add little beyond what was already covered in Seeded LoRA

A6: We chose to emphasize this to highlight that SeededLoRA is equivalent to well-established results in the literature (multi-head attention and ResNeXt). Other methods, like the closely related MoLoRA do not have this relationship. As such, our approach sits on a rich stream of literature which makes further study quite fruitful, particularly with multi-head attention being used everywhere, and ResNeXt being still superior to vision transformers in many vision problems.

As the discussion period is nearing its end, we wanted to ask Reviewer ouui if there are any follow-up points we can clarify. We have responded to all concerns raised. If there are no further points of clarification regarding the manuscript, experiments, our fine-tuning procedure, and details about the optimization subspace that makes our method possible, we would kindly ask that reviewer ouui to consider increasing their score to reflect this.

Thank you so much for taking the time to review or work and suggesting improvements. We appreciate it and will try to incorporate as many of them as possible in our final submission. Below, we add responses to some of the questions raised. We hope this additional information provides a complete picture of our work and contributions.

Q1: The work seems to be a simple adoption of the model soup method

A1: The model soup method averages the weights of multiple models fine-tuned with different hyperparameter configurations. Even though we also average the weights of our fine-tuned experts, our methods are different. In SeededLoRA each expert is trained on different data using the same hyperparameters. What we show in this work is that 1) importance of the Seed LoRA before training expert LoRA adapters, where the Seed LoRA trained on the combined data mix provides higher mode connectivity and better merging results 2) Using the Seeded LoRA approach, we can mitigate task inference by using a simple weight average of fine-tuned experts. 3) Our simple method works best and that other work that tries to do routing work might fail exactly because the optimization landscape is not initialized well.

Q2: Experiments seem to be insufficient. For example, the LoRA weights from different experts are averaged arithmetically. Intuitively, for different tasks, the weights of each expert should be different during aggregation. Even without a routing module, a set of learnable weights might be helpful to improve the final inference performance

A2: The main point was that our simple method works best and that other work that tries to do routing work might fail exactly because the optimization landscape is not initialized well. Section 6 shows the failure case of MoLoRA routing, that uses soft-merging (i.e. weighted average using learnable parameters) which is subtle but very important. The difficulty of training a good router that can utilize experts effectively is not specific to MoLoRA. There are many examples in the literature that discuss issues with routing in the Mixture-of-Experts models (SiRA, Zhu et al). Additionally, it has been shown that experts in MoE settings do not tend to specialize on specific tasks (Mixtral of Experts, Q. Jiang et al[1]). While we did not do controlled ablations on different seeds, we did find in our experimentation that seeded LoRA seems to be relatively stable with respect to what seed we use. This is in line with finding in the existing literature (see Branch-train-merge; Li et al.[2]). The advantage of arithmetic average is that it does not require any training (i.e. learnable parameter), therefore provides better flexibility. This is particularly important when a new LoRA expert is trained from the Seed LoRA in which we can just apply averaging without further training. [1]: https://arxiv.org/abs/2401.04088 [2]: https://arxiv.org/abs/2208.03306

Q3: Is it possible to apply the idea of MAML here to find the best seed LoRA weights?

A3: The exact data used for training the Seed Expert is not critical. For example, Li et al [1]. showed that training a seed model on javascript still achieved local mode connectivity across diverse, unrelated domains such as medical, law, arxiv, or C4. Theoretically, this is linked to how the eigenvalues of the Hessian are linked to the optimization (such as optimal learning rate, and local near-linear optimization). Please see Ghorbani et al[2], and Gur-Ari et al[3], for more theoretical details. MAML gives good initialization per task, but here we want to have a good initialization for expert LoRA to be successfully merged. [1]: https://arxiv.org/abs/2401.04088 [2]: https://arxiv.org/abs/1901.10159 [3]: https://arxiv.org/abs/1812.04754

As the discussion period is nearing its end, we wanted to ask Reviewer R2th if there are any follow-up points we can clarify. We have responded to all concerns raised. If there are no further points of clarification regarding the manuscript, our contributions and how our method differs from model soups, our experiments, and how we compare to other approaches (MAML), we would kindly ask that reviewer R2th to consider increasing their score to reflect.