SpectR: Dynamically Composing LM Experts with Spectral Routing

Thank you for taking the time to review and for considering our discussion of your concerns.

1. The SVD itself is of little concern as it computed once offline using efficient approaches (line 160). The SVD aligns the LoRA adapter, but it does not change its memory footprint or individual compute requirements. The same SVD is required for our Arrow baseline.

2. This paper focuses on demonstrating that SpectR solves shortcomings in the rank-1 prototypes of Arrow routing using a similar experimental setting. Our future work involves large LoRA libraries, but this was not the focus of the Arrow approach we improve upon in this work.

3. We conduct experiments across 36 combinations of models and datasets, but agree that more models and datasets would be interesting to explore, especially across different modalities.

4. We devote Section 5 to discussing trade-offs, including quantifying the storage and VRAM costs as a function of the number of experts.

Thank you for taking the time to review and for considering our response to your questions and concerns.

Addressing concerns:

1. The rank required for a LoRA expert is independent of Arrow or SpectR. LoRA trades off memory for performance. Task performance generally increases with rank, but the parameter efficiency decreases. Given a set of LoRA adapters, we show that SpectR is more effective than Arrow even at smaller ranks, and the relative gain from SpectR increases even more as rank increases (Figure 6). It is common for practitioners to train LoRAs with ranks even higher than what is explored in this paper.

2. We compare Arrow to SpectR across 4 models and 9 datasets. We separate the routing performance by task in Table 1. In general, the performance of our models would improve with higher rank, as the models have more capacity to learn. Our experiments show that at a given rank SpectR will outperform (Table 2) and SpectR’s effectiveness increases with rank (Figure 6). Using lower ranks and higher top-k would decrease performance on two fronts: the lower rank would reduce model capacity to learn new tasks and the higher k would increase interference between adapters (line 92+).

Addressing questions:

1. We evaluate the model on tasks associated with the trained adapters, but we do not give the model information about which task a query belongs. We discuss several related works demonstrating that merging small numbers of LoRAs can lead to good performance in and out of domain (Section 2.2) and focus our work on training-free approaches to routing.

2. Different aspects of LoRA (including rank) have been heavily discussed in other works. We hope our previous comments have mitigated this concern for you.

Thank you for the detailed review. We attempt to mitigate your stated concerns and answer your questions below.

1. Scope and Generalizability

We discuss existing work showing that alternative finetuning strategies can be converted into LoRAs using the difference in weights and state that we focus on only traditional LoRA models (Line 83). However, we can make it clear in the abstract that the experiments use only LoRA-based experts which make up a significant share of expert fine-tunes in model repositories such as huggingface.

2. Theoretical Foundation

We ground our experimental findings in the spectral theory covered in Section 2.4, namely that the right singular vectors (those used by SpectR) are the “orthogonal directions of maximum variation induced by the LoRA expert in the space of input vectors”. Unlike Arrow, “we capture all covariance structure directly in the new adapter representation” (165) which intuitively leads to better performance as seen by the experimental results.

3. Improvement

Our improvement over Arrow routing is statistically significant with a paired t-test on Table 2 producing p=0.036. While μ-routing is a strong baseline for a smaller number of tasks, existing work shows that as the number of tasks grow, interference becomes a significant problem (line 43). With only 9 tasks we still outperform μ-routing on average across all four models, although the gain is less significant.

Answers to Questions

1. The top-k selection distributes higher weights to higher scores by giving zero weight to the lowest scoring experts. We did explore softmax vs uniform weighting of the selection and found the results to be very similar. The vector norms are discriminative but their variance is small, leading to softmax outputs that are essentially uniform already. We originally chose a uniform weighting to encourage a more stable distribution of experts across layers while also aware of the previous work showing uniform merges work well with smaller k. This ended up not being a major factor, as the routing is similar either way. We can include more discussion of this in an appendix, but the main body details the chosen approach which was used for all our experiments.

2. Our baselines effectively ablate sparse vs. dense (mu) routing and rank-1 (Arrow) vs full rank prototypes. Our problem statement is specific to training-free approaches which isn’t applicable to the referenced paper which trains models to produce larger activations for the correct expert.

3. All adapters are initialized with the same weights. We will add a clarifying sentence to our LoRA training details.

4. We choose top-4 following the original Arrow paper to make comparisons with the previous work easier. The value of k could be tuned with out-of-sample data but this would break our training-free goal. Previous work has shown that smaller k merges are better due to increased interference at larger k. We do not experiment with a larger number of tasks in this work, but have positive results in our follow-up work.