Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead

Thank you for the review. Please see our responses below.

---

1. Shared Basis and Clustering Approach

Our method relies on a shared basis for each cluster. Although this approach implicitly assumes some similarity among LoRA adapters, the clustering strategy is designed to work effectively even when reconstruction errors (as measured in the L2 sense) are high. This is because the merging effect within clusters—akin to averaging weights—preserves high performance, as evidenced by our experiments on 1000 LoRAs across multiple languages.

2. Evaluation on Additional Model Architectures

While our current evaluation is performed on a Mistral model, prior work (including the original LoRA: Low-Rank Adaptation of Large Language Models paper) shows that LoRA operates similarly across various transformer-based architectures. Nonetheless, we can run experiments on an additional LLM before the camera-ready version to further substantiate the generalizability of our approach.

3. Incorporating New LoRA Adapters

Adding new LoRA adapters to an already compressed set can be challenging because the trained up and down projections are tailored to the existing adapters. Our recommendation is to rerun the joint compression algorithm when new LoRAs are introduced. As we mention in Section 6.5, this can be managed via batched cron jobs (e.g., on a daily schedule), where new LoRAs are initially compressed as individual clusters and then incorporated into the overall joint compression process.

4. Routing and Selection of LoRA Adapters

The deployment framework is designed such that the server maintains the full collection of LoRA adapters. Each request includes the identifier of the specific LoRA to be used, thereby eliminating the need for an additional classifier to route inputs. This straightforward mechanism ensures that the correct adapter is selected for each task.

---

We hope these clarifications address your concerns regarding the reliance on a shared basis, the evaluation on a single model, the integration of new LoRA adapters, and the routing mechanism for adapter selection.

Thank you for the review. Please see our responses below.

---

1. Deciding the Optimal Number of Clusters

Please see Section 6.5 for hyperparameter recommendations. Determining the optimal number of clusters does require some hyperparameter tuning; however, our experiments indicate that the method is not highly sensitive to this parameter. In Appendix G, we outline a practical tuning procedure that leverages the L2 reconstruction error from a single LoRA module to guide the selection of an appropriate number of clusters for the entire collection. This approach simplifies the tuning process and ensures robust performance.

2. Impact of LoRA Model Distribution on Joint Compression

The motivation for introducing the clustering strategy was based on the observation that joint compression is considerably more sensitive to variations in the LoRA models when performed directly. In our experiments—covering 1000 LoRAs across diverse tasks and languages—we found that the inherent variability of the LoRA models can negatively impact compression effectiveness. By clustering the LoRAs, the impact of this diversity is significantly reduced, thereby stabilizing and enhancing the performance of the joint compression.

---

We hope these clarifications address your concerns regarding the optimal clustering strategy and its role in mitigating the effects of LoRA diversity. We appreciate your insightful feedback.

Thank you for the review. Please see our responses below.

---

1. Throughput Comparison and SOTA s-LoRA

The vLLM multi-LoRA baseline in our experiments already incorporates advanced optimizations such as efficient scheduling and non-blocking CPU-GPU communication when swapping LoRAs, as well as techniques from S-LoRA. Consequently, the throughput comparisons in Figure 4 inherently reflect these state-of-the-art improvements without the need for an additional baseline comparison to S-LoRA.

2. Evaluation Hardware Platform

Experiments were conducted on an H100 80GB GPU with memory consumption capped at 40%. This setting is intended to simulate scenarios where service providers might serve many LoRAs on more economical hardware with lower memory capacity than high-end GPUs.

3. Compression Overhead

In Figure 5 of the Appendix, we present detailed measurements of memory load, data transfer time, and forward-pass performance for both standard LoRA and our joint diagonalization approach (JD-LoRA). This analysis provides a clear view of the compression overhead and its impact on overall performance. Please note that the whole compression process can be done on CPU without interfering with serving of the LLMs. E.g. new LoRAs are temporarily compressed using SVD individually, and then a cron job, say, runs once a day, to compress the full set of LoRAs using joint compression.

4. Scalability to 100,000+ LoRAs

While our current work scales to 1000+ LoRAs—currently the world’s largest open collection of LoRAs with documented training parameters—we believe that scaling to 100,000+ LoRAs is feasible in principle by scaling the number of clusters in our algorithm.

5. Hyperparameter Tuning

Please see Section 6.5 for hyperparameter tuning. Although choosing the optimal rank and number of clusters requires some hyperparameter tuning, our experiments indicate that the method is not overly sensitive to these choices. In Appendix G, we describe a practical tuning procedure that leverages the reconstruction error (in the L2 sense) from a single LoRA module to efficiently determine appropriate hyperparameters for a collection of 1000 LoRAs, thereby reducing deployment complexity.

---

We hope these clarifications address your concerns regarding throughput, hardware, compression overhead, scalability, and hyperparameter tuning. We appreciate your feedback and believe these additions significantly strengthen our work.

Thank you for the review. Please see our responses below.

---

1. Clustering Basis: Weight Structure vs. Task Similarity

Our approach relies solely on the LoRA weights, meaning that the clustering is driven by the intrinsic weight structure rather than an explicit measure of task similarity. While we recognize that the link between weight structure and task similarity is an area ripe for further exploration, our focus is on how these weight patterns facilitate compression.

2. Theoretical Analysis of Clustering Behavior

Our theoretical section argues that a tight clustering of LoRAs is not a prerequisite for success. Indeed we expect (and observe) very large reconstruction errors (in the L2 sense) for the LoRA weight matrices.

Instead, we argue that in addition to simple weight matrix recovery, there is also a very important effect of our approach more akin to merging of LoRAs by averaging their weights (see the later parts of the theoretical section). We believe this explains why the observed LLM outputs do not degrade at all until the 2-norm reconstruction error becomes very large indeed (see Figure 3). This is intuitive given the inner product structure of the attention mechanism in transformers.

Nonetheless, as demonstrated in the merging literature, combining too many LoRAs eventually leads to degradation. This insight motivated our strategy of performing joint diagonalization on smaller, independent clusters. Another indication for the diversity is the low results of TIES merging (Appendix H.3), which might have succeeded otherwise.

3. Multi-Domain Clustering and Task Diversity

Our experiments include LoRAs trained across multiple domains and languages, which reinforces the importance of robust clustering in multi-domain settings. The diversity of our task set is further underscored by Table 3 in the Appendix, which details all 1000 tasks sourced from Super-Natural Instructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. Despite the high reconstruction errors—indicating that the LoRAs are not overly similar—the clustering process effectively exploits the underlying structure, leading to improved compression efficiency.

---

We hope these clarifications adequately address your concerns regarding the clustering rationale, the theoretical underpinnings of our approach, and the diversity of our task set. We appreciate your feedback and are open to any further suggestions.

Thank you for the review. Please see our responses below.

---

1. LoRA Adapter Diversity and Task Coverage

Please see Table 3 in the Appendix, which lists all 1000 tasks drawn from Super-Natural Instructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. This dataset covers a wide range of tasks, including multiple languages, and provides detailed insight into our data collection protocol.

Moreover, the relatively high reconstruction error observed in our experiments indicates that the LoRA adapters are not overly similar. Another indication for the diversity is the low results of TIES merging (Appendix H.3), which might have succeeded otherwise. The significant gains from clustering further suggest a meaningful clustered structure among tasks.

In response to your suggestion, we will add an analysis of cosine similarity between LoRA adapters prior to compression. If you have specific similarity thresholds or metrics in mind, we welcome your recommendations.

2. Evaluation on Additional Model Architectures

We trained an unprecedented number of LoRA adapters on the Mistral-7B-Instruct-v0 model to rigorously validate our approach.

While we acknowledge that additional experiments on more architectures could provide further insights, our current evaluation robustly supports our claims, and we hope you consider this work as a meaningful step forward. Still, we aim to extend our evaluation to include results from another model architecture, which further demonstrates the generalizability and practical utility of our method.

3. Analysis of Clustering Parameter Sensitivity

In Section G of the Appendix and Figure 6, we study how the number of clusters $K$ varies with the number of LoRA adapters $N$ (with experiments conducted for $N=100$ and $N=500$). This analysis helps clarify the sensitivity of the algorithm’s performance with respect to different values of $K$. Further, see Section 6.5 about selecting hyperparameters.

4. Comparative Evaluation and Extensions to Other Methods

In addition to the original uncompressed LoRAs and the SVD-based compression baseline, we compare to Ties-Merging in Appendix H.3 (see Table 7). Regarding potential extensions, we acknowledge that other parameter-efficient fine-tuning methods such as prompt-tuning might also benefit from exploiting shared structures across tasks. We agree that exploring this possibility is an interesting direction for future work.

---

We hope that these clarifications and our planned experiments sufficiently address your concerns.

We thank the reviewer for recognizing the novelty of our method and its throughput gains over naive solutions.

We emphasize that our joint compression approach effectively addresses the GPU memory constraints and the overhead associated with loading and unloading LoRA adapters and is accompanied by both theoretical and empirical validation.

If there is any additional clarification or experiment we can provide to further strengthen your confidence in our approach, please let us know. Otherwise, we would greatly appreciate it if you could consider revising your score upward based on our response.

Thank you again for your valuable feedback.