LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades

Q1: What are the statistics of the training dataset?

A1: Thank you for your question. To ensure reproducibility, we use the same training and evaluation dataset as LLM-Adapters, which has over 1.2k stars on GitHub and has been widely adopted in published studies. We will include a detailed description of the dataset statistics in the experimental setup section for clarity and completeness.

---

Q2: What is the training dataset for lightweight fine-tuning?

A2: Thank you for your valuable feedback. The lightweight fine-tuning dataset is currently a randomly selected subset of the dataset used for full-scale LoRA training. We will clarify this in the revised manuscript.

---

Q3: Could you present the training curve of the lightweigh for LoRA (100), LoRASuite with LFT (100), and LoRA (10k)?

A3: Thank you for your valuable feedback. As we cannot include additional figures during the rebuttal period, we instead provide the step-wise training loss for LoRA (100) and LoRASuite with LFT (100) under the settings of batch size = 16 and micro batch size = 4.

---

Q4: Could you show the training parameters size for LoRA (100), LoRASuite w LFT (100), and LoRA (10k)?

A4: Thank you for your valuable feedback. Under the default setting with rank 32 for MiniCPM-2B, the number of trainable parameters for LoRA (100), LoRASuite with LFT (100), and LoRA (10k) is the same—23.59M, which is approximately 0.78% of all model parameters. We will clarify this in the revised manuscript.

---

Q5: It is a little weird why LoRASuite w/o LFT and base model get almost the same results. For example, since you use the LoRA adapters from MiniCPM-S-1B. For MiniCPM-2B, the LoRASuite w/o LFT get (23.96) compared to the base (23.85).

A5: Thank you for your valuable feedback. As noted in Line 268, we believe this phenomenon occurs because the transformed LoRA, when relying solely on matrix multiplication without backpropagation, has limited compatibility with the new model. This is why a small-scale fine-tuning step is necessary. The significant performance improvement achieved after this fine-tuning further supports our assumption.

---

Q6: Although LoRASuite requires small-scale fine-tuning, this results in a modest memory reduction of 5.5GB, primarily due to the smaller sample size" Why can a smaller sample size reduce memory?

A6: Thank you for your valuable feedback. While the core memory footprint of the model—such as parameters, gradients, and optimizer states—is independent of the dataset size, several other factors contribute to increased memory usage when the dataset is larger.

First, data pipeline overhead and auxiliary states (e.g., DataLoader prefetching, Trainer’s internal buffers, dataset shuffling, and caching) scale with both the number of samples and training steps. Second, if the full dataset is loaded into memory (e.g., using HuggingFace Datasets load_dataset(...) or pre-tokenized and stored as a list), the total number of samples directly affects host DRAM usage. Lastly, for variable-length inputs, larger datasets increase the likelihood of batches containing extremely long sequences, which can trigger excessive padding and dynamic memory allocation, resulting in transient memory spikes.

Summary of Our Rebuttal

• Q1 & Q2 – Data Details: We use the same LLM-Adapters dataset (≈1.2K GitHub stars) that’s widely adopted in the literature. The “lightweight” fine-tuning (LFT) dataset is a random subset of the full LoRA training data. We will clarify this selection process in the revised manuscript.

• Q3 – Training Curves: While we can’t add figures now, we’ve provided step-wise training losses for LoRA (100) vs. LoRASuite + LFT (100) under batch_size=16, micro_batch=4, demonstrating faster convergence for LoRASuite.

• Q4 – Parameter Counts: With rank = 32 on MiniCPM-2B, all three methods—LoRA (100), LoRASuite + LFT (100), and LoRA (10 K)—train 23.59 M parameters (~0.78% of model size). We will state this explicitly.

• Q5 – LoRASuite w/o LFT vs. Base Model: Transformed LoRA without fine-tuning relies solely on matrix multiplication, limiting compatibility with the upgraded model. The notable gains after a small LFT step confirm this assumption. Future research could investigate strategies to eliminate this LFT step without compromising performance.

• Q6 – Memory Reduction with Smaller Samples: Although core footprints (params, grads, optimizer states) are fixed, larger datasets increase overhead in data loading, buffering, caching, and padding for variable-length inputs, causing transient memory spikes. Reducing sample size mitigates these factors.

---

We appreciate your detailed feedback. Are there any other questions or concerns—and, if our responses address your points, would you kindly consider updating your final rating?

Q1& Q2: Although the paper proposes an interesting method for transferring LoRA adapters, it is primarily evaluated on small to medium-scale models (up to 9B parameters), whereas its practical value would be much greater for larger models (e.g., 13B to 70B+), where adapter retraining is significantly more resource-intensive. This leaves the impact on truly large-scale LLMs unverified. The selected models used for evaluation are relatively dated, for example, Llama-3 to Llama-3.1/3.2 and Qwen1.5 to Qwen2/2.5/3. As a result, it remains unclear whether the proposed method can bring notable improvements when applied to more recent or stronger model baselines.

Thank you for your feedback. Our initial motivation was primarily aimed at mobile scenarios, where an on-device foundation model supports multiple application-specific LoRA adapters for different downstream tasks (as detailed in AICore). Thus, we initially focused on small to medium-scale models. We acknowledge your valuable point that applying LoRASuite to larger-scale models (e.g., 13B to 70B+ parameters) would significantly enhance its practical value, especially given the higher resource costs of retraining adapters at this scale. Due to time constraints, we will include additional experiments with more recent and larger models in the revised manuscript to address this concern and highlight the broader applicability of our approach.

---

Q3: The main advantage of the proposed method, saving compute and improving performance in low-data regimes, is only demonstrated when the fine-tuning dataset is extremely small (e.g., 100 samples), and the benefit diminishes or disappears as the dataset size increases, according to results in Section 4.2. Furthermore, the paper does not describe how the small fine-tuning datasets are selected, nor does it report results averaged over multiple random draws. This omission is problematic given the potential instability and variance in performance when using such small data subsets.

Thank you for your constructive feedback. Since the converted LoRA parameters already encapsulate the knowledge learned by the original model, the purpose of small-scale fine-tuning is merely to help the parameters adapt to the upgraded model. As the dataset size increases, the additional fine-tuning can easily lead to overfitting, which explains the diminishing benefit. Currently, the small fine-tuning datasets are randomly selected. We will explicitly describe the dataset selection process in the revised manuscript.

---

Q4: line 292: what does "full-scale LoRA retraining" mean? Is it the same as "LoRA (10k)" in Table 4/5? Same for "LoRA (small)". Please define it in the paper explicitly and keep it consistent throughout the paper.

Thank you for your feedback. "full-scale LoRA retraining" refers to the scenario labeled as "LoRA (10k)" in Tables 4 and 5. Additionally, as described in Line 293, LoRASuite involves a small-scale fine-tuning step to mitigate potential performance degradation caused by direct matrix multiplication; thus, "LoRA (small)" corresponds to the same small-scale fine-tuning labeled as "LoRA (100)" in Tables 4 and 5 when upgrading from MiniCPM-S-1B to MiniCPM-2B. We will explicitly define these terms and ensure consistency throughout the revised manuscript.

---

Q5: Why did you choose to upgrade Qwen1.5-1.8B to Qwen2.5-3B, while skipping Qwen2-1.5B? It could be interesting to see a comparative study on the performance of LoRASuite when upgrading to Qwen2-1.5B vs. Qwen2.5-3B, as the former is a more direct upgrade in terms of model size and architecture.

Thank you for this insightful suggestion. We initially chose the upgrade path from Qwen1.5-1.8B to Qwen2.5-3B to highlight LoRASuite’s effectiveness across intermediate size, layer count, and attention heads. However, we agree that evaluating a more direct upgrade to Qwen2-1.5B offers valuable insights into LoRASuite’s performance under incremental changes. The following table presents the performance on math tasks when upgrading from Qwen1.5-1.8B to Qwen2-1.5B, further confirming the effectiveness of LoRASuite in this setting. We will include this comparative study in the revised manuscript.

Summary of Our Rebuttal

• Model Scope (Q1 & Q2): We initially focused on on-device scenarios—hence, evaluations on small-to-medium models (up to 9B). As requested, we rapidly completed the Qwen2-1.5B adaptation. Owing to time constraints, migrating the 70B model wasn’t feasible during the rebuttal phase, but we will include it in the camera-ready version. In any case, this does not diminish our paper’s core contribution—the first LoRA adaptation methods during LLM upgrade with competitive performance.

• Low-Data Regime Behavior (Q3): The converted LoRA parameters already carry the original model’s knowledge, so only a small fine-tuning step is needed. As dataset size grows, overfitting can reduce gains, which explains the diminishing returns. We will (1) detail how we randomly select the small fine-tuning subsets and (2) report results averaged over multiple random draws to capture variance.

• Terminology Consistency (Q4): “Full-scale LoRA retraining” corresponds to “LoRA (10k)” in Tables 4–5, and “LoRA (small)” corresponds to “LoRA (100).” We will define these terms explicitly in the manuscript and ensure consistent usage throughout.

---

We appreciate your insightful feedback. If there are any remaining questions or concerns, please let us know—and, if you feel our revisions address your points, would you kindly consider updating your final rating?

Q1: The LoRASuite is an expensive method (in the order of cubic) which might limit the applicability of this method.

A1: Thank you for your valuable comment. While LoRASuite involves algorithms with cubic complexity, the practical cost remains manageable. This is because the number of layers and attention heads in mainstream LLMs typically remains under 100, keeping the actual computation well within the capabilities of modern processors.

---

Q2: It has not been clearly explained why they used different methods for head and layer mapping.

A2: Thank you for your thoughtful comment. We use different methods for layer and head mapping based on the distinct structural roles they play. For layer mapping, we adopt a dynamic programming approach inspired by prior studies in image classification with CNNs, where different layers capture different levels of abstraction, and higher layers build upon the representations of lower layers. This motivates a sequential alignment strategy. In contrast, attention heads within the same layer are parallel and independent, so we treat head mapping as a bipartite matching problem and use the Hungarian algorithm to find the optimal one-to-one correspondence. We will clarify this design choice in the revised manuscript.

---

Q3: As shown in table 4 and 5, the performance of LoRASuite is not consistent across different datasets.

A3: Thank you for your observation. The performance variation of LoRASuite across different datasets can be attributed not only to the task differences but also to the underlying pretraining data of both the original and upgraded models. If the pretraining corpora differ significantly, the representations learned by the models may diverge, which can affect the compatibility and effectiveness of the transferred LoRA weights. We will clarify this important factor in the revised manuscript.

---

Q4: Their method doesn’t seem to be applicable to customized architectures.

A4: Thank you for your feedback. Our current evaluations focus on mainstream LLMs to ensure broad applicability and reproducibility. However, LoRASuite is also applicable to customized architectures, as long as the original and upgraded models share a similar structural design—for example, when both follow the Transformer paradigm. In such cases, the core idea of layer mapping and attention head mapping remains valid.

---

Q5: Why in figure 4-b, the change of learning rate has a significant impact on the performance of LoRASuite? What’s the justification?

A5: Thank you for your insightful feedback. Unlike LoRA (Small), which employs a linear learning-rate scheduler with a warm-up stage to stabilize optimization from random initialization, LoRASuite starts with transformed parameters that are already close to a good solution. Hence, we omit the warm-up phase. However, this initialization results in a steeper optimization landscape, where a small learning rate combined with limited data can cause the model to underfit or fail to converge effectively. We will clarify this in the revised manuscript.

---

Q6: Why in Algorithm 2, the input to the cka_layer_mapping() is S? S has not been introduced properly.

A6: Thank you for your feedback. As described in Line 175, the matrix $S$ is defined such that each element $S_{ij}$ represents the CKA similarity between the $i$-th layer of the original model and the $j$-th layer of the upgraded model. We will clarify the introduction of $S$ in Algorithm 2 in our revised manuscript.

Thanks for the clarification. That said, I’m still unsure why LoRASuite without Head Mapping is considered equivalent to the approach I described earlier. Could you clarify this reasoning in more detail?

It seems to me that the papers use overly complex language, which ends up hiding some of the most important points.

Concerning the SVD cost, if the overhead from layer and head mapping becomes insignificant compared to the dominant runtime of the SVD itself, doesn’t that render the current complexity analysis somewhat misleading?

# LoRASuite without Head Mapping
new_delta_W_Q = W_x.T @ delta_W_Q @ W_x 

# Your proposed approach
new_delta_W_Q = W_x.T @ (W_old_Q + delta_W_Q) @ W_x - W_new_Q

Q1: Why is LoRASuite much more sensitive to the learning rate compared to vanilla LoRA (fig4-b)?

A1: Thank you for your insightful feedback. Unlike LoRA (Small), which employs a linear learning-rate scheduler with a warm-up stage to stabilize optimization from random initialization, LoRASuite starts with transformed parameters that are already close to a good solution. Hence, we omit the warm-up phase. However, this initialization results in a steeper optimization landscape, where a small learning rate combined with limited data can cause the model to underfit or fail to converge effectively. We will clarify this in the revised manuscript.

---

Q2: How do you set the maximum offset constraint in Algo 1?.

A2: Thank you for your feedback. The maximum offset constraint in Algorithm 1 is set to the absolute difference in layer counts between the original and upgraded models. We will clarify this detail in the revised manuscript.

---

Q3: What happens if the optimal head mapping is not one-to-one, i.e., the information could be distributed across or aggregated from multiple heads)?

A3: Thank you for your insightful feedback. Indeed, as you pointed out, information being distributed across or aggregated from multiple heads is a valid scenario. In such cases, one would need to compute similarity scores between all attention heads across every layer of the original and upgraded models, distribute information from each attention head in the original model to multiple heads in the upgraded model, and aggregate these mappings accordingly. However, this comprehensive method would have a time complexity of approximately $O((n_{layer} \times n_{head} )^3)$ due to the cubic complexity of the Hungarian algorithm. Hence, our current method can be viewed as a practical simplification to mitigate computational overhead.

---

Q4: What happens if the tokenizer or embedding changes significantly?

A4: Thank you for this valuable question. As described in Line 130, we manually calculate the transformation matrix using shared tokens between models. However, if the tokenizer or embedding changes significantly, additional alignment techniques—such as supervised or unsupervised embedding alignment methods [1], or entity alignment methods from graph domains [2]—would be required before applying our proposed method. Investigating these alignment strategies is an interesting direction for future research, and we will clarify this in our revised manuscript.

[1] (EMNLP’2018) Gromov-Wasserstein Alignment of Word Embedding Spaces.

[2] (NIPS’24) Entity Alignment with Noisy Annotations from Large Language Models.

---

Q5: What could be some potential solutions for implicit upgrades, such as changes in pre-training datasets and post-training methods?

A5: Thank you for your insightful feedback. For implicit upgrades involving changes in pre-training datasets or post-training methods, prior studies on knowledge editing [1, 2, 3, 4] provide promising directions. Integrating knowledge-editing techniques with our proposed methods would be an interesting area for future research. We will clarify this in our revised manuscript.

[1] (NIPS’22) Locating and editing factual associations in gpt.

[2] (ICLR’23) Massediting memory in a transformer.

[3] (NIPS’24) WISE: Rethinking the knowledge memory for lifelong model editing of large language models.

[4] (NIPS’24) Knowledge circuits in pretrained transformers.

---

Q6: Does the LoRASuite still work when the LoRA weights are for different tasks?

A6: Thank you for this insightful question. LoRASuite is designed to adapt LoRA weights across model upgrades and remains effective even when the weights are trained on different tasks. Prior studies such as DARE and TIES have shown that merging task-specific LoRA modules is both feasible and effective. In our case, each task-specific LoRA can first be transformed using LoRASuite, and then merged using established methods like TIES to ensure compatibility with the upgraded model while preserving task-specific knowledge.

For example, we merged the MiniCPM-S-1B math and commonsense LoRA adapters using TIES with equal weights (0.5 each), and performed the same merging on the LoRASuite-transformed adapters for MiniCPM-2B. Compared to the simple average of individual task performances, the merged LoRASuite-transformed adapters improved math and commonsense scores by 1.37× and 1.96×, respectively. We will include these results in the revised manuscript.

---

Q7: Curious whether there are relatively different sensitivity when matching various factors, e.g., is layer matching more important/sensitive than head mapping?

A7: Thank you for your insightful feedback. In Appendix 6.1, we conducted ablation studies on both layer mapping and head mapping. Based on the results shown in Figures 11 and 12, we observed that layer mapping has a significantly greater influence on the average performance compared to head mapping. We will clarify this observation in our revised manuscript.