PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches

We sincerely thank reviewer YYSP for the positive feedback and constructive suggestions. We are particularly encouraged that the reviewer acknowledges our training-free framework’s novelty, comprehensive empirical validation, and significant efficiency gains. The constructive feedback from reviewer has been really helpful in improving our paper as well. Below, we address each concern reviewer YYSP raised in detail.

[Cons 1. Application to Large-scale LLMs] We agree with reviewer YYSP that our framework could be particularly beneficial for large-scale commercial LLMs. In fact, our theoretical analysis suggests that PortLLM’s benefits should scale well to larger models due to the following reasons:

1. The residual matrix remains negligible relative to the naive update term regardless of the model size, i.e., $\||\theta^\prime + \Delta \theta_i\|| >> \||\Delta\theta_i^\prime - \Delta\theta_i\||$, regardless of the number of parameters a model has, as shown in Section 3.3.
2. The computational savings would be even more significant for larger models as fine-tuning larger models (even with LoRA or its variants) is more expensive than smaller models, further showcasing the usefulness of our PortLLM framework.

Based on the reviewer YYSP’s feedback and given the time constraints, we quickly performed the same set of experiments on Llama-2-13B with 4 different datasets {BoolQ, SST-2, MRPC, WNLI}. Similar to previous experimental settings, we continued pretrained the model on OpenOrca dataset to simulate time, i.e., going from $\theta$ to $\theta^\prime$. The results are summarized in the table below

These results demonstrate that our PortLLM framework maintains its effectiveness when scaled to larger models in a number of ways:

1. Consistent Performance : Our method achieves comparable or better performance to both fine-tuned baselines across all tasks with a difference of at most 0.43% in accuracy. This validates that our theoretical insights about patch portability hold true even at larger model scales.
2. Zero-Shot Improvements : Similar to our findings with smaller models, our patching method significantly improves the zero-shot performance of the updated model $\theta\prime$. For instance, we observe substantial gains in WNLI (+25.1%).
3. Resource Efficiency : These results are particularly noteworthy given that fine-tuning Llama-2-13B with LoRA requires significantly more computational resources (approximately 2.5x more GPU Memory) compared to Llama-2-7B. Our training-free approach circumvents these scaling challenges entirely while maintaining competitive performance.

These findings further strengthen our argument that PortLLM provides a practical solution for personalizing large-scale LLMs, making it particularly valuable as models continue to grow in size.

Given these promising results, we plan to extend our evaluation to even larger architectures like Llama-2-70B in the final version of our paper. While time constraints during the rebuttal period preclude us from including these experiments for now, we believe these additional results will further validate PortLLM's scalability and effectiveness across the full spectrum of model sizes.

[Cons 2. Additional Related Works] We thank the reviewer YYSP for pointing out these relevant works to our paper. We have expanded our manuscript to include the following works:

• LMaaS (Sun et al., 2022)
• kNN-Adapter (Huang et al., 2023)
• CombLM (Ormazabal et al., 2023)
• IPA (Lu et al., 2023)
• Proxy-Tuning (Liu et al., 2024)
• BBox-Adaptor (Sun et al., 2024)

The related work section has been updated with changes being highlighted in a cyan color for the reviewer to easily gauge the differences, and how each of these methods differs from our framework.

Here is the final version of the related works section for PEFT Methods:

"Parameter Efficient Fine-tuning (PEFT). The rapid growth in the size of pretrained LLMs has posed significant challenges for efficiently fine-tuning LLMs to specific downstream tasks. To address this, numerous PEFT methods have been developed, aiming to balance efficiency and accuracy. Early approaches focused on inserting trainable adapters—feed-forward networks placed between the layers of the pretrained model (Houlsby et al., 2019; Lin et al., 2020). Recent advancements have led to more sophisticated adapter-based PEFT methods (Mahabadi et al., 2021; Pfeiffer et al., 2020; Luo et al., 2023) including LMaaS (Sun et al., 2022) for service-oriented adaptation and kNN-Adapter (Huang et al., 2023) for retrieval-augmented fine-tuning. A notable example is LoRA (Hu et al., 2021), which introduces trainable low-rank weight perturbations to the pretrained model, significantly reducing the number of parameters required for fine-tuning. LoRA’s key innovation lies in its use of the product of two low-rank matrices to approximate weight changes. Building upon this concept, several methods have emerged including Q-LoRA (Dettmers et al., 2023), CombLM (Ormazabal et al., 2023), and IPA(Lu et al., 2023). Concurrently, prompt-based learning methods have demonstrated effectiveness across various NLP tasks. Methods such as prompt-tuning (Lester et al., 2021), prefix-tuning (Li & Liang, 2021) and more recent approaches like Proxy-Tuning (Liu et al., 2024) and BBox-Adadpter (Sun et al., 2024) incorporate learnable continuous embeddings into the model’s hidden states. They condition the frozen model to adapt to specific tasks without modifying the underlying architecture. Despite these advances, fine-tuning each updated LLM with PEFT to equip personalized knowledge remains highly costly, and how PEFT can bridge the gap in personalized settings within this evolving environment in a portable manner is yet to be fully explored. To this end, we develop in this paper the theory behind portable model patches that can be plugged into an updated model to carry over the personalized knowledge from the first fine-tuned model."

[Cons 3. Superior Performance Analysis Against Fine-tuned Models] This is an excellent observation by reviewer YYSP. An effort was made to highlight these results by splitting the two different kinds of observations, using a column splitter in Table 2, into (1) datasets where we have comparable but worse performance against the Fine-tuned Updated Model, and (2) datasets where we beat the Fine-tuned Updated Model with quite a margin. We hypothesize that our method sometimes outperforms direct fine-tuning for two main reasons:

1. Knowledge Complementarity Argument: Continued Pretraining ($\theta \to \theta^\prime$) may alter the optimization landscape of the model as the parameters are updated. While both pre- and post-update models can achieve improved downstream performance through LoRA fine-tuning (following the directions of $\Delta\theta_i$ and $\Delta\theta_i^\prime$, respectively), the modified landscape after continued pretraining might lead to the descent directions of $\theta_i$ and $\theta_i^\prime$ to be different. As a result applying $\Delta\theta_i^\prime$ might end up in a sub-optimal local minimum in some cases. Overall both descent directions, $\Delta\theta_i$ and $\Delta\theta_i^\prime$, should be generally similar - enabling PortLLM’s effectiveness - the original direction $\Delta \theta_i$ might occasionally lead to superior local minima in the modified landscape. Furthermore, the impact of continued pretraining on these optimization paths is task-dependent, which explains the varying performance patterns we observe across different downstream tasks.
2. Regularization Effect Argument: The simplified patch structure may act as an implicit regularizer, preventing overfitting by constraining the model's adaptability to only the most essential task-specific changes especially in the cases where the training dataset for the downstream task has a small number of samples.

[Cons 4. Additional Evaluation Benchmarks] We appreciate this valuable suggestion. Based on reviewer YYSP’s suggestion, we performed a quick experiment in domain-specific adaptation task, in this case, MedQA. We used Mistral-7B with continued pretraining on OpenOrca dataset to showcase the universality of our model patches across domain-specific adaptation tasks as well. We fine-tuned the model on MedQA Training dataset for 1 epoch, kept the rest of the hyperparameters mentioned in Section 4 of the paper unchanged. The results are summarized as follows:

These results on MedQA, a challenging domain-specific task, demonstrate several key points:

1. PortLLM maintains strong performance in specialized domains, which is within 1% of the both fine-tuned baselines.
2. Notably, continued pretraining on OpenOrca (a general-domain dataset) shows no improvement or slight degradation in zero-shot performance (50.59% vs 50.82%), highlighting the importance of domain-specific adaptation. 3.The effectiveness of our patches in bridging this domain gap without training (7.38% improvement over $\theta^\prime$) further validates PortLLM's utility for domain adaptation tasks.

Given the time constraints of the rebuttal period, we were only able to expand our method to one of the domain-specific adaptation task. However, based on our theoretical results, it should be noted that the results will be similar on other tasks as well.

[Cons 5. Theoretical Performance Bounds] We are currently working on deriving the rigorous theoretical upper bounds on performance guarantees. We plan to share the results with the reviewer in a few days for further discussion.

We thank the reviewer 94Qe for their thoughtful feedback and acknowledge both the strengths and concerned raised. Below we address each point in detail:

[Cons 1. Excessive Content in Background/Related Works] We appreciate this feedback and have the streamlined these sections in the revision by:

1. Focusing on parameter-efficient adaptation methods in the related works section by expanding on relevant methods. Here is the final version:

“Parameter Efficient Fine-tuning (PEFT). The rapid growth in the size of pretrained LLMs has posed significant challenges for efficiently fine-tuning LLMs to specific downstream tasks. To address this, numerous PEFT methods have been developed, aiming to balance efficiency and accuracy. Early approaches focused on inserting trainable adapters—feed-forward networks placed between the layers of the pretrained model (Houlsby et al., 2019; Lin et al., 2020). Recent advancements have led to more sophisticated adapter-based PEFT methods (Mahabadi et al., 2021; Pfeiffer et al., 2020; Luo et al., 2023) including LMaaS (Sun et al., 2022) for service-oriented adaptation and kNN-Adapter (Huang et al., 2023) for retrieval-augmented fine-tuning. A notable example is LoRA (Hu et al., 2021), which introduces trainable low-rank weight perturbations to the pretrained model, significantly reducing the number of parameters required for fine-tuning. LoRA’s key innovation lies in its use of the product of two low-rank matrices to approximate weight changes. Building upon this concept, several methods have emerged including Q-LoRA (Dettmers et al., 2023), CombLM (Ormazabal et al., 2023), and IPA(Lu et al., 2023). Concurrently, prompt-based learning methods have demonstrated effectiveness across various NLP tasks. Methods such as prompt-tuning (Lester et al., 2021), prefix-tuning (Li & Liang, 2021) and more recent approaches like Proxy-Tuning (Liu et al., 2024) and BBox-Adadpter (Sun et al., 2024) incorporate learnable continuous embeddings into the model’s hidden states. They condition the frozen model to adapt to specific tasks without modifying the underlying architecture. Despite these advances, fine-tuning each updated LLM with PEFT to equip personalized knowledge remains highly costly, and how PEFT can bridge the gap in personalized settings within this evolving environment in a portable manner is yet to be fully explored. To this end, we develop in this paper the theory behind portable model patches that can be plugged into an updated model to carry over the personalized knowledge from the first fine-tuned model.”

2. Furthermore, we have tightened the section on “Large Language Models” section of the related works section to only include details relevant to the core theme of the paper. The final version looks like this:

“Large Language Models (LLMs). LLMs have transformed natural language processing, enabling models to perform complex tasks with remarkable accuracy and generalization. Models like GPT-3 (Brown et al., 2020), BERT (Devlin et al., 2019), and T5 (Raffel et al., 2023) have set benchmarks across a range of NLP tasks, from translation and summarization to question answering and text generation (Vaswani et al., 2023; Zhang et al., 2020; Rajpurkar et al., 2016). More recently, models like Llama (Touvron et al., 2023; Dubey et al., 2024), Mistral (Jiang et al., 2023), and Gemma (Team et al., 2024) have pushed the boundaries further by optimizing both performance and computational efficiency. LLama, Mistral, and Gemma represent recent advances in LLM architectures, each offering improvements in efficiency and performance. However, even with such improvements, the performance on domain-specific downstream tasks is sub-par making fine-tuning necessary. In this paper, we propose a training-free solution that enables the seamless transfer of personalized knowledge across evolving LLMs, reducing the need for costly fine-tuning and enhancing accessibility.”

3. Condensing introduction to remove discussion on LLM general capabilities. The beginning of the introduction looks like this:

“The rise of large pretrained language models has marked a significant shift in natural language processing (NLP), particularly in their ability to adapt to specific domains and tasks. These models, such as GPT-4 (Achiam et al., 2023), have achieved state-of-the-art performance by leveraging vast amounts of pretraining data (Antoniades et al., 2024). However, pretrained LLMs often require adaptation for specialized domains where context-specific knowledge is critical (Wang et al., 2022a; Bommasani et al., 2021; Qiu et al., 2020). Hence, while pretraining provides a strong foundation, fine-tuning (e.g., personalization) is essential for specific domains."

[Cons 2. Limited Rank Experimentation] To address this concern, we conducted additional experiments varying the LoRA rank for 3 different downstream tasks: {BoolQ, MRPC, WNLI}. We use Mistral-7B for this experiment with the OpenOrca dataset to go from $\theta$ to $\theta^\prime$. The rest of the settings are the same as in Section 4, except the LoRA rank for the downstream task. The results are summarized in the following table:

Based on these comprehensive results across different LoRA ranks, we can draw several important insights:

1. Performance Stability: The results demonstrate that both fine-tuning ($\theta_i$) and our method ($\theta^\prime + \Delta \theta_i) are stable across different ranks, with optimal performance generally achieved at rank 8 or 16.
2. Efficiency Sweet Spot: Notably, increasing rank beyond 8 provides diminishing returns and in some cases even leads to performance degradation, e.g., at rank 32, we observe decreased performance across most tasks compared to rank 8.
3. Resource-Performance Trade-off: The reviewer's suggestion helped us validate that our original choice of rank 8 provides an optimal balance between computational efficiency and model performance.

We genuinely appreciate the reviewer for suggesting this analysis. These findings not only validate our approach but also provide valuable insights into the robustness of our method across different adapter configurations. We have also included these results in the Appendix of our manuscript.

[Cons 3. Model Update Scenarios] We appreciate the reviewer's insightful observation about real-world model updates. To further validate our framework's robustness under periodic updates, we conducted additional experiments simulating multiple rounds of continued pretraining. Using Mistral-7B as our base model, we performed 4 sequential updates using different pretraining datasets in the following order: OpenOrca → OpenPlatypus → Alpaca → GPT4-LLM-Cleaned. For each of these updates, we evaluate our model patch on all seven downstream tasks: {BoolQ, SST-2, MRPC, RTE, Winogrande, WNLI, GSM8K}. The results are as follows:

The results demonstrate several important insights about our model patches' robustness under periodic updates:

1. Consistent Performance Improvements: Across all update stages (T=1 to T=4), our method consistently provides substantial zero-shot improvements over the updated model $\theta^\prime$.
2. Stability Under Multiple Updates: While the zero-shot performance of the updated model $\theta^\prime$ shows some fluctuation across updates, our patched models maintain remarkably stable performance.
3. Complementary Knowledge Integration:For some tasks like GSM8K, we observe that different update stages can lead to varying but still significant improvements (+16.83% to +29.79%) , suggesting our patches effectively leverage the complementary knowledge introduced by each update.

We sincerely thank the reviewer for suggesting this important aspect of model evolution. These results not only validate the robustness of our approach under multiple updates but also provide valuable insights into how model patches interact with evolving model knowledge. To the best of our knowledge, PortLLM is one of the first steps toward understanding and addressing the challenges of knowledge transfer across evolving LLMs. While our work focuses on establishing the foundational principles of patch portability, we hope it will encourage the community to explore this crucial direction further. Future research could investigate aspects such as patch composition across multiple tasks, theoretical bounds on knowledge retention, and adaptation to more diverse model architectures.

[Cons 4. Theoretical Analysis Verification] We agree that our theoretical analysis would benefit from additional verification. We are currently working on the proof of the statement in question and plan to share the results with the reviewer in a few days for further discussion.

[Cons 4. Theoretical Analysis Verification] We thank the reviewer for the suggestion. We have provided a detailed, 2-page proof in Appendix C of the updated manuscript, in which we moved away from explicitly relying on the subadditive property. We hope the new proof can address the concerns of the reviewer.

[Reminder] As we go into the extended rebuttal period, we would love to hear from Reviewer 94Qe on our rebuttal and any other possible questions that might arise.

Dear Reviewer 94Qe,

This is a gentle reminder, as the deadline is approaching. We have made every effort to address your questions thoroughly. We would sincerely appreciate it if you could confirm whether our responses have adequately resolved your concerns and consider revisiting your score, or let us know if you have any additional questions.

Best regards, Authors

Dear Reviewer 94Qe,

We sincerely appreciate your thorough feedback on our work! We hope the rebuttal above addresses your concerns and look forward to further discussions.

Best regards,

Authors

Dear Reviewer 94Qe,

We would like to sincerely thank you for your useful comments and time invested in our work. We have revised our paper and added relevant experiments in the Appendix as described above.

At present, all your concerns are responded in the rebuttal and revised version of the paper. However, as the rebuttal deadline approaches (only 1 day remaining), we kindly request your feedback to confirm that our response and revision effectively address your concerns. If there are any remaining issues, we would greatly appreciate the opportunity to address them to ensure the quality of our work. We sincerely hope that you find our response convincing and kindly consider revisiting your rating.

Best Regards,

Authors

We sincerely thank the reviewer for their detailed feedback and thoughtful questions. We address each point below:

[Cons 1. Relationship to Task Arithmetic Work] : We respectfully disagree with the characterization of our work as incremental to task arithmetic (Ilharco et al.). While both approaches involve parameter differences, they are fundamentally different in the following aspects:

1. Different Purpose and Problem Space:
   • Task arithmetic focuses on combining multiple downstream tasks through model editing.
   • PortLLM specifically addresses the challenge of personalizing evolving LLMs while maintaining downstream performance.
   • Our work explores a novel direction of training-free knowledge transfer across model versions
2. Technical Innovation in Parameter Differences:
   • Unlike task vectors, our approach avoids computationally expensive full-model updates.
   • Our model patches are derived from theoretical insights about low-rank adaptations, not task arithmetic.
   • While our method works with full-model fine-tuning (Section 4.4), the low-rank version provides unique performance benefits (Section 4.5).
3. Novel Theoretical Foundation: Lastly, in our paper, we provide theoretical analysis of why patches remain effective across model updates, and how the negligible nature of the residual matrix leads to training-free model patches, which are all key insights, never explored before.

[Cons 2. Performance and Practicality Concerns] We respectfully disagree with the assessment of limited practical value. First, comparing our training-free approach with fine-tuning is unfair- as shown in Table 6, fine-tuning requires significant computational resources whereas our method has zero training cost. Despite this fundamental efficiency advantage, we still achieve comparable performance in most cases. Furthermore, our method can be scaled to any larger model without any additional cost as demonstrated in Table 6.

Moreover, our framework's value lies in its unique ability to transfer knowledge to evolved models without any additional training or data access. As acknowledged by Reviewers YYSP, 94Qe, and TS4M, this training-free approach has significant practical implications, especially for scenarios where:

• Repeated fine-tuning is prohibitively expensive.
• Quick adaptation to model updates is required.
• Access to original training data is restricted.

[Question 1. WNLI Performance]: Thank you for raising this interesting point. In Table 2, we deliberately organized our results to highlight two distinct performance patterns: (1) tasks where our method shows comparable but slightly lower performance compared to Fine-tuned Updated Model, and (2) tasks like WNLI where our method notably outperforms the Fine-tuned Updated Model. For WNLI specifically, we hypothesize 2 specific reasons:

1. Knowledge Complementarity Argument: Continued Pretraining ($\theta \to \theta^\prime$) may alter the optimization landscape of the model as the parameters are updated. While both pre- and post-update models can achieve improved downstream performance through LoRA fine-tuning (following the directions of $\Delta\theta_i$ and $\Delta\theta_i^\prime$, respectively), the modified landscape after continued pretraining might lead to the descent directions of $\theta_i$ and $\theta_i^\prime$ to be different. As a result applying $\Delta\theta_i^\prime$ might end up in a sub-optimal local minimum in some cases. Overall both descent directions, $\Delta\theta_i$ and $\Delta\theta_i^\prime$, should be generally similar - enabling PortLLM’s effectiveness - the original direction $\Delta \theta_i$ might occasionally lead to superior local minima in the modified landscape. Furthermore, the impact of continued pretraining on these optimization paths is task-dependent, which explains the varying performance patterns we observe across different downstream tasks.
2. Regularization Effect Argument: The simplified patch structure may act as an implicit regularizer, preventing overfitting by constraining the model's adaptability to only the most essential task-specific changes especially in the case of WNLI where the training dataset is very small compared to other downstream tasks.

[Question 2. Notation Error]: Thank you for catching this error. Yes, the notation should be $\theta_i$. It has been fixed in the revised manuscript.

[Question 3. Table 6 Columns are reversed]: Yes, they were. We have fixed the column headings in our revised manuscript.

[Additional Comments]: We really appreciate the reviewer's careful reading and constructive feedback. We would like to use this opportunity to mention that while building upon existing ideas in parameter-efficient adaptation, we believe PortLLM makes meaningful contributions by

1. Addressing the specific challenge of evolving LLMs.
2. Providing theoretical insights about knowledge transfer across model versions.
3. Offering a practical, training-free solution for maintaining personalized models.

Thank you for your detailed response. While I acknowledge the efficiency advantage of your training-free approach, my concern lies in the fundamental dependency of your method on the fine-tuned model $\theta_i$ to compute $\Delta\theta_i$. As evident from Table 4, the performance of your method, $\theta' + \Delta\theta_i$, is weaker than the fine-tuned model $\theta_i$ in most cases. I understand that directly comparing your method to further fine-tuning $\theta'$ (i.e., fine-tuning $\theta'$ again) may be considered unfair. However, the comparison here is not between your method and $\theta'$. Instead, it is between your method and the fine-tuned model $\theta_i$, which your approach inherently relies on. Therefore, I believe this comparison is reasonable and highlights an important limitation of your method in terms of practical performance. I would appreciate further clarification on how your method addresses this gap or compensates for the observed performance differences.

We sincerely thank the reviewer for their thoughtful feedback, particularly regarding the theoretical aspects of our work. We address each point below:

[Cons 1. Theoretical Foundation of the Lemma]: We appreciate this insightful observation about our theoretical analysis. We are currently working on strengthening our theoretical analysis and will share them for further discussion in a few days.

[Question 1. Performance Improvements]: Thank you for this excellent observation about PortLLM sometimes outperforming $\theta_i^\prime$. We hypothesize this maybe be because of the following of two key factors:

1. Knowledge Complementarity Argument: Continued Pretraining ($\theta \to \theta^\prime$) may alter the optimization landscape of the model as the parameters are updated. While both pre- and post-update models can achieve improved downstream performance through LoRA fine-tuning (following the directions of $\Delta\theta_i$ and $\Delta\theta_i^\prime$, respectively), the modified landscape after continued pretraining might lead to the descent directions of $\theta_i$ and $\theta_i^\prime$ to be different. As a result applying $\Delta\theta_i^\prime$ might end up in a sub-optimal local minimum in some cases. Overall both descent directions, $\Delta\theta_i$ and $\Delta\theta_i^\prime$, should be generally similar - enabling PortLLM’s effectiveness - the original direction $\Delta \theta_i$ might occasionally lead to superior local minima in the modified landscape. Furthermore, the impact of continued pretraining on these optimization paths is task-dependent, which explains the varying performance patterns we observe across different downstream tasks.
2. Regularization Effect Argument: The simplified patch structure may act as an implicit regularizer, preventing overfitting by constraining the model's adaptability to only the most essential task-specific changes especially in the cases where the training dataset for the downstream task has a small number of samples. This can be seen more evidently in the case of WNLI and GSM8K, as both of these downstream tasks have small training datasets.

[Question 2. Table 6 Columns]: We apologize for the confusion in Table 6. The column headers should have been reversed. We have corrected this in the revision to accurately reflect that our method requires no parameter training.

[Question 3. Approximation Error Analysis]: Thanks for the suggestion and we are currently working on it. We will include a comprehensive analysis along with our strengthened theoretical results in a few days.

[Cons 1. Theoretical Foundation of the Lemma] We appreciate this insightful observation about our theoretical analysis. We have provided a detailed, 2-page proof in Appendix C of the updated manuscript, in which we directly measured the ratio between the Frobenius norms of the residual matrix and the naive update. We also provided an error bound as a function of matrix ranks and singular values. We hope the new theoretical results can address the concerns of the reviewer.

[Question 3. Approximation Error Analysis] Please refer to the earlier response on the newly added theoretical results in Appendix C of the revised manuscript.

[Reminder] As we go into the extended rebuttal period, we would love to hear from Reviewer TS4M on our rebuttal and any other possible questions that might arise.

Dear Reviewer TS4M,

This is a gentle reminder, as the deadline is approaching. We have made every effort to address your questions thoroughly. We would sincerely appreciate it if you could confirm whether our responses have adequately resolved your concerns and consider revisiting your score, or let us know if you have any additional questions.

Best regards, Authors