SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning

Thank you for your constructive and positive feedback, we carefully address your concerns below.

W1:

Limited impact scope. LoRA and its variants have performed well in multiple tasks and are close to full fine-tuning. The proposed SeedLoRA focuses only on complex tasks such as code generation and mathematical reasoning. The fact that the SeedLoRA only improves the performance on those particular tasks seems quite limited. Would SeedLoRA be helpful in any other tasks outside of these? What are the patterns that the proposed approach would work and that the proposed approach would not work? The experiments in this paper do not provide those answers. That would be very limited if the proposed approach only works on single tasks in code generation or mathematical reasoning.

Thank you for your constructive comment. We want to respectfully clarify that SeedLoRA's impact extends beyond code generation and mathematical reasoning. Our comprehensive evaluation spans three major areas:

1. Mathematics Reasoning
2. Code Generation
3. General Domain Instruction Tuning [1]

Particularly, our evaluation on General Domain Instruction Tuning (as shown in Table 4 and Table 8) demonstrates SeedLoRA's broad applicability across diverse domains:

• Factual Knowledge (MMLU)
• Reasoning Ability (GSM8K, BBH)
• Multilingual Capability (TyDiQA)
• Coding Skills (HumanEval)

The experimental results show consistent improvements over vanilla LoRA across these domains. For instance, on LLaMA2-7b, SeedLoRA improved the average performance from 36.4 to 37.0, demonstrating its effectiveness not just in specialized tasks but also in general domain applications.

[1] Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2

W2:

Is SeedLoRA practical? A key motivation behind LoRA is its ability to fine-tune LLMs on relatively inexpensive and limited GPUs quickly. The proposed SeedLoRA, however, would significantly increase the time required to obtain multiple models for a single task, raising concerns about its practicality. The number of models needed for a single task is also unclear, further questioning the feasibility of the approach.

Thank you for this insightful comment. We demonstrate that SeedLoRA remains practical while offering significant performance benefits:

1. Optimal Number of Models for Merging We conducted extensive experiments varying the number of merged models from 2 to 5 on MetaMathQA. The results show:

Our findings reveal:

• Even merging just 2 models yields substantial improvements: GSM8K accuracy increases from 64.0% to 68.2%, and MATH accuracy from 15.3% to 16.2%
• Performance typically plateaus at 3 models, with minimal gains from additional merges. For instance, merging 3 models (68.6%, 17.3%) achieves comparable results to merging 4 (68.5%, 16.9%) or 5 models (68.9%, 16.5%)
• For practical applications, merging 2-3 models offers an optimal trade-off between performance and computational cost

2. Efficiency and Resource Requirements SeedLoRA maintains the key advantages of LoRA while offering enhanced performance:

• Memory Efficiency: SeedLoRA introduces no additional memory overhead compared to standard LoRA, preserving its compatibility with resource-constrained GPUs
• Training Time Optimization: While training multiple models does increase total computation time, we demonstrate an efficient alternative approach:
  • Instead of the standard 3-epoch LoRA fine-tuning, we can train 3 individual models for 1 epoch each and merge them.
  • Our experiments on MetaMathQA (Figure 6) show this approach outperforms standard 3-epoch LoRA training while maintaining the same total training duration.
  • This demonstrates that SeedLoRA can achieve superior results without necessarily requiring additional training time when properly configured.

These findings demonstrate that SeedLoRA provides a practical enhancement to LoRA, offering improved performance while maintaining its core benefits of efficient fine-tuning on limited hardware.

Thanks for your constructive comments, we carefully address your concerns below.

W1:

Lack of novelty: The motivation is good. However, the approach for model merging is quite straightforward. It is unclear why such simple averaging or model normalization works. It lacks insights and explanations.

Thanks for your constructive comment. Recent work has demonstrated that parameter values from different training runs can be interpolated without increasing loss due to linear mode connectivity (LMC) [1, 2, 3]. When multiple models are fine-tuned from the same pre-trained model, they share common optimization patterns. This shared foundation means these models can be effectively merged without complex neuron alignment procedures, a property known as permutation symmetry in technical terms.

Our analysis extends these insights by revealing fundamental differences between multi-task and single-task model merging scenarios. Through cosine similarity analysis, we show that models trained on different tasks exhibit near-zero similarity (orthogonality), while models trained on the same task with different seeds demonstrate high cosine similarity, indicating substantial shared information. This finding aligns with weight averaging (WA) literature [4, 5, 6], which shows that under shared pre-training, weights can be linearly interpolated despite architectural non-linearities.

Our experiments demonstrate that LoRA models trained with different seeds exhibit complementary strengths across task subdomains. For instance, our analysis of MMLU subdomains shows that different LoRA models excel in distinct areas - one seed may perform better in professional accounting while another excels in college mathematics. This complementarity, combined with their shared optimization foundation, enables effective merging without complex alignment procedures.

The effectiveness of our Weight Distribution Match technique is validated by empirical results: on GSM8K, SeedLoRA improves performance from 64.0% to 68.6%, surpassing both full fine-tuning (66.5%) and existing merging methods. Our SVD analysis shows broader non-zero singular value distributions in merged models, indicating successful knowledge fusion. This approach enables multiple benefits: enhanced performance on individual tasks (like math problem-solving), improved generalization to new scenarios, and robust performance across different subdomains of the task.

[1] WARM: On the Benefits of Weight Averaged Reward Models, ICML 2024.

[2] Linear Mode Connectivity and the Lottery Ticket Hypothesis, ICML 2020.

[3] What is being transferred in transfer learning?, NeurIPS 2020.

[4] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML 2022.

[5] Diverse Weight Averaging for Out-of-Distribution Generalization, NeurIPS 2022.

[6] Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization, ICML 2023.

W2:

Lack of clarity. The storyline was good at the beginning. I was confused by the proposed model merging or fusion. Specifically, in Line 290, it is unclear about the reference model. The authors mentioned that it can be the best-performing individual model. How do you know the best model? Model merging is supposed to be done without looking into the test data. If you merge models according to the testing performance, this is wrong!

We thank the reviewer for this important observation. We would like to clarify that in our current work and experiments, we treat all individual models as equally important and do not perform any selection based on test data performance. The mention of using a 'best-performing model' in Line 290 was intended only as a theoretical possibility for future work, and we agree that this reference may cause confusion. We will remove this statement in the revised version to maintain clarity and avoid any misunderstanding about our methodology. To be explicit, our model merging approach is conducted without any access to or consideration of test data performance. In the revised version, we will update Line 290 to explicitly state that our model merging approach treats all constituent models equally, removing any reference to the selection of a best-performing model.

Thanks for your constructive feedback, we carefully address your concerns below.

W1:

The analysis of LoRA and full fine-tuning in Sec. 3.2 is not persuasive enough. Can only one case demonstrate the statement "different LoRA models, each trained with unique random seeds, tend to excel in distinct subdomains"? This point may be the most important insight of this paper, so more sophisticated and strict experiments or derivations should be delved into.

Thank you for your constructive feedback regarding the need for more comprehensive evidence to support our claim about LoRA models with different random seeds excelling in distinct subdomains.

To thoroughly validate this observation, we conducted extensive experiments using general instruction tuning across diverse domains: Factual Knowledge (MMLU), Reasoning Ability (GSM8K, BBH), Multilingual Capability (TyDiQA), Coding Skills (HumanEval).

Our analysis encompasses two model architectures (LLaMA2-7b and Mistral-7b-v0.1) and two fine-tuning methods (LoRA and LoRA+), each tested with three different random seeds (11, 42, 202). The results consistently demonstrate domain-specific excellence across different seeds:

1. LLaMA2-7b with LoRA (TULU-v2):

• seed=11 excels in GSM8K (22.5%) and TyDiQA (51.8%)
• seed=42 leads in MMLU (50.1%) and BBH (45.7%)
• seed=202 shows strength in HumanEval (15.5%)

2. Mistral-7b-v0.1 with LoRA (TULU-v2):

• seed=11 achieves best performance in MMLU (59.4%) and TyDiQA (59.9%)
• seed=202 dominates in GSM8K (50.5%) and BBH (58.7%)
• seed=42 shows superior performance in HumanEval (35.5%)

To further validate our findings, we replicated these experiments using LoRA+, an advanced variant of LoRA:

3. LLaMA2-7b with LoRA+ (TULU-v2):

• seed=11 leads in HumanEval (17.9%)
• seed=42 excels in MMLU (50.3%)
• seed=202 shows strength in GSM8K (25.0%), BBH (46.5%), and TyDiQA (53.1%)

4. Mistral-7b-v0.1 with LoRA+ (TULU-v2):

• seed=11 maintains strong performance in HumanEval (34.1%)
• seed=42 leads in MMLU (61.2%), BBH (59.7%), and TyDiQA (59.7%)
• seed=202 excels in GSM8K (47.0%)

These comprehensive results consistently demonstrate that models fine-tuned with different random seeds develop distinct strengths across various domains. This systematic pattern holds true across different model architectures (LLaMA2-7b vs. Mistral-7b) and fine-tuning methods (LoRA vs. LoRA+), providing robust evidence for our observation. This finding directly motivated our proposed model fusion method, which leverages these complementary strengths to create a more capable combined model.

Thanks for your constructive and inspiring feedback, we carefully address your concerns below.

W1:

While the claims are clear and straightforward, making the paper understandable, the overall writing feels unpolished. Numerous issues, ranging from serious errors that hinder comprehension to minor typos, are apparent throughout the paper.

Thank you for this thorough review and valuable feedback regarding the manuscript's writing quality. We have carefully addressed these concerns through a comprehensive revision of the entire manuscript. Specifically:

• Figure 3: We have thoroughly revised Figure 3 by incorporating the SeedLoRA results and optimizing its placement within the paper to enhance readability and flow of the discussion.
• Line 296: We have corrected the mathematical error in the equation for $\hat{\sigma}$ on Line 296, and we appreciate you bringing this to our attention.
• Line 71: Thanks for your careful observation and we have fixed the error on Line 71 as noted.

W2 & Q1:

Key technical details are missing or unclear. For instance, in Section 3.4, under "Step 2," there is no explanation of how the importance weight $w_i$ is determined.

In Step 2 of Section 3.4, could you clarify how the importance weight w is determined?

Thank you for raising this important point regarding the importance weight calculation in Section 3.4.

In our current experiments, we set the importance weight $w_i = 1$ for all models, treating them as equally important since they differ only in their random seeds. While we used uniform weights in this study, we deliberately introduced $w_i$ in Step 2 to maintain generality in our formulation. This allows for future scenarios where different models might warrant different weights (for example, when using validation performance to determine model importance). We will add this clarification to Section 3.4 to improve clarity.

W3 & Q2:

Since the model weights are averaged (or weighted averaged), the overall weight distribution remains bound to the original distribution of the individual models (as is the case with Model Soup), potentially resulting in no significant shift. Moreover, in Step 3 of Section 3.4, the "reference model" is ultimately a simple average of the model weights. It is difficult to accept that such a minor distribution shift (or even no shift at all) would be practically meaningful. A theoretical analysis in the solution space or references supporting the necessity of this subtle distribution shift is needed.

Could you provide a more detailed theoretical rationale for the proposed model distribution matching?

Thank you for this careful consideration of our distribution matching approach. We would like to clarify several aspects of SeedLoRA that may have been misunderstood:

1. SeedLoRA goes beyond simple weight averaging. While the first step involves weighted averaging (Equation 4), a crucial second step performs distribution matching (Equation 5) that actively rescales the merged weights to match target statistical properties. This is fundamentally different from Model Soup, which only performs weight averaging.
2. Through empirical analysis of weight distributions, we observed:

• LoRA models trained on the same task consistently exhibit Gaussian-like characteristics, with most weights concentrated near zero
• After weight averaging (Step 2), there is a significant distribution shift:
  • Weights near zero in original models tend to increase in magnitude
  • Larger magnitude weights tend to decrease
  • This shift disrupts the careful balance of weight distributions learned during training
• For approximately Gaussian distributions, mean and standard deviation are sufficient statistics to characterize and correct these shifts
• These patterns hold consistently across different random seeds and training runs

3. The "reference model" parameters ($\hat{\mu}$ and $\hat{\sigma}$) are not simply averages of model weights, but rather averages of the statistical properties (mean and standard deviation) of individual models. Our formulation:

• Preserves the relative relationships between weights while adjusting their scale
• Maintains numerical stability through standardization before rescaling
• Efficiently captures the key statistical properties we observed in our weight distribution analysis
• Critically, helps restore the proper weight distribution that was disrupted during averaging

4. From a theoretical perspective, maintaining proper statistical distributions is crucial for their performance. Our distribution matching provides:

• A reliable way to preserve essential statistical properties while avoiding potential instabilities
• Robustness through standardization and rescaling steps
• Stability by focusing on key statistical moments rather than more complex distribution metrics
• A mechanism to correct the significant distribution shifts caused by weight averaging

I have carefully reviewed all of the author's responses and consolidated my feedback into a comprehensive comment.

The additional results provided by the authors, including full fine-tuning performance and performance with different random seeds, convincingly demonstrate the effectiveness of SeedLoRA. The evidence presented makes it difficult to refute its efficacy.

However, I would like to request further clarification on the following points:

1. Although I have read the author's responses to Reviewer tN6o, questions regarding “Distribution Matching” and the “Reference Model” remain unresolved.

• Assuming that each delta model $\tau_i$ is independent and the importance weights are $w_i=1$ (i.e., $\tau_m = \frac{1}{n} \sum_{i=1}^n \tau_i$), based on the information provided in Lines 286–291, it follows that: $\mu(\tau_m) = \mu\left(\frac{1}{n} \sum_{i=1}^n \tau_i\right) = \frac{1}{n} \sum_{i=1}^n \mu(\tau_i) = \hat{\mu}.$ This indicates that the mean of the merged model remains completely identical before and after distribution matching.
• Similarly, the standard deviation of the merged model is: $\sigma(\tau_m) = \sqrt{\text{Var}(\tau_m)} = \sqrt{\frac{1}{n^2} \sum_{i=1}^n \sigma^2(\tau_i)} = \frac{1}{n} \sqrt{\sum_{i=1}^n \sigma^2(\tau_i)}.$ And, the standard deviation of the reference model, considering the author's intent and correcting any errors in the paper, is $\hat{\sigma} = \frac{1}{n} \sum_{i=1}^n \sigma(\tau_i).$
• In summary, the distribution matching proposed in the paper, under the general case ($w_i=1$ and the reference model being determined by the statistics of all individual delta models), fails to alter the mean at all but adjusts the standard deviation from $\frac{1}{n} \sqrt{\sum_{i=1}^n \sigma^2(\tau_i)}$ to $\frac{1}{n} \sum_{i=1}^n \sigma(\tau_i).$ While the empirical efficacy of this approach has been demonstrated, a mathematical rationale supporting this technique is necessary. Should this be viewed as merely a heuristic technique? Additionally, please clarify exactly how the reference model was defined in the experiments presented in the paper.

2. Are all the full fine-tuning results presented in the paper taken from existing literature, or were they obtained through the authors' own training? Additionally, can the authors provide details on the absolute GPU hours required and the GPU models/numbers used to complete the full fine-tuning and LoRA fine-tuning experiments under the experimental settings described in the paper?

Thanks for your constructive and inspiring feedback, we carefully address your concerns below.

1. Although I have read the author's responses to Reviewer tN6o, questions regarding “Distribution Matching” and the “Reference Model” remain unresolved.

Thank you very much for taking the time to read our rebuttal and the other reviewers' comments. We will address your concerns regarding Distribution Matching and the Reference Model.

Firstly, we appreciate your thorough analysis, which is indeed correct.

Regarding your observation about the mean values: You are correct that the mean of the merged model remains identical before and after distribution matching. This is intentional, as the mean values of open-source pre-trained models and fine-tuned models typically hover close to 0, and we aim to preserve this characteristic.

Concerning the standard deviation analysis: Our method adjusts the standard deviation from $\frac{1}{n}\sqrt{\sum_{i=1}^{n} \sigma^{2}(\tau_{i})}$ to $\frac{1}{n}\sum_{i=1}^{n}\sigma(\tau_{i})$. We observed that the averaging merge in step 2 typically results in a smaller standard deviation than individual models ($\frac{1}{n}\sqrt{\sum_{i=1}^{n} \sigma^{2}(\tau_{i})}$ is generally smaller than $\sigma(\tau_{i})$). This reduction in standard deviation occurs because averaging tends to increase weights that were near zero in the original models while decreasing larger magnitude weights. This behavior potentially risks losing important information encoded in the larger magnitude weights of individual models. Therefore, we aim to increase the standard deviation of the averaging model to match $\frac{1}{n}\sum_{i=1}^{n}\sigma(\tau_{i})$, which better represents the standard deviation of individual models.

We can further explain equation (5) from a layer-wise normalization perspective. The equation can be rewritten as: $\tau_{m} = \hat{\sigma} \cdot \frac{\tau_{m}-\mu(\tau_{m})}{\sigma(\tau_{m})} + \hat{u} \approx \hat{\sigma}\cdot\frac{\tau_{m}}{\sigma(\tau_{m})}$, when $\mu(\tau_{m})$ and $\hat{\mu}$ are close to 0. As mentioned earlier, this assumption holds for most pre-trained and fine-tuned models, which we confirmed through analysis of our step 1 fine-tuned models. Given that the mean values are close to 0 and smaller than the standard deviation, we can approximately rewrite the definition of standard deviation as: $\sigma(\tau_{m})\approx\sqrt{[(\tau_{m}^{0}-0)^{2}+(\tau_{m}^{1}-0)^{2} + \cdots +(\tau_{m}^{i}-0)^{2}+\cdots+(\tau_{m}^{N}-0)^{2}]/N}=\sqrt{\sum_{i=1}^{N}(\tau_{m}^{i})^{2}/N}=\frac{||\tau_{m}||}{\sqrt{N}}$. Similarly, $\hat{\sigma}=\frac{1}{n}\sum_{i=1}^{n}\sigma(\tau_{i})=\frac{1}{n}\sum_{i=1}^{n}\frac{||\tau_{i}||}{\sqrt{N}}$

Therefore, equation (5) can be rewritten as: $\tau_{m} = \hat{\sigma} \cdot \frac{\tau_{m}-\mu(\tau_{m})}{\sigma(\tau_{m})} + \hat{u} \approx \hat{\sigma}\cdot\frac{\tau_{m}}{\sigma(\tau_{m})}\approx \frac{1}{n}\sum_{i=1}^{n}||\tau_{i}||\cdot\frac{\tau_{m}}{||\tau_{m}||}$. This reformulation demonstrates that our distribution matching can be interpreted as a layer-wise adaptive weight normalization.

2. Are all the full fine-tuning results presented in the paper taken from existing literature, or were they obtained through the authors' own training? Additionally, can the authors provide details on the absolute GPU hours required and the GPU models/numbers used to complete the full fine-tuning and LoRA fine-tuning experiments under the experimental settings described in the paper?

Thank you for your questions regarding our experimental details. We confirm that all fine-tuning results presented in the paper are from our own training experiments, not from existing literature. To address your question about computational resources, we have detailed the GPU hours required for both LoRA and Full Fine-tuning approaches on MetaMathQA in the table below. All experiments were conducted on H100-80GB GPUs, maintaining consistent settings including global batch size and gradient accumulation across configurations.

Thanks for your constructive feedback, we carefully address your concerns below.

Even without the assumption that the mean is close to 0, the rationale behind the "normalization" mentioned by the authors is already self-evident. The key question is why the standard deviation is rescaled to $\frac{1}{n}\sum_{i=1}^{n}\tau_{i}$, and I was unable to find any insight into this in the response.

To address your concern, we try to explain it from the perspective of linear connectivity:

Prior studies [1,2,3] have shown that models trained from similar initializations tend to maintain a low-loss path between them—a property known as linear connectivity. Linear connectivity describes the linear relationship between changes in model weights and their corresponding differences in output features resulting from fine-tuning: the features produced by the weight-interpolated model closely approximate the linear interpolation of features from the two fine-tuned models. We can define Linear connectivity as: $f(\theta_{0}+\alpha \tau)=f(\theta_{0}) + \alpha \Delta f(\theta_{0}+ \tau)$ and $f(\theta_{0}+\alpha \tau_{m})=f(\theta_{0}+\frac{\alpha}{n}\sum_{i=1}^{n}\tau_{i}) \approx f(\theta_{0}) + \frac{\alpha}{n} \sum_{i=1}^{n}\Delta f(\theta_{0}+\tau_{i})=f(\theta_{0})+\alpha \cdot \frac{\Delta f(\theta_{0}+\tau_{1}) + \cdots + \Delta f(\theta_{0} +\tau_{i}) + \cdots + \Delta f(\theta_{0} + \tau_{n}))}{n}$, where $\theta_{0}$ represents the pre-trained model, $\tau_{i}$ is the i-th delta model weights $\tau_{i}=\theta_{i}-\theta_{0}$, $\alpha$ is a scale parameter, and $\Delta f(\theta_{0} + \tau_{i}) = f(\theta_{0}+\tau_{i})-f(\theta_{0})$.

These equations reveal that averaging in weight space can be interpreted as averaging in output (feature) space. This insight explains why model averaging effectively combines information from different models and performs well across diverse sub-tasks.

As previously analyzed, distribution matching can be viewed as multiplying by a scale parameter $\frac{\frac{1}{n}\sum_{i=1}^{n}|\tau_{i}|}{|\tau_{m}|}$, which is greater than 1. Therefore, we set $\alpha=\frac{\frac{1}{n}\sum_{i=1}^{n}|\tau_{i}|}{|\tau_{m}|}$ in the above equation for distribution matching. Based on the equation $f(\theta_{0}+\frac{\alpha}{n}\sum_{i=1}^{n}\tau_{i})=f(\theta_{0}+\alpha \tau_{m})=f(\theta_{0})+\alpha \cdot \frac{\Delta f(\theta_{0}+\tau_{1}) + \cdots + \Delta f(\theta_{0}+\tau_{i}) + \cdots + \Delta f(\theta_{0} + \tau_{n}))}{n}$, we can identify two key components in the final outputs: the pre-trained model output $f(\theta_{0})$ and the average of delta outputs $\alpha \cdot \frac{\Delta f(\theta_{0}+\tau_{1}) + \Delta f(\theta_{0}+\tau_{i}) + \cdots \Delta f(\theta_{0} + \tau_{n}))}{n}$. The second component enables the combination of outputs from different models, explaining the effectiveness of model averaging. However, as the number of models increases, the delta output distribution becomes smoother, potentially diminishing the important information from individual fine-tuned models while increasing the dominance of pre-trained outputs. To counter this undesirable effect, we introduce a scale hyperparameter $\alpha$ (larger than 1) on outputs to preserve the valuable information from individual models.

In SeedLoRA, for the equation $f(\theta_{0}+\frac{\alpha}{n}\sum_{i=1}^{n}\tau_{i})=f(\theta_{0}+\alpha \tau_{m})=f(\theta_{0}+\frac{\frac{1}{n}\sum_{i=1}^{n}||\tau_{i}||}{||\tau_{m}||}\tau_{m})$, we employ the scale value $\alpha=\frac{\frac{1}{n}\sum_{i=1}^{n}||\tau_{i}||}{||\tau_{m}||}$ to ensure the merged delta model maintains a norm similar to individual delta models. This approach effectively scales the output features while maintaining stability in a controllable manner. Consequently, it reinforces the important information from individual models and demonstrates strong performance across various sub-tasks.

[1] On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm. ICML 2024.

[2] Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. MeurIPS 2023.

[3] Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity. NeurIPS 2023.