SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning

Thanks for your insightful comments, we carefully address your concerns below.

W1: However, a downside of the paper is that the training cost is not mentioned or discussed.

While SeedLoRA maintains the same inference memory footprint as vanilla LoRA, we acknowledge that the training computation increases proportionally with the number of seed models. However, since each training process is independent, we can leverage parallelization techniques to accelerate the overall training.

Additionally, we conducted experiments to evaluate SeedLoRA under comparable computational budgets as vanilla LoRA (Line 429). For instance, we compared vanilla LoRA trained for 3 epochs against SeedLoRA with 3 merged models (each trained for only 1 epoch). Our results demonstrate that SeedLoRA achieves superior performance.

Q1: How do the merging methods affect the final performance?

We try to provide the results of ablation study on LLaMA2-7b fine-tuning on MetaMathQA. We can obtain that these results show both stages are valuable, with their combination yielding optimal performance. Stage 2 alone performs slightly better than Stage 1 alone, and the combined approach consistently improves results across different model architectures.

Q2: Would the authors explicitly state the values of $\sigma$ used for each dataset and model?

For all models and datasets, we determined the threshold $\sigma$ using the top 50% magnitude of weights across adapters in each layer. This approach provides an adaptive threshold that scales appropriately with the distribution of weight magnitudes in different layers and models.

Q3: Could the authors define what “multiple distinct groups” means and explain how to calculate and determine the conflicting dimensions?

We determine conflicting dimensions through the following specific process:

For each parameter dimension $j$, we examine the values $\tau_i(j)$ across all n adapters. We classify adapters into sign groups based on the sign of their values in dimension $j$. A dimension is considered "conflicting" when it meets both of the following criteria:

(1) It contains at least two sign groups (positive and negative).

(2) Each sign group contains at least $\lfloor n/3 \rfloor$ adapters with magnitude $|\tau_{i}(j)| \geq \sigma$.

This means that for dimension j to be conflicting, there must be a substantial number of adapters (at least one-third of the total) strongly pulling in opposite directions. For example, with 3 adapters, a dimension is conflicting if at least 1 adapter has a large positive value and at least 1 has a large negative value.

Q4: The algorithm for fusing the set of coordinates $\widetilde{Z}$ is also unclear.

The calculation of the fused coordinate matrix $\widetilde{Z}^{(l)}$ proceeds as follows:

• After projecting each adapter's moderate parameters $\tau_i^{(l)}$ onto the common subspace to obtain coordinate matrices $Z_i^{(l)}$, we compute the element-wise average of these coordinate matrices: $Z_{\text{avg}}^{(l)} = \frac{1}{n}\sum_{i=1}^{n} Z_i^{(l)}$

• We then examine each element $(m,k)$ in these coordinate matrices across all adapters: \begin{enumerate}

  • If the element values $Z_i^{(l)}(m,k)$ have the same sign across all adapters, we retain the average value: $\widetilde{Z}^{(l)}(m,k) = Z_{\text{avg}}^{(l)}(m,k)$
  • If there are sign conflicts among adapters, we follow a modified TIES approach: $\widetilde{Z}^{(l)}(m,k) = \frac{1}{|I^+|}\sum_{i \in I^+} Z_i^{(l)}(m,k), \quad \text{if } |I^+| \geq |I^-|$ $\widetilde{Z}^{(l)}(m,k) = \frac{1}{|I^-|}\sum_{i \in I^-} Z_i^{(l)}(m,k), \quad \text{otherwise}$ where $I^+$ and $I^-$ are the sets of adapter indices with positive and negative values for element $(m,k)$, respectively.

This approach preserves agreement between adapters while resolving conflicts by favoring the dominant sign group.

Q5: Would you compare the training time? What if we fix a seed and train it for n $\times$ longer so that the total training time matches the entire SeedLoRA process?

We have conducted the experiments comparing SeedLoRA with longer training of a single model. In the "Training with More Epochs" section (line 408) and Figure 4(a)-(b), we compare merging 3 LoRA models (each trained for 3 epochs) against a single LoRA model trained for 9 epochs, ensuring the training computation cost is equivalent. Our results show that SeedLoRA outperforms the longer training approach on both mathematical reasoning and code generation tasks.

Thanks for your constructive and inspiring feedback, we carefully address your concerns below.

W1: Effectiveness across different ranks.}

To directly address this concern about the scalability of SeedLoRA across different rank settings, we have conducted comprehensive additional experiments with higher-rank LoRA adapters (r=16, 32, 64 and 96) on the MetaMathQA benchmark.

• When merging 3 models (LLaMA2-7b):

These results clearly demonstrate that SeedLoRA consistently improves performance across all tested rank settings. This confirms that SeedLoRA's effectiveness is not limited to low-rank settings but extends to the higher-rank adapters.

W2: Merging adapters to longer training (9 epochs) lacks granularity.

To address this concern, we have conducted additional experiments comparing SeedLoRA with equivalent or higher-rank single LoRA adapters:

These results demonstrate that SeedLoRA offers superior performance compared to single higher-rank alternatives. SeedLoRA (merging three r=8 adapters) achieves +3.5% better performance on GSM8K compared to a single r=24 LoRA adapter. Moreover, the improvement scales to very high ranks, with SeedLoRA (merging three r=32 adapters) outperforming a single r=96 LoRA adapter by +3.3% on GSM8K.

W3: The proportion of parameters classified as robust/conflicting during stage 1, nor does it justify the threshold .

To address these concerns, we've analyzed the proportion of parameters classified in each stage across different layers.

(1) Layer-wise parameter distribution analysis

We measured the proportion of parameters classified across different layers and tasks:

• LLaMA2-7b on MetaMathQA:

Overall, approximately 13-16% of parameters are classified as robust or conflicting in Stage 1, with variations across layer types.

(2) Threshold selection and ablation study

The threshold was determined using the top 50% magnitude of weights across adapters in each layer. This approach provides an adaptive threshold that scales appropriately with the distribution of weight magnitudes in different layers and models.

To verify the importance of Stage 1, we conducted an ablation experiment:

• LLaMA2-7b and Mistral-7b on MetaMathQA

These results show both stages are valuable, with their combination yielding optimal performance.

• Merging all parameters through subspace fusion: we conducted additional experiments about SeedLoRA (rank=8) with only subspace fusion for all parameters, which resulted in 67.5% on GSM8K and 16.4% on MATH (SeedLoRA: 69.1% on GSM8K and 17.1% on MATH).

W4: If performance continues to improve with more seed models.

To investigate scaling behavior when merging additional models, we conducted experiments merging up to 6 different seed models:

These results show significant initial gains when merging 2 models (+3.7% on GSM8K), with continued improvements up to 4 models (+5.7% on GSM8K). Beyond 4 models, we observe diminishing returns without performance degradation.

Thanks for your constructive and inspiring feedback, we carefully address your concerns below.

Clarification on seed reporting and cosine similarity calculations

Thank you for your feedback on the experimental design and reproducibility. We agree that additional clarity on seed variation would strengthen our presentation.

(1) Reporting of LoRA results in Tables 2 and 3

In Tables 2 and 3, we report individual results for each seed (seed=11, seed=42, seed=202) rather than averages. Each column represents the performance of a single LoRA model trained with that specific seed. We chose to present the individual seed results rather than averages to highlight the performance variation across seeds, which is a key motivation for our SeedLoRA approach.

(2) Clarification on cosine similarity in Figure 2

Regarding Figure 2, the cosine similarity values shown are computed as averages across all layers of the models. Specifically, for each adapter pair, we first computed the cosine similarity between corresponding parameters in each layer, then averaged these similarities across all layers to obtain the single value shown in the figure.

Possibly more thorough discussion of SWA-like arguments (which they partially reference) could reinforce the “wider optimum” viewpoint in merging.

Thank you for your insightful comment regarding the potential connection between SeedLoRA and SWA-style techniques.

While both SWA and SeedLoRA involve combining model weights, their underlying assumptions and practical implications differ substantially. SWA averages model weights sampled along the same optimization trajectory, typically assuming that these checkpoints lie within the same basin of attraction in the loss landscape. This allows SWA to converge to flatter optima, often associated with better generalization and robustness.

In contrast, SeedLoRA merges multiple LoRA adapters trained from different random seeds, which often leads to models residing in different regions of the parameter space—potentially even in distinct basins. As a result, direct weight averaging, as in SWA, can be ineffective or even detrimental in this context.