Decoupling Angles and Strength in Low-rank Adaptation

We thank the reviewer for their feedback, and provide our response to each raised issue separately below.

[Wa - Q1] Lack of broader experimental setting (e.g. GLUE on DeBERTa-v3)

We thank the reviewer for the benchmark suggestion. In the current manuscript, we tested how DeLoRA performs on the GLUE benchmark. Rather than DeBERTa-v3, we decided to finetune RoBERTa-base for time/resources reasonings, and in order to compare in the same setting of [Wu Z et al., Wu M et al.], which we find to be fairer than usually reported benchmarks on the validation sets.

For details about the GLUE benchmark, we refer to the Global Authors Response above, and to the updated manuscript.

[Wb] Encourage the authors to include additional robustness evidence (and other analysis)

We thank the reviewer for the suggestion. As highlighted by reviewer pmk9, we notice that we omitted to describe Fig.2 (previously Fig.4) in the manuscript.

We updated the manuscript with proper description, and further updated Fig.2 (previously Fig.4) introducing analysis on the weights updates (Fig.2, Right).

In Appendix D, we also provide a further ablation demonstrating DeLoRA robustness to learning rate changes on angular and scale components (Fig.6). We agree that further analysis, for example in continual learning settings, could be an interesting future research direction

[Q2] Could it happen that a column of A and B is driven towards zero during training, causing instabilities?

When testing different initialization schemes, we indeed found that initializing B with zeros (with tolerance to avoid explosion) would sometimes lead to instabilities. In our final scheme where we initially subtract a copy of BΞA from the original weights instead, we didn’t see this happening in practice. We notice that to potentially avoid instabilities issues, we did not use weight decay.

References:

[Wu Z et al.] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Reft: Representation finetuning for language models, 2024b

[Wu M et al.] Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Advancing parameter efficiency in finetuning via representation editing, 2024a.

We thank the reviewer for their feedback, and provide our response to each raised issue separately below.

[W1a] Contribution feels somewhat incremental in comparison to DoRA.

We thank the reviewer for the raised concern, we provide a more detailed analysis on the differences with DoRA in our Global Authors Response above, and we added a paragraph in Section 2.2 of the manuscript to better clarify the differences.

[W1b] Similar to DeLoRA, DoRA can limit its scale term within a fixed range, thereby preventing over-drifting in downstream adaptations. Would this lead to the same robustness effect?

With DeLoRA, in order to let the boundary to optimally adapt to different layers, the λ parameter is learnable, making its boundaries equivalent to the current DoRA implementation (being unbounded, in principle).

At the same time, as described in our comparison with DoRA in our Global Authors Response above, divergence does not happen. This suggests that our proposed method is able to decouple the transformation strength from the angular learning more effectively than DoRA, leading to the robustness properties as shown in Fig.2 (previously Fig.4).

To provide a more concrete test, we decide to test how DoRA would perform while fixing its magnitude term: if this would lead to robustness properties similar to DeLoRA. Experiments and preliminary results are reported in Appendix C in our updated manuscript.

Summarizing our findings, we test DoRA with fixed magnitude (i) on performance, where it shows to be only slightly underperforming w.r.t. the original DoRA, and (ii) on robustness, where plots between the two DoRA variations closely overlap.

This behavior suggests that keeping column norms constant might not be restrictive enough, whereas DeLoRA inner normalization seems to be more effecting to avoid model divergence.

[W2a] LoRA and DoRA algorithms are typically evaluated on well-established NLP and NLU tasks, such as the GLUE benchmarks and Commonsense Reasoning

We thank the reviewer for the suggestion. We now tested our proposed DeLoRA method on the GLUE benchmark. We refer to the Shared Response for the details. For time/resources reasons we are not able to evaluate on the Commonsense Reasoning benchmark.

Given the same pretrained network (Llama2-7B), we argue our Instruction Tuning benchmark could be sufficient to show DeLoRA abilities in finetuning large-scale Language Models

[W2b] Is the Frobenius ball restriction always necessary? There might be cases where the pretrained model diverges significantly from the target task

We agree that there might be tasks that might require stronger divergence from the pretrained network. With our DeLoRA proposal, based on the domain of the target task, one can arbitrarily control this ball restriction by initially tuning λ.

In Fig. 2 (and Fig.6 in Appendix D) of our updated manuscript, we demonstrate that using high learning rates for the boundary does not lead to divergence. This suggests that our proposed method can effectively adapt to domains where the optimal ball variance differs significantly across layers.

We believe exploring task-specific lambda values is a promising future direction, particularly in determining how these values should be adjusted based on the similarity between the current task and the original pretraining domain.

We thank the reviewer for their feedback, and provide our response to each raised issue separately below.

[W1] For clarity the DeLoRA method could have its own section

We thank the reviewer for the suggestion. In our revision, we added a paragraph to the updated manuscript in section 2.2 named “DeLoRA formulation” summarizing our DeLoRA setup and key elements

[W2-3] From Fig.3, DINO scores for DeLORA seem to degrade over time, plus current plot would gain if averaged over multiple runs

Indeed, DeLoRA performance degrade over time, however qualitative examples in Fig.3 (bottom) and in Appendix show that this degradation happens slower than for LoRA (artifacts for DeLoRA slightly appear at iteration 1200, and strongly appear at iteration 2400, while for LoRA artifacts are already strong from iteration 900).

In this regard, we agree with the reviewer that DINO scores for prolonged finetuning as reported in Fig.3 are confusing. In this regard, we highlight that results with high distortion/artifacts can still show good DINO score (e.g. LoRA finetuned model at iteration 1500, which reaches a DINO score above 0.7, while showing a strong presence of artifacts).

For this reason, we decide to remove the DINO score plot in the updated version of the manuscript, and decide to only compare different methods on qualitative examples. For a fair evaluation, we include additional randomly sampled qualitative examples of prolonged generation in Appendix E.

[Q1] Derivation of DeLoRA from ETHER, especially the step from multiplicative to additive FT, is not clear

The transition from multiplicative to additive finetuning transforms the multiplicative update W' = (I - λ/r(BΞΑ - DΦC))W with weight change ΔW = λ/r(BΞΑ - DΦC)W into DeLoRA's additive form W' = W + λ||W||/r(BΞΑ - DΦC), where DΦC remains frozen at BΞΑ's initial value. To facilitate comparison, we added a column in Tables 1 and 2 that details these different formulations.

We thank the reviewer for their feedback, and provide our response to each raised issue separately below.

[W1] In DeLoRA, treating the boundary λ as a learnable parameter will remove the guarantee of non-divergence etc, making DeLoRA a special case of LoRA

Since DeLoRA's weight updates are rank-r and impose stricter Euclidean distance constraints, its potential update space necessarily forms a subset of LoRA's update space.

However, in DeLoRA, by decoupling the update strength (with learnable parameter λ) from the angular learning (BA matrices), the resulting training dynamics lead to extremely different behaviors, as shown when evaluating Frobenius distances in Fig.3 (left). Plot in Fig.3 shows that LoRA and DoRA updates exhibit near-linear growth with each iteration, whereas DeLoRA updates rapidly approach the boundary value before gradually decreasing.

At the same time, while having a learnable boundary λ might seem to potentially lead to the same divergence issues of LoRA, thanks to DeLoRA’s decoupling this does not happen. This can be seen in the robustness plots in Fig.2 (previously Fig.4), where DeLoRA maintains optimal performance at learning rates up to 6e-1, while LoRA's performance degrades at just 2.4e-3.

In our updated manuscript we include an additional analysis on the Frobenius distance of a projection layer in an attention module at convergence, with respect to its pretrained counterpart (Fig.2, Right). These measurements reveal that while LoRA and DoRA diverge from the initial weights, DeLoRA remains stable despite having the capacity for larger deviations.

We further analyze DeLoRA’s robustness separating variations on boundary (λ0 and anglular components (BA), further demonstrating its robustness (Fig.6, Appendix D).

[W2] Given the similarities with DoRA, the differences should be stressed more. Plus, comparison with DoRA should be moved to Method section.

We thank the reviewer for the suggestions. We addressed this issue in the Global Authors Response above.

[Q1] In Table 1,2 how was the controllable boundary combined with LoRA?

The “normalization + controllable boundary” refers to the addition of the normalization matrix Ξ and of the scale λ to LoRA, which will determine the upper bound on the weight updates in terms of Frobenius distance. We refer to the boundary as controllable because its initialization can be arbitrarily tuned, to better fit the task at hand.

To improve clarity, in our ablations tables (Tab.1,2) we added a column describing all different formulations.

[Q2] In Fig 3, DeLoRA's distance to initialization was in fact decreased, how was this happened?

As derived from Section 2.2, Eq. 7, the λ parameter determines an upper bound, allowing the distance of the weight updates to be lower than the maximum value. Interestingly, DeLoRA finetuning dynamics show that the updates’ distance initially gets close the boundary and then, as training continues, it tends to decreases, significantly different behavior w.r.t. LoRA and DoRA dynamics.

[Q3] Can you elaborate more on Fig 4? I didn't find any discussion in the main body.

We thank the reviewer's observation regarding Fig.4 (now Fig.2), which we had inadvertently omitted from the manuscript. We have added a detailed description of the updated figure in Section 3.4 of the updated manuscript.

We report below here its description:

“We conducted a comprehensive learning rate robustness analysis in the setting of the Subject-driven Generation task of Section 3. Evaluation is done reporting DINO scores (Fig.2, Top) and Euclidean distance between finetuned and pretrained weights of an inner layer (Fig.2, Bottom) across multiple methods, using a range of learning rates derived from each method’s base learning rate (Fig.2, Left). Our analysis shows that DeLoRA is able to achieve the same robustness of ETHER+, while improving performance, while both LoRA and DoRA performance degrade at 4× the base learning rate. We also notice how LoRA updates’ distance grows at higher learning rates, while interestingly DoRA, after 8×, does not diverge further, likely due to its magnitude control. However this does not lead to better performance in these regimes.”

[Q4] Can you test the proposed method on more language tasks, such as math reasoning?

As suggested by multiple reviewers, we have now evaluated our DeLoRA method in finetuning for Natural Language Understanding on on the GLUE benchmark. We discussed this additional benchmark in the Global Authors Response above.

As the end of the discussion period approaches, we again thank the reviewer for their valuable feedback, which helped us improve our manuscript, and hope that our answers positively addressed the reviewer’s concerns: (W1) by clarifying how empirically DeLoRA does not lead to divergence, supported by new experiments and analysis, (W2) by extensively discussing differences with DoRA, also testing how DeLoRA achieves better decoupling with new experiment fixing DoRA’s magnitude, in Appendix C, and (Q4) by providing new results on the GLUE natural language understanding benchmark.. Plus, we hope to have fully answered reviewer’s questions (Q1, Q2, Q3), which helped us improve our manuscript by clarifying our ablation studies with the addition of mathematical formulations in Tab.1,2, and by creating a separate section for DeLoRA’s robustness analysis.

We take the chance to ask the reviewer if they have any further concerns, giving us the chance to address these too, and hope that our answers and the overall manuscript improvements were satisfactory. If so, we hope that the reviewer would consider raising their score.

[W- vsop, pmk9] Given the similarities with DoRA, the differences should be stressed more (vsop, pmk9)

We thank the reviewers for the suggestion. We agree that a more detailed comparison with DoRA could be beneficial to better highlight the substantial differences, given that both methods address the finetuning process with similar premises, i.e. by trying to decouple the learning of magnitudes and angles.

Following reviewer pmk9's suggestion, we added a new paragraph to the Methods section that clarifies the fundamental differences between the approaches, particularly in their implementation of normalization and scaling, which results in substantially distinct properties for each method.

We report the paragraph below, and refer for Figures to the updated manuscript:

“DoRA (Liu et al., 2024a), similarly to our work, addresses the finetuning process by trying to decouple the learning of magnitudes and angles, by using a formulation that leads to weight updates W ′ = m (W +∆W)/||W +∆W||.

We can summarize the key differences of DoRA with respect to our proposal into two: (i) the normalization and scaling operations happen on the fully finetuned weights, and (ii) these operations happen on the column space of the weight matrices, which draw a significant difference to our proposal.

We argue that DeLoRA finetuning (i) by introducing the normalization and scaling operations directly on the weight updates ∆W , it more directly tackles the goal of not diverging from the pretrained model, and (ii) by normalizing the inner low-dimensional space (rather than the column space), it actually results in an implicit Frobenius-distance boundary, which acts as a mathematical guarantee for non-divergence.

These eventually lead to (i) peculiar training dynamics (as shown in Fig.3, whereas DoRA and LoRA show similar behavior), and (ii) better decoupling, supported by the strong robustness results in Fig.2 (previously Fig.4).

In this regard, we notice that even if DeLoRA, by having a learnable boundary, in principle also has an unbounded Frobenius distance, in practice divergence does not happen, as shown in Fig.2 (previously Fig.4). This demonstrates that during finetuning, DeLoRA’s learnable boundary is able to effectively adjust and avoid divergence from the pretrained weights, behavior that does not happen with DoRA.”

[W- pmk9,vsop,EKmU] Paper would benefit from evaluating on GLUE benchmark (vsop, EKmU, pmk9).

We thank the reviewers for the benchmark recommendation. We now evaluated our proposed method by finetuning RoBERTa-base on the GLUE benchmark following [Wu M. et al.] setting, which was proposed to fairly evaluate GLUE on the publicly available validation set following best practices. Results are reported below (Table 4 in the updated version of the manuscript), while corresponding standard deviations and best hyperparameters are reported in Appendix B). Results for all baselines come from [Wu M. et al., Wu Z. et al.]. Our method achieves superior performance on CoLA (64.7%), QNLI, and STS-B, and attains a significantly higher average score that matches full finetuning results.

Experimental details: by following [Wu Z. et al., Wu M. et al.], we split the validation data in two subsets, and tune the hyperparameters on the smaller subset with seed 42. Then, we use the best validation hyperparameters, re-train on the training data for seeds {42,43,44,45, 46}, and get test results on the larger validation split. In addition, to be consistent with LoRA reported results, we only apply DeLoRA (rank 8) to Q and V matrices, which is likely underperforming with respect to applying lower-rank DeLoRA modules to more linear layer types (so with comparable number of parameters) [Hu et al.2022]

References:

[Wu M. et al.] Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Advancing parameter efficiency in finetuning via representation editing, 2024a

[Wu Z. et al.] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Reft: Representation finetuning for language models, 2024b

[Hu et al.2022] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.