Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation

Thanks for your appreciation of our work and constructive comments. Below, we try to resolve your concerns one by one.

Q1: Improve the structure of the paper and highlight the motivation for building a unified adaptation framework.

A1: We have detailed the pros and cons of LoRA and OFT in the Related Work section. Regarding their advantages, LoRA assumes that weight changes during model adaptation have a low "intrinsic rank," while OFT preserves pre-trained knowledge by maintaining the pairwise angles between neuron vectors. As for their drawbacks, LoRA cannot ensure the preservation of angles between neuron vectors, and OFT can only achieve low-rank weight updates when $(R−I)$ is low-rank. HRA combines these two strategies, leveraging their advantages jointly while suppressing their drawbacks simultaneously.

Moreover, building a unified adaptation framework is insightful for revisiting the recent rapid development of various adaptation methods, which may help inspire new technical routes. We have placed an overview of the motivation, novelty, and organization at the end of the Introduction. We plan to polish our paper in the final version further. Thanks for your suggestion.

Q2: How to set r for real applications?

A2: We have discussed the setting of $r$ in the above general response, which may have resolved your concern to some extent. In this study, we set $r\leq8$ for most datasets and tasks because this setting is sufficient to achieve superior adaptation results with fewer trainable parameters. Therefore, we empirically recommend starting with $r=8$. If the model performance is not satisfactory, increasing $r$ will usually work better. As we mentioned in the above general response, like AdaLoRA, we will consider adjusting the $r$ of HRA adaptively as our future work, but it is not the main contribution of this paper.

Q3: The training and inference time of HRA.

A3: We have shown the training time and computational efficiency of HRA in the general response. After training, we multiply the learned orthogonal matrices with the weight matrices, leading to a new model without increasing parameters. Therefore, like LoRA and OFT, HRA does not change the model's inference time.

Q4: In section 4.3, which part of the model is applied with the proposed HRA? Does each layer have a similar r?

A4: We follow the experimental settings of LoRA and OFT for a fair comparison. For the stable diffusion model, we apply HRA to its attention modules. In this study, we apply the same $r$ to the weight matrices of each attention module. Note that such a simple setting has resulted in superior performance. In the future, we can leverage the same idea of AdaLoRA, adjusting $r$ for different layers.

Q5: Reference about OFT. Citation 33 was published in 2023.

A5: Thank you for pointing out this mistake. We will correct it in the revised paper.

Thanks for your appreciation of our work. We have resolved your concerns about the limitations and the computational efficiency of HRA in the above general response. For your remaining concerns, we provide our answer below.

Q1: The evidence on the retention of prior knowledge.

A1: Thanks for your constructive suggestion. To verify our claim, we fine-tune LLaMA-2 7B on the MATHQA dataset by LoRA and HRA, respectively, and check the degradation of model performance on classic NLP tasks, including typical language tasks in ARC, HellaSwag, MMLU, and Winogrande, and a coding task in HumanEval. For a fair comparison, we apply the same number of trainable parameters and the same batch size for LoRA and HRA. Ideally, after adaptation, we hope that the model can still maintain its high performance in the NLP tasks.

According to the above results, we find that compared to LoRA, HRA retains more of the original model's knowledge, whose performance degradation is less severe than LoRA's. In the HumanEval task, its performance is even better than that of the original model (which we think is because the MATHQA dataset contains many samples relevant to logic and reasoning tasks and thus is useful in the HumanEval task).

We will add this experimental result in the final paper. Thanks again for your suggestion.

Thanks for your comments. Below we try to resolve your concerns one by one.

Q1: The novelty of the proposed method.

A1: We respectfully disagree with the comment that our work's novelty is relatively limited for the following three reasons. Firstly, to our knowledge, our work makes the first attempt to bridge the gap between low-rank and orthogonal adaptation techniques. Our HRA is a new implementation of OFT. At the same time, it is also an adaptive LoRA, as we mentioned in Section 3.3. Secondly, existing OFT restricts the interactions between different dimensions of the weight matrix due to the block diagonal structure of its orthogonal matrix, while BOFT overcomes this issue by time-consuming butterfly matrix multiplication. Unlike these two methods, our HRA implements a chain of Householder reflections to implement orthogonal adaptation, as discussed in Section 3.2. This implementation leads to better efficiency and performance. Thirdly, focusing on HRA, we analyze the impact of the orthogonality of reflection planes on the adaptation performance (Section 3.4), proposing an orthogonal regularizer in the training phase, which can further boost the adaptation performance.

We believe all three of the above contributions are new to the research community, and thus, the novelty of our work is sufficient. Actually, in your first comment in the Strengths session, you admitted the novelty of our work. Therefore, we hope that the above response helps you reconsider our work.

Q2: The reasons for the superiority of HRA compared with LoRA and OFT are not explained.

A2: In fact, we have explained the reasons for the superiority of HRA. In particular, in Section 3.2, we compared HRA with OFT and BOFT on their implementations and computational complexity. As shown in Lines 147-148 and 157-158, we have shown that HRA can be more efficient (i.e., using fewer trainable parameters) in a mild condition, which is easy to meet in our experiments.

Regarding the retention of pre-training knowledge, we have shown in Lines 168-172 that HRA can preserve the angular information of weight matrices like OFT and BOFT do, which is better than LoRA.

In addition, by introducing an orthogonal regularizer during adaptation, we control the orthogonality of reflection planes and thus help achieve a trade-off between the model capacity and regularity. Compared to LoRA and OFT, this mechanism can further reduce the risk of over-fitting, which helps further boost our performance. We plan to add this point to the end of section 3.4.

Q3: More experimental results with other values of $\lambda$.

A3: We conduct the experiments with other values of $\lambda$, and the results are as follows.

We can find that a) the performance of HRA is relatively stable concerning the change of $\lambda$, b) in the wide range of $\lambda$, HRA is superior to the baselines, c) even if ignoring the regularizer $(\lambda=0$), our method still outperforms the baselines, which demonstrates the effectiveness of implementing orthogonal adaptation based on Householder reflections.

Q4: Will the orthogonal regularizer also improve LoRA and OFT?

A4: This is an interesting question. Firstly, it should be emphasized that our comparison experiments are fair because, in all Tables and Figures, we have considered the HRA with $\lambda=0$ (i.e., without the orthogonal regularizer) and compare it with the baselines. The experimental results show that HRA can outperform LoRA, OFT, and other competitors even without the regularizer. In other words, the superiority of our method is mainly from the proposed Householder reflection chain, and the proposed regularizer can further boost performance.

Secondly, it is nonsense to apply the orthogonal regularizer to OFT because the block diagonal structure of its orthogonal parameter matrix has ensured that the columns of the matrix are orthogonal to each other. For BOFT, the columns of different butterfly orthogonal parameter matrices are also orthogonal to each other. In other words, OFT and its variants already have intrinsic and strict orthogonality constraints.

Finally, for the vanilla LoRA in the formulation $W+AB$, we can impose our orthogonal regularizer on its parameter matrices, which was never considered before. Although our regularizer is motivated by the orthogonality of reflection planes rather than LoRA, to resolve your concern, we imposed our regularizer on the matrix $B$ of LoRA and tested it in the mathematical reasoning task.

The above results verify our claim --- the superiority of HRA is mainly caused by the Householder reflection chain. Note that, without a strong theoretical motivation like the orthogonality of reflection planes, we have too many options to combine LoRA with our regularizer, e.g., imposing our regularizer to $A$, $B$, $AB$, and so on. Considering so many variants of LoRA is out of the scope of this work.

We hope the above responses can resolve your concerns. We are willing to discuss with you in the next phase if you have any other questions.

We are grateful that our response has positively impacted the rating. If you have any further comments or questions, we would be happy to address them.

Thanks for your appreciation of our work. We believe your concerns have been resolved in our general response, and we hope that our response can help increase your confidence score. We are willing to discuss with you in the next discussion phase if you have any other questions.