HoLoRA: Combining Orthogonal Fine-Tuning and LoRA with Householder Reflectors

1. All the theorems and lemmas do not have proofs.
2. As shown in Eqs. (15-17), HoLoRA also increases computational costs, which is contrary to the presentation in Line 20, “without increasing computational costs”.
3. First, the author employed a bunch of fancy mathematics to derive an orthogonal function in the end. However, these mathematical proofs were not originally proposed by the author; in fact, the author merely used an existing orthogonal tool. Second, to keep A orthogonal and without introducing a regularizer, why not using Cayley Parameterization? It is more easier and computation efficient. One can learn A, B and let LoRA be \delta W = B (I+A)(I−A)^{−1}. Third, there lacks ablation studies with other orthogonal methods like Cayley Parameterization.
4. The most critical issue with this paper is the lack of experiments. Currently, PEFT methods are typically tested on large models such as LLaMA-7B. Even taking into account the possibility that the authors may have limited GPU resources, the NLU experiments only include a comparison between two different parameter sizes. Moreover, considering that NLU experiments are susceptible to the influence of random seeds, it is unclear whether the results reported by the authors are the averages of multiple experimental runs.
5. Line 405-406, ”and has a total parameter size of 86M”. The total parameter size of DeBERTaV3-base should be 184M.
6. The paper misses a subsection: an explanation of why the orthogonality constraint can mitigate feature loss. While the authors state that it can, they do not provide a clear rationale or reasoning behind this claim.
7. Regarding weakness 6, there is no experiment validating that HoLoRA can decrease the feature loss.

1. Orthogonal fine-tuning (OFT) has been the subject of extensive research in recent years. Although this paper introduces a method based on the Householder reflector, its contributions appear to be closely aligned with existing studies, resulting in a novelty that is somewhat limited.
2. A thorough comparison with previous OFT algorithms is warranted. Specifically, it remains unclear whether the proposed Householder method, which enforces hard orthogonality, demonstrates superiority over approaches that utilize regularization techniques for soft orthogonality. Additionally, the authors should discuss the computational and memory requirements associated with their method.
3. The authors’ comparisons in Table 1 are primarily limited to classical methods. Recent advancements, such as DoRA, propose alternative strategies by decoupling the magnitude and direction of LoRA, suggesting that enforcing orthogonality is not the sole approach to enhancing overall performance.
4. The comparison of the proposed algorithm to LoRA is restricted to experiments conducted on DeBERTa with relatively small datasets. It would strengthen the findings if the authors tested their approach on larger backbone models to evaluate its scalability and effectiveness.
5. Several equations in the paper appear to contain inaccuracies. For instance, H(p(u)) is not equivalent to H(\tilde{u}) as stated in Equation (8). A thorough review of these equations is recommended to ensure correctness.
6. The axis labels in Figure 2 are too small, making them difficult to read.

1. The experiments focus solely on the DeBERTaV3-base model, without testing on other pre-trained models (e.g., GPT, BERT, T5). This limited scope makes it difficult to assess HoLoRA's effectiveness across different model architectures.
2. The evaluation is based on the GLUE benchmark, primarily covering natural language understanding tasks. HoLoRA’s performance on other types of tasks, such as generative or multimodal tasks, remains untested, potentially limiting its applicability.
3. While HoLoRA ensures strict orthogonality, the paper does not explore in detail how this constraint affects feature representation, model stability, or training dynamics. A deeper analysis could provide more insight into HoLoRA’s impact.
4. While HoLoRA does not introduce new hyperparameters, factors like low-rank parameter r may still affect results. The lack of sensitivity analysis for various parameter configurations leaves questions about its robustness and tuning flexibility.
5. While the paper claims compatibility with other LoRA variants, empirical tests on these combinations are absent, leaving uncertainty about potential optimization challenges or performance trade-offs when integrating with other methods.
6. The paper claims that HoLoRA has advantages over other LoRA variants that also explore orthogonality, such as OLoRA. However, it avoids direct comparison with these methods, making it difficult to be convinced of HoLoRA’s performance and time efficiency benefits.

Householders have been used in numerical computation of operations on orthogonal matrices. Therefore, the proposed method is a trivial application of Householders with orthogonal constraints. Therefore, the novelty of the method is limited.

In order to improve the novelty, the proposed HoLoRA should be compared with all the related work conceptually, theoretically and experimentally.