WeGeFT: Weight‑Generative Fine‑Tuning for Multi‑Faceted Efficient Adaptation of Large Models

Dear reviewer Zn72,

Thank you for your feedback and efforts in reviewing our submission. We address your concerns as follow, which will be carefully incorporated in the revision.

Experimental designs, especially for using what foundation models and on what datasets, are not logically clear

We follow prior works [1, 2, 3] in selecting foundation models and datasets, all of which are standard in the PEFT literature. We appreciate any suggestions on how to further clarify our choices.

The claimed scalable performance is not validated by experiments

We have evaluated WAFT at scales of 7–8B parameters on large fine-tuning datasets for arithmetic reasoning, code completion, commonsense reasoning, and instruction following, all of which are tasks of practical interest. Based on these results, we believe our experiments demonstrate scalability. However, we welcome further insights on how to strengthen this validation.

References

[1] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Ka-Wei Lee: LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models. EMNLP 2023: 5254-5276

[2] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, Christopher Potts: ReFT: Representation Finetuning for Language Models. NeurIPS 2024

[3] Shaowen Wang, Linxi Yu, Jian Li: LoRA-GA: Low-Rank Adaptation with Gradient Approximation. NeurIPS 2024

Dear reviewer M1uG,

Thank you for your feedback and efforts in reviewing our submission. We address your concerns as follow, which will be carefully incorporated in the revision.

Comparison with FacT

We acknowledge that Factor Tuning (FacT) shares a similar parameter-sharing concept with WAFT. However, unlike WAFT, FacT shares parameters across both layers and modules (Self-Attention and MLP blocks). While this reduces trainable parameters, it has a drawback: Transformer modules serve distinct functions - attention layers largely handle syntactical and in-context learning abilities [1, 2], while MLP layers encode factual knowledge [3]. Sharing parameters across these modules may therefore be suboptimal.

For a comprehensive evaluation, we compare FacT on the VTAB and Math10k benchmarks. We use the open-source implementation from the original paper for VTAB, employing the same model as in our FGVC experiments. For Math10k, we adapt FacT to the HuggingFace PEFT package and evaluate it on LLaMA-1 and LLaMA-2 (7B). The table below shows that WAFT outperforms both FacT variants on VTAB while using fewer parameters.

WAFT outperforms FacT on the Math10k benchmark, with a larger performance gap on LLaMA-1 than Llama 2. Since LLaMA-1 is weaker and requires more adaptation for arithmetic reasoning, this highlights WAFT’s ability to adapt both weak and strong models effectively, whereas FacT struggles with weaker models.

Effectiveness of the identity transformation

While we do not have a theoretical understanding yet, we hypothesize that the superior performance of the identity operation over more complex and non-linear operations is because of difficulty in optimization.

---

References

[1] Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Hervé Jégou, Léon Bottou: Birth of a Transformer: A Memory Viewpoint. NeurIPS 2023

[2] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, Ivan Titov: Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

[3] Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov: Locating and Editing Factual Associations in GPT. NeurIPS 2022

Dear reviewer NsTU,

Thank you for your feedback and efforts in reviewing our submission. We address your concerns as follows, which will be carefully incorporated in the revision.

Performance in constrained environments

We use an A100 GPU to accelerate experiments, but the implementation is not tied to any specific GPU architecture. WAFT’s memory and wall-time requirements are comparable to or lower than LoRA, while VeRA and DoRA significantly increase both. Table 2 in the manuscript provides a detailed analysis, confirming WAFT’s suitability for any device that supports LoRA.

For further verification, we conduct additional experiments with LLaMA-1 (7B) using mixed-precision float16 instead of mixed-precision bfloat16. The table below shows that WAFT and LoRA experience a similar relative performance drop with float16 while maintaining comparable memory and wall-time, consistent with float16’s known limitations as compared to bfloat16.

The performance drop due to float16 can be offset by increasing the number of trainable parameters in WAFT. As shown in the table, WAFT with a rank of 128 outperforms LoRA even with float16, while using four times fewer parameters. These results further confirm WAFT’s compatibility with any device that supports LoRA.

Generalization to other architectures

We recognize the importance of exploring generalization. However, Transformers have proven to be highly scalable and flexible, and have emerged as the standard architecture due to their scalability and adaptability. Consequently, both our study and most prior PEFT research focus on Transformers. We believe extending WAFT to other architectures is a valuable future direction, and its strong performance in the context of prior research on fine-tuning Transformer models makes it a meaningful contribution.

Dear reviewer UHSy,

Thank you for your feedback and efforts in reviewing our submission. We address your concerns (C1-C5) as follows, which will be carefully incorporated in the revision.

C1: Weight-awareness in LoRA and WAFT

In general, any PEFT methods have to be (pretrained) weight-aware to be effective for downstream tasks. However, prior art like LoRA has implicit weight-awareness between the fine-tuning residual ($\Delta W^l = B^l\cdot A^l$) and the pretrained weights ($W^l$) via gradient backpropagation in learning (see Eqn.10 for a toy example in Appendix A). Our proposed WAFT harnesses explicit weight-awareness (in the forward computation) via learning $\Delta W^l = W^l\cdot \phi \cdot \psi$. In the abstract (lines 025-027), we define WAFT, a novel approach that learns to generate fine-tuning weights directly from the pretrained wegiths. (the generic formulation is in Sec.3.1). To address your concerns, we propose to use generative weight-aware fine-tuning (GenWAFT) to better highlight the characteristics of our proposed method in revision. The comprehensive experiments have verified its effectiveness.

C2: Training time for VeRA

VeRA's long training time results from the large intermediate dimension used in our experiments. Since VeRA relies on fixed random matrices, a large dimension is necessary for reasonable accuracy. As shown in the table below, reducing the dimension to 1024 (as in the original paper) significantly degrades performance, despite lowering training time and parameter count. In contrast, our proposed WAFT maintains parameter and training efficiency while achieving comparable or superior performance to state-of-the-art methods.

C3: Effect of weight-aware finetuning in cases of weak zero-shot performance of models

This is certainly a valid concern, especially when evaluating methods that reduce parameter count. Our experiments on the Math10k benchmark with LLaMA-1 (7B) address this issue (Table 2). Given LLaMA-1 7B’s low zero-shot accuracy of 11.0 on GSM8k [1], it can be considered weak for arithmetic reasoning which would require significant adaptation to perform well on Math10k. The results show:

• Simply reducing parameters by lowering the rank in LoRA and DoRA leads to a significant performance drop. LoRA's accuracy with rank=16 is 50.9 vs. 48.9 with rank=2. With an equivalent parameter count (~0.05%), WAFT outperforms all baselines (LoRA, DoRA, VeRA, and VB-LoRA), and achieves accuracy close to LoRA $^{r=16}$. This demonstrates that generating the fine-tuning residuals from the pretrained weights improves adaptation with minimal parameters, whereas LoRA $^{r=2}$ struggles to achieve good performance.

We have also included further experiments with the VTAB benchmark as suggested. We follow the same settings as the FGVC experiments (Section 4.5). The table below shows that WAFT can perform on par or better than baseline methods even on Specialized and Structured tasks on the VTAB benchmark.

C4: More exploration could be done on the relevance of using the pre-trained weight for the matrix multiplication, for example is a properly scaled random matrix performing close to the pretrained weights?

We are not certain that we fully understand your concern. Were you suggesting to test a variant like this: converting WAFT update from $\Delta W^l= W^l\cdot \phi\cdot \psi$ to $\Delta W^l= \mathbb{W}^l\cdot \phi\cdot \psi$ in learning fine-tuning residual weights, where $\mathbb{W}^l$ is ``a properly scaled random matrix"? It seems very unlikely that a random matrix can do better than the pretrained weights. With the random matrix, the overall fine-tuning residual weights $\Delta W^l$ would behave similar in spirit to VeRA. Could you kindly elaborate your concern?

C5: Ablations on using other types of matrices for the multiplicative updates

The purpose of our simple formulation is that it enables memory and computationally efficient updates, since the low-rank structure and linear updates result in smaller matrix multiplications. Our ablations studies (Table 7) show that more complex updates are not necessary. Hence, we leave the further exploration to future work, and we would welcome any suggestions from the reviewers and the broader community.

---

[1] Touvron et al., "LLaMA: Open and Efficient Foundation Language Models", ArXiv 2023