On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation

Thank you for your constructive feedback and insightful comments. We hope that we can address your concerns with the responses below.

Q1: Comparison between Non-Linear and Linear prompt on HellaSwag and TruthfullQA with LLaMA-7B setting:

Thank you for your comments. Our main study in this paper is to focus on providing detailed theoretical analysis and experiments to understand the benefits of the zero-initialized attention over the random-initialized attention (conventional prefix-tuning approach) based on their connection to MoE models. Additionally, our analysis also indicates that non-linear prompts can be also optimally estimated as linear prompt with greater flexibility, suggesting that non-linear prompts are potential alternative to linear prompts.

To justify the potentials of non-linear prompts for LLaMA-Adapter models, we perform several experiments to compare the performance when using the linear prompts versus when using non-linear prompts. From these experimental results, we observe that non-linear prompts achieve higher performance in most settings (ranging from 0.5% to 4%), specifically across all results on LLaMA-13B and the first two datasets on LLaMA-7B. However, there are certain cases where non-linear prompts yield only comparable results, with performance differences ranging from 0.1% to 0.5% lower than the linear prompt settings.

This variation is expected because no single prompting method universally excels across all tasks and models. The effectiveness of non-linear prompts can depend on several factors, such as the dataset characteristics, model capacity, and the complexity of task-specific adaptations. In some cases, the additional expressivity of non-linear prompts may not be necessary, leading to performance that closely approximates linear prompts.

Q2: Comparison among several initialization methods:

Thank you for your suggestion. Indeed, we would like to clarify that the Random and Zero Initializations in the manuscript do not refer to the initialization of some vector. Let us take this opportunity to explain each of these initializations.

First of all, Random Initialization (equivalently, Random-Init) follows the conventional prefix-tuning approach, where traditional attention is applied to all tokens, including prompt vectors and previous output tokens, without incorporating the zero-initialization mechanism introduced in the original paper. The term "Random-Init" is used because, at the initial stage, all prompt vectors are randomly initialized. When combined with traditional attention mechanisms, this randomness affects the model’s convergence robustness. The terms "Random-Init" and "Zero-Init" are also used in the original LLaMA-Adapter paper.

On the contrary, Zero Initialization (in short, Zero-Init) setting introduces a learnable gating factor within the attention layers that use prompt, which is initially set to zero. Then, tanh activation function is applied to this factor to regulate the scale of this factor into [−1,1]. Additionally, separate softmax operations are applied to the attention scores of prompt tokens and word tokens independently, after which the gating factor is applied to the attention score of prompt tokens. This mechanism, as detailed in Equation (7) of the original LLaMA-Adapter paper, plays a crucial role in controlling the contribution of prompt tokens to the overall attention mechanism. By initializing this gating factor at zero, this factor can first eliminate the influence of under-fitted prompts at the early stages of training, allowing the model to gradually adjust its magnitude to incorporate meaningful instruction semantics into LLaMA.

Finally, in our study, we provide a detailed theoretical analysis and experiments to understand the benefits of zero-initialized attention over the random-initialized attention (conventional prefix-tuning approach). We also provide a theoretical analysis and experiments to show the effectiveness and flexibility of the Non-Linear prompt combined with the zero-initialized mechanism.

We hope our response answers your question about the initialization. Otherwise, please feel free to let us know, we are more than happy to address your further concerns.

Thanks for your constructive feedback and insightful comments. We would like to address your concerns as follows:

Q1: Explanation for results in Table 2.

Thank you for your comments. We want to clarify that most PEFT methods (e.g., LLaMA-7B + zero-init + linear, LLaMA-7B + zero-init + non-linear) perform worse than fully fine-tuning of LLaMA-7B, as shown in Table 2 (e.g., ARC dataset). While PEFT sometimes matches full fine-tuning, this is likely because the pre-trained model already contains fundamental knowledge for certain downstream tasks. However, in general, PEFT methods lag behind in performance compared to full fine-tuning.

However, it's important to emphasize that fully fine-tuning models like LLaMA is computationally expensive, making it impractical for low-resource settings. PEFT significantly reduces the number of learnable parameters while achieving comparable results, offering a more efficient alternative in such environments.

Q2: Additional analysis on other Parameter-Efficient Fine-Tuning (PEFT) techniques.

Thank you for your suggestion. To further validate our method, we conducted additional experiments comparing it with other PEFT methods, including Prompt Tuning [1], IA3 [2], and VeRA (r=128 and applied to the same modules as LoRA) [3], on LLaMA-7B setting. These comparisons provide a broader perspective on how our approach performs relative to established fine-tuning techniques. Additionally, we would like to clarify that the name Random-Init prompt in our study follows the traditional prefix-tuning approach. The results is presented in table below.

From our results, we observe that except Fully Fine-tuning setting, the Zero-Init approach combined with a non-linear prompt consistently outperforms other PEFT methods overall. This further point out the effectiveness of our method in achieving stability compared to traditional fine-tuning techniques.

[1] The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP, 2021

[2] Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. NeurIPS, 2022

[3] VeRA: Vector-based Random Matrix Adaptation. ICLR, 2024

Q3: The paper overlooks the Switch Transformer model.

Thanks for your comment. There seems to be a misunderstanding about the contributions of the paper. Our main focus is to study the LLaMA-Adapter for parameter-efficiently fine-tuning the LLaMA models rather than efficiently scaling LLMs with MoE models.

In particular, we study the LLaMA-Adapter, a PEFT method for LLaMA models. Since the LLaMA models do not replace feed-forward network layers with sparse MoE layers in their Transformer architecture, we do not consider any MoE-based Transformer variants, including the Switch Transformer. Instead, we establish a connection between the zero-initialized attention and MoE models in Section 3. We show that the zero-initialized attention can be represented as an MoE model. From that perspective, we demonstrate that using the zero-initialized attention with either linear prompts or non-linear prompts is more sample efficient than using the random-initialized attention (traditional attention).

Q4: The novelty of non-linear prompts.

Thanks for your comment. We would like to emphasize that our main contribution is to provide a theoretical study for understanding the benefits of the zero-initialized attention over the random-initialized attention based on their connection to MoE models. To the best of our knowledge, there had not been any similar studies in the literature prior to our work.

Furthermore, our analysis indicates that in addition to linear prompts in the original LLaMA-Adapter, non-linear prompts can be also optimally estimated with greater flexibility. Therefore, we perform several experiments to justify the efficacy of the LLaMA-Adapter with non-linear prompts. Although the idea of using non-linear prompts in PEFT methods may not be new, our paper is the very first work to propose employing non-linear prompts in the LLaMA-Adapter to enhance its performance with both theoretical guarantee and empirical evidence.

Thank you for your constructive feedback and insightful comments. We hope that we can address your concerns with the responses below.

Q1: Alignment of Theory and Experiments:

Thanks for your comments. We would like to clarify that the convergence analysis of prompt estimation under the random-initialized attention has been conducted in prior work (see Appendix C in [1] or Appendix A in [2]). Thus, in response to your concern, we will include the following comparison in the revision of our manuscript (below Theorem 4.2), indicating that using the zero-initialized attention is more sample efficient than using the random-initialized attention:

1. Prompt convergence in random-initialized attention [1,2]: The convergence rates of prompt estimation are significantly slow, standing at the order of $O(1/\log^{\tau}(n))$ for some constant $\tau>0$, where $n$ is the sample size. Thus, to approximate prompts with a given error $\epsilon$, we need exponentially many data points $O(\exp(\epsilon^{-1/\tau}))$, which is not sample efficient.

2. Prompt convergence in zero-initialized attention (Ours): As shown in Theorem 4.2 and Theorem 5.2 in our manuscript, the convergence rates of prompt estimation are of polynomial orders, ranging from $O(n^{-1/2})$ to $O(n^{-1/4})$. Therefore, we only need polynomially many data points to approximate the prompts with a given error $\epsilon$.

Hence, in the experiments, we conduct comparisons between Zero-init and random-initialized attention to validate the above theoretical analytic as presented in Table 1. Therefore, we believe that our work consistently aligns with our theoretical results.

[1] P. Akbarian et al. Quadratic gating functions in mixture of experts: A statistical insight. arXiv preprint, 2024.

[2] M. Le et al. Mixture of experts meets prompt-based continual learning. Advances in NeurIPS, 2024.

Q2: Discrepancy of LLaMA-Adapter on LLM benchmarks and ScienceQA (Multi-Modal):

The discrepancy in performance improvement between our paper and the original study on LLaMA-Adapter can be attributed to the differences in the datasets and task settings. The original paper only conduct ablation study to compare Zero-initialized Attention to Random-Init setting (without the zero-initialized mechanism) on the ScienceQA dataset, which is a multi-modal dataset used for Visual Question Answering. In contrast, we extend more experiments on language-only tasks, specifically fine-tuning on the Alpaca dataset based on the original code of LLaMA-Adapter paper. The Reviewer XuJQ also mention that in these language-only tasks, the performance gain in Table 1 is notable. This fundamental difference in task type can influence how the zero-initialized mechanism impacts model performance.

Moreover, ScienceQA consists of 21,208 questions, making model convergence slow and unstable when the zero-initialized mechanism is not applied. This issue is highlighted in Figure 7 of the original paper, where models without zero initialization struggle with robustness and efficiency. The multi-modal nature of ScienceQA, which involves both visual and textual inputs, further complicates the training process, making stabilization techniques like Zero-Init more beneficial.

For language-only tasks, we fine-tuned the Random-Init setting (without the zero-initialized mechanism) on the Alpaca dataset, which contains 52,000 samples. Although the Random-Init setting led to unrobust convergence in sample efficiency, as shown in Figure 2 and 3 in our paper, the final convergence values when fine-tune the Random-Init setting on 100% Alpaca dataset remained sufficiently low compared to the Zero-Init setting. This indicates that when fine-tuning a random-initialized model on a sufficiently large language-only dataset, the model can still achieve reasonable stability despite the slower convergence.

Overall, the smaller performance gap observed in our study compared to the original paper can be explained by the difference in dataset size and modality. While Zero-Init plays a crucial role in stabilizing multi-modal training on ScienceQA, its impact is less pronounced in large-scale language-only fine-tuning, where instability can be mitigated to some extent by the sheer volume of training data.

Q3: Fragmentation of Sections:

Thanks for your suggestion. We agree that both Section 4 and Section 5 focus on the convergence analysis of prompt estimation. Therefore, in the revision of our manuscript, we will merge them into one big section by relabeling Subsection 5.1 and Subsection 5.2 as Subsection 4.2 and Subsection 4.3, respectively.