From Instance Training to Instruction Learning: Task Adapters Generation from Instructions

Dear reviewer 2ySK:

We greatly appreciate your professional review of our article. Here is our feedback.

Response to W1:

W1. Because the proposed method underperformed the standard fine-tuning...

1. Training Phase: Gradient computations for each sample are unnecessary. Our method aims to minimize training costs for new tasks. Unlike traditional methods that require extensive labeled data and gradient updates, our approach bypasses gradient updates by directly generating task model parameters from the task description.

2. Inference Phase: Instruction processing is only required once. We agree with the reviewer's point about our method's reduced inference costs: by processing the instruction a single time, we cut down on both time and computational expenses. However, it appears there may be a misunderstanding—our method's time and self-attention costs for a single example are similar to those of standard approaches since the instruction isn't repeatedly processed.

The primary cost savings of our method are evident when inferring multiple instances for the same task. For $n$ inference instances, our method incurs a computational cost of $L^2 + n \cdot (M^2 + LM)$, whereas the standard self-attention method costs $n \cdot (L + M)^2$. Our approach is most effective when handling more instances of the same task with sufficiently lengthy instructions.

To compare inference costs, consider processing $n$ samples of length $i$ with a task instruction of length $t$, ignoring the output sequence length. Let $N$ represent the number of FLOPs per token for an encoder-decoder model. The total computational cost for full fine-tuning is $FLOPs_{\text{standard}} = N \cdot n(t + i)$. For TAGI, the cost is $FLOPs_{\text{TAGI}} = N \cdot (t + ni)$. It can be seen that the effectiveness of TAGI is significantly increased with the increase of n and t.

Due to length limitations, we reported the analysis of length in the global author rebuttal Q1.

Response to W2:

W2. This paper lacks the most important ablation study. Specifically, Table 5 ...

Generating LoRA weights from the hypernetwork is central to our parameter-efficient fine-tuning approach. The TAGI model converts instructions and task-specific data (like few-shot examples) into efficient modules within the pre-trained language model. This not only reduces inference costs but also boosts few-shot learning in instruction-based meta-learning. Furthermore, our method can be adapted to create other efficient modules, such as adapters and prefixes, for fine-tuning.

For the ablation involving cross-attention, we combined it with the instruction for the experiment, as indicated by "Instr. Fus." in Table 4. However, during our experiments, we observed that the cross-attention module's contribution was limited, and as such, we did not report it separately.

We have included the following ablation experiments: the LoRA results for Tk-Instruct, and the outcomes of our method without the hypernetwork, directly using LoRA. The results are as follows:

The results in the last row show a significant performance drop without the hypernetwork, highlighting its crucial role in cross-task generalization. This is consistent with Table 6 of the HINT study [1]. The hypernetwork enhances task understanding by dynamically generating efficient module parameters from instructions, rather than just embedding knowledge, as with LoRA. This dynamic adaptability improves cross-task generalization. We will further explore the hypernetwork's advantages in the appendix.

[1] HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation, ACL 2023

Response to W3:

W3. This paper lacks implementation details. More importantly, the proposed...

We have detailed the experimental procedures in Section 4.3 and Appendix A.4, and the relevant code has been submitted. The hypernetwork described in Section 3.2.1 (L-237) uses a 2-layer MLP. We will address it by updating the appendix accordingly.

parameters Costs

The T0/Tk-Instruct baselines represent full fine-tuning, as illustrated in Figure 2. Our approach involves updating only the hypernetwork and generating parameters for efficient fine-tuning modules. Adding cross-attention does not significantly increase the number of fine-tuning parameters since it is incorporated only into the even layers of the encoder. The additional fine-tuning parameters are as follows:

Memory Costs

Let $l$ be the number of layers, $d$ the model hidden dimension, $h$ the number of heads, $k$ the size of the keys/values, and $s$ the sequence length to be saved.

In the default TAGI settings, we store two main components: the processed instruction sequence with $ds$ values (one vector per token), and the LoRA weights, totaling $2 \times 32 \times l \times h \times k \times 2$ values (32-rank LoRA for both key and value per layer and head). The total memory cost is $ds + 128 lhk$. With $kh = d$, the memory cost of TAGI scales proportionally with $ds + ld$. Note that the default LoRA rank is 32 but can be adjusted to manage memory costs.

Dear Reviewer 2ySK:

We would like to thank you again for your detailed reviews. We hope that we have satisfactorily addressed your concerns. The deadline for the rebuttal period is approaching, and we sincerely hope to receive your comments and suggestions soon. Please feel free to let us know if there are any further issues we can address.

Respectfully,

Authors

Thank you for your follow-up comments and for acknowledging the clarifications provided in our response.

Response to W1:

W1. regarding effectiveness (the RougeL score): In comparison with Tk-Instruct in Table 2, there was little improvement. Although TAGI outperformed T0 in Table 3, it may be due to the knowledge distillation from Tk-instruct because TAGI used Tk-instruct, which is a refined version of T0, as a teacher model. That is, TAGI would not be effective in comparison with the standard instruction tuning.

In Table 2, our method outperforms similar hypernetwork-based approaches, surpassing the previous best method HINT by more than 1.5. Although the improvement in the 'Def' case for the traditional full fine-tuning method Tk-Instruct is modest, our approach uses fewer fine-tuning parameters, training resources, and less training time. In the 'Def + 2 Pos.' case, our method significantly outperforms Tk-Instruct, except for the 11B model.

The results in Table 3 do not pertain to Tk-Instruct. Instead, we fully meta-trained a model, replacing 'metatrain,' which incorporates versions of two different models: T5-lm and T0. From Table 3, it is evident that our method surpasses existing results for meta-training, including multi-task standard instruction tuning and all TAGI results. Additionally, our training parameters and overhead are considerably reduced, leading to a 10% reduction in computational requirements for inference.

Additionally, in the ablation study shown in Table 4, the T5-XL model achieves a score of 57.3 even without knowledge distillation (w/o L_kl), which surpasses the metatrain result of 53.1.

Response to W2:

W2. regarding efficiency (inference cost): I understood that there is an advantage when there are multiple samples. On the other hand, even with standard self-attention, it is possible to decode multiple samples efficiently by using cache KV if the model is decoder-type. L237 states `We limit our scope to encoder-decoder models for our experiment'. However, it is misleading to state that the scope of the experiment is limited. I consider that the scope of the research (or, the usefulness of the proposed method) is limited to the encoder-decoder-type and this should be discussed as the limitation.

Thank you for highlighting this issue. We acknowledge it as a limitation and have discussed it in Appendix B.1. Our approach aimed to compare with prior work, and therefore, we followed the settings of those studies. However, contemporary decoder-only models such as LLaMA3, QWen2, and other advanced models have already undergone extensive QA for Supervised Fine-Tuning (SFT), serving as robust baselines. Additionally, the continued improvement in model performance using efficient parameter fine-tuning methods like LoRA on decoder-only models suggests the feasibility of generating efficient fine-tuning modules via hypernetworks. Even though the use of KV-cache can reduce computation, it does not undermine our method's initial goal of encoding instructions only once to minimize computational load from the input side. In Table 2, we used the results from the HINT paper for GPT-2 XL (1.5B) and OPT (13B), which perform lower than encoder-decoder models of the same caliber. However, these results do not represent the most advanced decoder-only models available today.

Next, let's analyze the KV cache.

Here, we will let $l$ be the number of layers, $d$ the model hidden dimension, $h$ the number of heads, $k$ the size of the keys/values, and $s$ be the length of the sequence we want to save. We ignore biased terms for simplicity.

For decoder-only models, if we want to cache the key/value pairs for a given sequence, we will store $2lhks$ values - a key and value for every head in every layer, for each item in the sequence. Assuming $kh = d$, the memory cost of decoder-only models is proportional to $lsd$.

In the default TAGI settings, we store two main components: the processed instruction sequence, which consists of $ds$ values (one vector per token), and the LoRA weights, totaling $2 * 32 * l * h * k * 2$ values (with a 32-rank LoRA for both key and value per layer per head). The total memory cost is therefore $ds + 128 lhk$. Note that the default LoRA rank is set to 32, but this parameter can be adjusted to manage memory costs. Assuming $kh = d$, the memory cost of TAGI is proportional to $ds + ld$.

In conclusion, although decoder-only models can save computations through KV caching, our method only computes the instruction once, directly reducing the processing length at the input end. Therefore, it has better scalability in terms of sequence length (larger $s$) and model size (larger $d$, $l$).

Thank you for offering such a thorough response to my concerns and additional results. I understood that TAGI is not comparable to Full-FT on the target task because TAGI is not fine-tuned on the target task. I raised the score accordingly. On the other hand, I have the following concerns:

W1. regarding effectiveness (the RougeL score): In comparison with Tk-Instruct in Table 2, there was little improvement. Although TAGI outperformed T0 in Table 3, it may be due to the knowledge distillation from Tk-instruct because TAGI used Tk-instruct, which is a refined version of T0, as a teacher model. That is, TAGI would not be effective in comparison with the standard instruction tuning.

W1. regarding efficiency (inference cost): I understood that there is an advantage when there are multiple samples. On the other hand, even with standard self-attention, it is possible to decode multiple samples efficiently by using cache KV if the model is decoder-type. L237 states `We limit our scope to encoder-decoder models for our experiment'. However, it is mis-leading to state that the scope of the experiment is limited. I consider that the scope of the research (or, the usefulness of the proposed method) is limited to the encoder-decoder-type and this should be discussed as the limitation.

In general, I am concerned that the effectiveness of the proposed method does not improve the standard instruction tuning, and that the efficiency of the proposed method is limited to encoder-decoder type models. In other words, I am concerned that TAGI may provide neither effectiveness nor efficiency in instruction-tuning of the current mainstream decoder-type transformer.

Dear reviewer Wqw9:

We greatly appreciate your professional review of our article. Here is our feedback.

Response to W1:

W1. Limited model size exploration: The authors primarily focus on models up to 3B parameters, with only limited experiments on an 11B model. This leaves questions about how well TAGI scales to larger models, which are increasingly common in state-of-the-art NLP. The authors acknowledge this limitation but could strengthen their work by solving the problem.

We fully acknowledge this concern, as highlighted in the limitations section. Due to resource constraints and the challenges associated with training larger models, our results may not be as comprehensive as desired. We plan to refine our methodology and extend the scope of our experiments in future work.

Response to W2:

W2. The narrow focus on encoder-decoder models: The paper exclusively uses encoder-decoder models (T5-LM and T0), citing their superior cross-task generalization. However, this limits the generalisability of their findings.

Thank you for your insightful comments, which align with our own concerns. In Appendix B.1, we explained our rationale for selecting Encoder-Decoder models. Encoder-decoder (enc-dec) structure has advantages over decoder-only (dec-only) models regarding task generalization and instruction-following capabilities.

We also recognize the potential of current decoder-only large models (such as Llama3, Qwen2) that have undergone extensive fine-tuning on instruction tasks, potentially serving as a strong baseline similar to TK-Instruct. However, the effectiveness of our method on these models remains to be determined through further experimental evaluation.

In the near future, we plan to conduct experiments with the latest decoder-only large models to assess their applicability.

Response to W3:

W3. Limited analysis of instruction quality: While the paper focuses on learning from instructions, there's little discussion on how the quality or format of instructions affects performance.

We primarily explored the impact of the number of instructions on performance, as shown in Figure 3. Under our experimental settings, the quality and format of the instructions, which are optimal ("gold"), play a crucial role. The main experiment results in Table 2 show that methods based on hypernetworks do not perform as well as Tk-Instruct which also hints at the importance of instructions to complete tasks. Furthermore, Table 4 demonstrates a significant decline in performance in our ablation experiments when instruction fusion is omitted (w/o Instr. Fus.), underscoring the importance of instructions.

Response to W4:

W4. Limited exploration of hyperparameter sensitivity: While the authors provide some ablation studies, a more comprehensive analysis of hyperparameter sensitivity would strengthen the paper. This could include: a) Exploring the impact of different LoRA ranks beyond the chosen value of 32; b) Analysing how the size of the hypernetwork affects performance and efficiency; c) Investigating the trade-offs between pretraining steps and finetuning steps;

Thank you for the reviewers' valuable feedback.

Firstly, in response to point a), we present the impact of varying LoRA ranks on performance in Table 7 in the appendix. We selected a rank of 32 based on a balance between performance and computational overhead.

Regarding point b), we employed a two-layer MLP as the supernetwork for efficiency. While exploring different supernetwork sizes, such as larger models like T5 or GPT-2, might yield additional insights, it would significantly increase overhead, as observed in prior work.

For point c), Table 7 in the appendix details the effects of different fine-tuning steps on performance.

Dear reviewer UQYV:

We greatly appreciate your professional review of our article. Here is our feedback.

Response to W1:

W1. While I do appreciate the meta-train varying experiments, the distribution shifts of tasks in meta-train or test datasets seem to be consistent (i.e., I did not find explicit evidence in the manuscript saying that some of the meta-test tasks are altered). Experiments on a few trials of the train-test splitting combinations are worth investigating to provide a more comprehensive analysis (with mean and variance).

As demonstrated in Tables 10 and 11, using the P3 dataset as an example, the meta-test tasks include Sentence Completion, Natural Language Inference, Coreference Resolution, and Word Sense Disambiguation. In contrast, the meta-train tasks consist of Multiple-Choice QA, Sentiment Analysis, Structure-to-Text, and Summarization, among others. The distributions of these tasks are not expected to be consistent.

Figures 3 and 4.5 illustrate the effects of varying the number of meta-training tasks on performance. We will include mean and variance in the results in future updates.

Response to W2:

W2. Why is there no comparison between TAGI to prefix-based [1] methods?

Certainly, during our experiments, we attempted to use generated prefixes for fine-tuning, but the results were not as promising as expected. We also explored generating prefixes alongside LoRA, but the benefits remained minimal. This may be due to potential issues in our implementation, which we have yet to identify. We appreciate your feedback and will revisit the experiment with prefixes to address this issue. Thank you for highlighting it.

Here, we provide supplementary experimental results for LoRA fine-tuning (rank=32)(Tk-Instruct-LoRA) and prefix fine-tuning (num_virtual_tokens=32) (Tk-Instruct-prefix), as well as results of TAGI from replacing the hypernetwork with LoRA (w/o Hypernetwork):

We observe that the prefix performs exceptionally well in the "Def + 2Pos." scenario, likely because it effectively incorporates information from the positive examples. In contrast, the "Def" scenario shows less satisfactory performance, suggesting that instructions alone are insufficient for achieving optimal results.

Response to W3:

W3. Slightly minor: Particularly for the ICL, how would this work complement or benefit methods such as sample selection [2] or for example optimization style?

Better examples of Instructed Codeword Learning (ICL) typically result in improved performance. Our method leverages this principle by incorporating ICL instances into the hypernetwork's encoding process, alongside the instructions, as illustrated in Figure 2. The information from ICL examples is integrated into the decoder through instruction fusion and concatenation. This approach allows the hypernetwork to generate more accurate inference parameters based on high-quality examples. We have included results comparing regular ICL examples with those without ICL examples, demonstrating that we benefit from ICL. Similarly, our method also shows advantages from enhancements to ICL techniques.

Response to W4:

W4. This is more like a suggestion: It would be good to also have a plotted analysis on the parameter size (which can be presented in terms of percentage of the entire network size) of the generated hypernetworks against the corresponding performance.

In Table 7 of the appendix, we present the results of varying LoRA ranks and their impact on model performance. The table illustrates how different parameter sizes affect the model's efficacy. The corresponding table is included below:

The global author rebuttal Q3 has updated the images, where you can further examine the trends.

Dear reviewer z6rG:

We greatly appreciate your professional review of our article. Here is our feedback.

Response to W1:

W1. Following a method that assigns indices to the LoRA layer, the flexibility of the LoRA architecture structure is reduced.

Our method dynamically generates LoRA weights, offering the same level of flexibility as the LoRA approach. We can customize the rank of the LoRA as well as select which modules require the addition of LoRA weights. Furthermore, our method can be extended to generate parameters for other efficient fine-tuning modules.

Response to W2:

W2. This methodology seems limited to encoder-decoder structured LLMs.

Thank you for your insightful comments, which align with our own concerns. In Appendix B.1, we explained our rationale for selecting Encoder-Decoder models. Encoder-decoder (enc-dec) structure has advantages over decoder-only (dec-only) models regarding task generalization and instruction-following capabilities.

We also recognize the potential of current decoder-only large models (such as Llama3, Qwen2) that have undergone extensive fine-tuning on instruction tasks, potentially serving as a strong baseline similar to TK-Instruct. However, the effectiveness of our method on these models remains to be determined through further experimental evaluation.

Response to Q1:

Q1. Please explain how $\lambda_1$ and $\lambda_2$ were tuned.

In the experiment, we set $\lambda_1$ to two different values: $\lambda_1 = 5$ and $\lambda_2 = \text{sigmoid}(L_{\text{ins}})$. The effects of these different $\lambda$ values on the results are illustrated below. We will include this supplementary experiment in the appendix in the near future.

The global author rebuttal Q2 has updated the images, where you can further examine the trends.

Response to Q2:

Q2. Could you provide more details on the process of updating LoRA parameters via $L_{ins}$?

Certainly, as detailed in Equation 4 of the main text, we calculate the Mean Squared Error (MSE) loss between the LoRA parameters generated by the hypernetwork and those of the specific task model that have been fine-tuned using LoRA. As outlined in Section 3.2.2, similar to LoraHub, we fine-tune specific task models during meta-training with LoRA and save the LoRA parameters. During the training process, the hypernetwork generates LoRA parameters based on the provided instructions and examples during the forward pass. We then compute the MSE loss between the LoRA parameters $\Delta$ of the specific task model corresponding to the instruction and the parameters generated by the hypernetwork. This loss is subsequently backpropagated to update the hypernetwork.

Response to Q3:

Q3. How were the tasks categorized? Please specify the criteria used.

For the SNI dataset, we adhere to the original methodology and dataset setup for pure English tasks as established by the authors [1], with the number of meta-training tasks set to 756. The major task categories and the number of associated subtasks are detailed in Appendix Table 10. The meta-test set comprises 119 previously unseen tasks. For the P3 dataset, we follow the task division approach outlined in the original T0 work[2], separating it into meta-training and meta-testing sets. The specific divisions are presented in Appendix Table 11.

[1] SUPER-NATURALINSTRUCTIONS: Generalization via Declarative Instructions on 1600+ NLP Tasks, EMNLP 2022

[2] MULTITASK PROMPTED TRAINING ENABLES ZERO-SHOT TASK GENERALIZATION, ICLR 2022