Iterative Vectors: Boost In-Context Learning within Activations

We sincerely thank the reviewer for the positive feedback and high rating of our paper. We are grateful for their thoughtful review and are pleased to know that they find our work valuable. The suggestions provided are greatly appreciated.

W1 (Table 1): We will added a comparison section in Appendix D.

W2 (Figure 3): We will revise the figure as suggested.

W3 (Table 4): We tested using a plot instead of a table; however, we found it to be a less efficient use of space, which is a concern raised by Reviewer WqMZ. Consequently, we have decided to retain the table. We appreciate the reviewer's suggestion nonetheless.

W4 (inference time): We will add FV and TV to Table 2. Additionally, we have also added the extraction time.

Q1 (emoji dataset): The emoji task involves predicting the most likely emoji for a given tweet. This task only considers tweets containing a single emoji, regardless of its position, which serves as the classification label. The dataset for this task is readily available for download and viewing on HuggingFace, identified by the tag tweet_eval/emoji.

Q2 (larger models): We will incorporate experiments using the llama-2-70b model. However, due to our computational budget constraints, we are unable to complete all tasks with a model of this size. Therefore, we have chosen to conduct a multi-shot experiment similar to Table 3 to more effectively demonstrate the efficacy of IV.

We're grateful for the effort the reviewer invested in identifying the areas for improvement. We have reorganized our response to the concerns according to their logical structure and will address the questions as follows.

W1:

Section 3 discusses the theory of transformers as meta-optimizers, but its connection to the main contributions and approach of IVs is unclear.

We regret that the reviewer appears to have misunderstood how our method is derived from the interconnected theoretical foundations. We believe this is a crucial aspect of the paper for comprehending the motivation behind our method. As noted by Reviewer sVxj, "the framing of iterative vectors as a way to capture the meta-gradients that occur during ICL is well-done and well-motivated (Section 3.1)."

Figure 2 also seems to be redundant

As discussed in our response to weakness 1 from Reviewer QraR, Figure 1 serves as a general illustration of how activation vectors function, while Figure 2 specifically demonstrates our proposed method. We consider Figure 2 an essential part of our paper.

Moreover, there are formatting issues, [...] would improve the overall presentation and clarity.

We appreciate the reviewer's diligent efforts in identifying how adjusting the empty spaces can enhance the clarity of our paper. We will make the necessary revisions.

W2 & Q2: Our research aims to introduce a new direction rather than exploring prompt instability, as examined in previous studies [1-3]. We have, in fact, already cited two of the three papers referenced by the reviewer.

Regarding the works on multi-modal and visual models [4-6], while we appreciate the suggestion, our paper is entirely focused on NLP. The background, motivation, theories, evaluation protocol, baselines, and experiments are all firmly rooted in the domain of NLP. Introducing content related to CV seems incongruous with the rest of our paper and might confuse the readers.

W3: We recognize that Figure 3 can be challenging to interpret and will simplify the figure.

W4: Upon closer examination, it becomes evident that FV, TV, and our IV differ significantly in the construction of their vectors and the underlying rationale. This distinction goes beyond simply adding aggregation or tweaking prompt design to a previous method. A more detailed comparison of these methods will be provided in Appendix D to elucidate this further.

W5 & Q1 & Q3: Evaluating a single modality does not imply that the tasks are not based on real-world scenarios. We have incorporated Llama-3.1-8b into our main experiment, as shown in Table 1, where it has demonstrated similarly promising results. Furthermore, our newly added experiments with Llama-2-70b in Table 8 corroborate our findings. This demonstrates that our method is effective across a range of models and model sizes, including one of the more recent models.

W6: As stated in the Introduction, increasing the number of in-context examples does not necessarily improve performance. For further details, please refer to our response to question 2 from Reviewer sVxj.

Q4 & Q5: No, but there are valid reasons for this decision.

As foundational work, previous research on FV and TV has naturally focused on identifying the properties of activation vectors, such as feature space and compositionality. In contrast, our approach prioritizes successful adaptation and application to real-world ICL tasks. Consequently, we have deliberately set aside the exploration of feature space to more thoroughly demonstrate the efficacy of our method within the constraints of limited pages.

Feature space investigation is not strictly essential for the application of a method, nor does it always provide useful insights (see the "English-French" row in Table 5 from the FV paper).

Furthermore, when evaluating real-world datasets, the process of composition is not well-defined, as we are no longer dealing with simple synthetic tasks involving easily deconstructed and recombined word pairs.

I thank the authors for the detailed responses and additional clarifications. The revised paper and the method are now clearer.

After reading all the rebuttals as well as the revised paper,I have decided to raise my score to 5.

We are impressed by the reviewer's in-depth review and their thorough examination of the details of our method. We are deeply grateful for their commendation on the effective and well-motivated capture of the meta-gradients, as well as their acknowledgment of the importance of the problem we address. We will strive to provide clear answers to their direct and concise questions.

W1 & Q1: We recognize that formulas can be challenging to understand, and we apologize for any lack of clarity. To assist with this, we will provide an appendix that includes pseudocode to elucidate our method, including the iterative procedure.

W2 & Q2: We apologize for the omission of evidence supporting this claim. We considered it a relatively common instability for standard ICL and did not specifically address it, as it frequently occurred in our experiments. However, this is suggested in Table 2 (on the emoji dataset), where the 3-shot clean result is inferior to the 2-shot one.

Here, we include several other experiments that demonstrated this phenomenon. The results are evaluated based on 10,000-episode clean runs on llama-2-7b.

Q3: We intended to let $\mathcal{C}_j$ represent the set of indices of the pairs $(x, y)$ that belong to class $j$ with the accompanying text description. We apologize for the lack of precision and have reformulated it to adhere to the convention recommended by the conference. We now redefine it formally as follows: $\mathbb{C}_j = \\{ i \mid \operatorname{Class}(x_i)=j \\}$ where $\operatorname{Class}(\cdot)\in \\{1,2,...,n,q\\}$ was previously specified below Formula 24. We will update the paper to reflect changes made to the formulas.

W3 & Q4: Lines 350-351 indicate that, rather than offering the model imbalanced demonstration examples—as is common in studies examining the instability of ICL—we present one example from each class to ensure simplicity and completeness. However, this does not address the reviewer's concern regarding zero-shot experiments. We will expand upon the rationale mentioned in line 482, providing a more detailed explanation of why zero-shot experiments are not included, in Appendix E.

Q5: We appreciate the reviewer's suggestion and will include a detailed comparison in Appendix D.

We would like to follow up on our recent rebuttal comments. Your feedback is invaluable to us, and we are eager to address any remaining concerns. Thank you for your time and consideration.

Q1: As the reviewer suggested, we agree that including a detailed algorithm is beneficial and will add pseudocode in Appendix A. Furthermore, we intend to make our code publicly available in the future to support ongoing research.

(Q2 is addressed in conjuction with W3 above.)

Q3: We appreciate the reviewer's keen observation regarding the initial performance drops in Table 4. The solution might be more straightforward than anticipated: using standard ICL instead.

As demonstrated in Table 4, this drop appears when only two episodes (equivalent to 2-shot) are available for inference, the IV performance is inferior to the standard 1-shot ICL. In scenarios with such a limited number of examples, complex strategies may not even be necessary; a straightforward selection of examples can suffice to achieve standard ICL performance, although outcomes may vary.

IV aims to incorporate a moderate number of examples to address the instability and ineffectiveness that arises when standard ICL is performed on these examples. This means that examples cannot be casually selected nor incorporated into a single prompt, while still maintaining minimal temporal overhead. As evident in Table 4, when more than two episodes are available, IV surpasses 1-shot ICL, thereby fulfilling its intended purpose.

It is important to recognize that the results presented in Table 4 were obtained using a fixed set of hyperparameters, which may not be optimal for each episode amount. It is advisable to search for improved hyperparameters if IV were to be employed.

As an additional note, data augmentation techniques could enhance stability in these contexts. By generating additional training examples through transformations or modifications of existing data, these techniques may provide further benefits. However, this topic is not confined to IV and extends beyond the scope of our paper.

Q4: We appreciate the suggestion. We will provide a detailed explanation of the hyperparameters, accompanied by a guide to tuning the hyperparameters, in Appendix F.

We appreciate the reviewer's acknowledgment of the simplicity and intuitiveness of our proposed method. We will now address the concerns raised.

W1: We apologize for the confusion. Regarding the contributions, we highlighted three key points: our adaptation of previous works to this new direction, our unique invention—a novel activation vector method—and the state-of-the-art performance of our method. Additionally, we emphasized that we are the first to apply activation vectors to diverse real-world ICL tasks.

Figure 1 serves as a basic, general illustration of how various activation vectors operate within the activation space, rather than providing a detailed explanation of our IV. In contrast, Figure 2 is specifically designed to demonstrate the workings of IV. We will revise the content and caption of Figure 1 to clarify this distinction.

W2: We appreciate the reviewer's acknowledgment of our statements in the Limitations section. Although our focus is primarily on classification tasks, it is important to note that previous studies often concentrate on synthetic classification tasks or a single type of generative task, such as sentiment transfer or truthfulness elicitation. In contrast, we "rigorously evaluate IVs across multiple models and diverse tasks, providing a comprehensive assessment of their performance", as noted by Reviewer mGxX.

W3 & Q2: We regret that the reviewer did not find our derivation—demonstrating how IV more effectively captures the meta-gradients and addresses the limitations of ICL—to be sufficiently convincing. To resolve this, we provide further explanations:

1. Meta-gradients derived from limited in-context examples may not fully capture the task, instead, they can be adversely influenced by outlier examples. To address this issue, IV extracts gradients from multiple episodes and averages them. Achieving this with a large number of examples using an ICL prompt would be costly, if not unfeasible.
2. Meta-gradients derived from limited in-context examples may not scale appropriately with the original parameters. The gradients might be either too weak to effectively influence the original parameters or too strong, potentially destabilizing them. To address this issue, we introduce the strength hyperparameter $\alpha$. For further insights on this design, please refer to our response to Q4 below.
3. The extraction is conducted in an iterative manner, as analyzed in Section 4.4 and illustrated in Figure 3 (where $b$ changes from 0 to 1), resulting in a significant performance enhancement. The vectors are incorporated into the activations during the inference process, achieving a boost that is unattainable within the constraints of the discrete prompt space.
4. In addition to averaging the extracted vectors, the iterative updates performed by IV are also batched. This standard approach stabilizes the gradients by averaging them over multiple episodes before applying them, thereby reducing the variance in gradient estimates. Consequently, more reliable and consistent updates can be achieved with an appropriate selection of $b$ during extraction.

We hope these explanations will assist the reviewer in recognizing the capture as well-executed, as noted by Reviewer sVxj.

W4: In addition to the explanations provided above, we will conduct experiments using a newer model, llama-3.1-8b, as well as a larger model, llama-2-70b, to further substantiate our approach.

Thank you for your insightful review! Before we address your concerns, could you please specify which "FT paper" you are referring to (in the second question)?

We appreciate the insightful review provided by Reviewer P6ha and are honored by their assessment that our paper is "well-written", our idea is "interesting", and our experiments are "principled". We address their concerns as follows.

W2 (Comparison with PEFT): We are pleased to observe that the reviewer has a profound understanding of our method and provides a comprehensive comparison with PEFT methods. However, there is a significant distinction in the "training" aspect that cannot be considered overlapping between the two approaches, as we outline below.

• IV does not involve training. PEFT involves additional parameters that are trained through backpropagation, which is costly in terms of both computation and VRAM. Although it is parameter-efficient, it still results in significant memory usage. In contrast, IV only needs vectors to be stored during inference, eliminating the need for training. This disparity makes it challenging to fairly compare IV with PEFT methods, leading us to choose FV and TV as baselines. Nevertheless, we conducted empirical tests on llama-2-7b with the ag_news dataset and found the following results:
  • A clean run requires 15.2 GiB of VRAM.
  • Training a dummy LoRA module with only 0.06% of parameters uses 19 GiB of VRAM after the first episode and exceeds 30 GiB as the prompt sequence lengthens with more verbose episodes. While checkpointing may help reduce VRAM usage, it also results in increased training time.
  • IV requires 15.7 GiB of VRAM. The minor surplus is partly due to our retention of vectors from all episodes to facilitate coding and research efforts. This is unnecessary for deployment, as only a vector average is required.
• Therefore, IV can be applied in an online manner. The absence of a training phase allows IV to be used immediately. Upon receiving a batch of examples for inference, IV can iteratively extract and refine vectors without interrupting the inference process, virtually dedicating all computational resources to solving the task without spending them on an upfront training phrase.
  • Although determining hyperparameters might be a concern, this issue also exists in PEFT methods, which introduce their own hyperparameters, not to mention those required by the training process—something IV does not need to address.
• Moreover, IV requires significantly fewer additional states. The Iterative Vectors, each corresponding to the model's hidden dimension $D$, are extracted for each of the $L$ layers, resulting in a total additional state size of only $L \times D$.
  • For PEFT methods, such as LoRA, the additional state size may be reduced to around 0.1% of the model size.
  • For IV, the vectors amount to less than 0.002% of the model size, which is essentially negligible.

In light of the points discussed, we consider the study of activation vectors to be an even more lightweight approach than PEFT that merits separate exploration.

Q1: They are resampled each time to continuously incorporate information from more examples.

Q2: Regrettably, we were unable to locate any references to "GPT 1.3B", "GPT 2.7B", or the provided numbers in the FV paper available on either ArXiv or OpenReview, despite thoroughly reviewing the main text and conducting multiple searches through all supplementary materials. Upon reviewing the code, we confirmed that the GitHub file for FV responsible for loading models have not implemented these two models. Additionally, Table 9 in the FV paper, which provides a summary of all the datasets used, does not include SST-5; instead, it mentions SST-2.

Still, we have invested considerable effort into optimizing the hyperparameters of FV, and we have now achieved more satisfactory results. However, this improvement comes at a substantial cost in terms of time, as its design leads to extremely slow extraction processes. Please refer to the updated results in the paper and the limitations of FV in incorporating additional examples in the main experiment, detailed in Section 4. An example of the extraction time required by FV will be included in Table 2, with a discussion of its design provided in Appendix D.

Q3: We apologize for the misunderstanding. The emoji dataset, designated by the HuggingFace tag tweet_eval/emoji, is an established dataset that accompanies the various other datasets we use from the TweetEval framework. We will revise the wording in Section 4.1 where it is first mentioned and ensure it is appropriately included in the table of all datasets.

We have revised our paper in accordance with the reviewers' feedback and would greatly appreciate your response. Your insights are essential for the enhancement of our work, and we welcome any additional comments or approval.