Response to the review of reviewer smfw

Dear reviewer smfw,

We would like to thank you for the review and for highlighting the novelty of our approach.

We would like to address your concern as follows.

1. Connection between the MNIST experiment and the LLM experiment

We wish to clarify that we provide the MNIST experiment mainly as a visualization to show how DETAIL works as we believe it is more intuitive to understand visual similarity than semantic similarity of text. We also wish to use the MNIST experiment to highlight that our approach applies to ICL with transformers in general although we focus on transformer-based language models. We appreciate your feedback on the disconnection between the two parts. We will make the transition smoother by highlighting the above clarification in our revision.

2. Accuracy on AG News

We believe two main factors are causing the gap in test accuracy. First, we use only $10$ demonstrations for ICL while AG News typically requires more than $30$ demonstrations to reach SOTA performance (e.g., Figure 3 (c) of [1]). Second, we modify the label names in our experiments to make sure it does not carry the semantic meaning to enforce ICL behavior (lines 171-176). The original label names (Worlds/Sports/Business/SciTech) make it easier for the language model to understand the task, leading to higher accuracy. The performance with altered label can also be observed in Figure 3 (c) of [1] (plotted line with legend $\tau_{\text{RA}}-ICL$) and is similar to the accuracy reported in our work.

Once again, we would like to thank you for taking the time to review. We hope that our response sufficiently addresses your concerns.

[1] Jiaoda Li et. al., "What Do Language Models Learn in Context? The Structured Task Hypothesis", ACL 2024.

Response to the review of reviewer oPDt

Dear reviewer oPDt,

Thank you for the review and for acknowledging the motivation and main idea of our work.

We would like to address your concerns in the following. Kindly note that all citation numbers follow the reference list from the main paper.

1. Comprehensiveness of experiments

We wish to first highlight that our experiments serve mainly to demonstrate the interpretability of DETAIL's attribution and the applicability of DETAIL in various scenarios including demonstration curation and re-ordering. For models, in addition to Vicuna-7b, we have also considered Llama-2-7b, Llama-2-13b models in the paper, as well as an SSM architecture model, Mamba-2.8B, with some of the results in Appendix.

We acknowledge that while DETAIL itself can be useful in many tasks, our experiments only considers classification tasks currently. To extend our work to support other types of tasks, we may modify the loss function accordingly. For example, to support generation tasks where the ground truth is a sequence of desired outputs, we may modify the loss in Eq(3) to be an autoregressive cross-entropy loss instead of an MSE loss.

We thank you for suggesting including experiments for Table 1 and Table 3 using Llama-2-13b. Due to the character limit, we include the additional experimental results together with the transfer results in the PDF in the global rebuttal. We provide an analysis of the additional results in bulletpoint 4 below.

2. Placement of demonstrations in the re-ordering task

We would like to clarify that figure 5 serves to verify how test accuracy changes with the position of the corrupted demonstration. Figure 5 shows that placing corrupted demonstrations at the two ends generally leads to better test accuracy for various datasets (e.g., subj, SST-2, and Rotten tomatoes). Hence, we use the DETAIL score as a proxy for the corruptedness of the demonstration and place a few demonstrations with the highest DETAIL score in front. We expect the same trend to hold on a random new task and we can use the same technique (placing a few demonstrations with the highest DETAIL score in front). The additional experiments in bulletpoint 4 are all conducted by placing 2 demonstrations with the highest score in front.

3. White-box access and Transfer performance on re-ordering task

We acknowledge that DETAIL specifically considers the white-box setting to provide interpretable attribution. While methods under a black-box setting (e.g., [41, 46]) may offer wider applicability, they may also have limited interpretability and poorer performance due to the absence of access to the model's internal working (as shown in Table 2). We highlight that interpretable attribution of ICL demonstrations is the main focus and contribution of this work.

We acknowledge that transfer performance is less significant for the re-ordering task in Table 3 when no demonstration is corrupted. We think it is because GPT-3.5/GPT-4o can already achieve very high test accuracy with the baseline ordering ( baseline accuracy with no corrupted demo in Table 3 is on average 12.0 percentage points higher than that in Table 1).

However, while reordering demonstrations may not improve performance when there are no corrupted demonstrations, it does not deteriorate performance either, nor does it require heavy extra computation (only 1 pass). As such, reordering offers a way to obtain "free" potential performance gain without the costly curation or obtaining more demonstrations.

On your comment that it is less "realistic" to have corrupted demonstrations in the ICL dataset, we argue that although a carefully curated ICL dataset contains few corrupted demonstrations, in various real-world scenarios, ICL demonstrations might not come from a well-prepared dataset but are generated in real-time. For example, LLM-powered search engines (e.g., Perplexity.AI) obtain ICL demonstrations in real-time from web-scrapped data based on the user query, which very often contains corrupted demonstrations. Recent works that consider self-generating ICL demonstrations from LLM (e.g., Chen et. al. "SELF-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations", EMNLP 2023) are also subjected to corrupted demonstrations due to LLM's hallucination. More importantly, as ICL demonstrations are typically generated in real-time in these scenarios, costly curation is not tractable and thus the proposed DETAIL-based re-ordering technique that can indeed be computed in real-time, can be highly effective.

4. Analysis of additional results on re-ordering task

We kindly refer you to the PDF attached in the global author rebuttal for the additional experimental results.

For Llama-2-13b, performance improvement with no corrupted demonstration is less significant compared to Vicuna-7b, although with 3 corrupted demonstrations, we can still observe 1-2% improvement. We attribute this to the stronger baseline performance (baseline (no corrupted) average accuracy on Llama-2-13b is 16.4 percentage points higher than that on Vicuna-7b).

For transfer performance, we observe that the performance gain is not significant with no corrupted demonstrations. With 3 corrupted demonstrations, GPT-4o shows limited performance gain, which is probably because GPT-4o has stronger pretrained prior and can achieve very high test accuracy even with some bad demonstrations. However, for GPT-3.5, the performance improves by more than 3% in this scenario. On the other hand, when 6 demonstrations are corrupted, re-ordering shows a significant improvement of over 3% on GPT-4o as well. This demonstrates the benefit of re-ordering, even for large models, with noisy demonstration data.

Thank you for providing a constructive review and the questions. We hope our response has helped address your concerns. We are glad to provide further clarifications.

Response to the review of reviewer EuFx

Dear reviewer EuFx,

Thank you for your review, compliment for the innovative application of the influence function in our work, and the acknowledgment of the real-world applications of DETAIL.

We would like to address your concerns as follows. Kindly note that all citation numbers follow the reference list of the main paper.

1. Strong assumption

We wish to clarify that while the pioneering work [51] provided the theoretical proof under the learn-to-learn setting, follow-up works have theoretically shown that transformers can implement internal optimizers even when they are pre-trained on random instances [2, 52]. [20] further demonstrated the internal optimizer phenomenon w.r.t. GPT.

While we acknowledge that the kernelized ridge regression does not have a formal theoretical guarantee, it should be noted that the formulation closely follows that mentioned in [51] with slight modification, as we apply it with the influence function (lines 128-131). Following the kernelized ridge regression, equation 6 is then rigorously derived by following the exact definition of the influence function.

2. Sensitivity to demonstration ordering

We would appreciate it if you could kindly provide sources that have made this claim so that we could better address this concern.

Assuming that it is the case where very large models are indeed insensitive to the ordering of demonstrations, we highlight that the claim in line 35 is based on previous works considering relatively smaller language models (typically less than 10b) [35, 37].

Smaller language models are also widely used in real-life scenarios because of their cost-effectiveness (a 7b model can be run on a personal PC). In these applications, the models typically suffer from the problem of being sensitive to demonstration ordering.

Additionally, while it is understandable that very large models are more robust to re-ordering, recent work (e.g., Table 12 of Wu et. al., Arxiv 2405.16122) has shown that there can be up to 10% accuracy difference when large noise is injected by carefully optimizing the order of demonstrations on GPT-3.5.

3. Curation experiment with BERT embedding

Thank you for suggesting the additional comparison baseline (i.e., curating demonstration with BERT-based methods). We have conducted an additional experiment under the setting of Table 2 using BERT score from a popular sentence-transformer model sentence-transformers/msmarco-bert-base-dot-v5. The results are tabulated below:

While we acknowledge that BERT score is relatively faster than DETAIL ($d=1000$), the speed gain is mainly due to BERT scores being model-agnostic and thus likely comes at the cost of poorer performance compared to DETAIL (as reflected by the lower accuracy in the table above). We wish to highlight that one advantage of DETAIL is that its attribution is dependent on the transformer used which provides more interpretability and improved performance.

Moreover, while BERT score is fast, we note that DETAIL is also very computationally efficient: DETAIL is much faster than most attribution methods shown in Table 2. DETAIL can be even faster by reducing $d$. As shown in the table, when $d=50$, DETAIL has a comparable running time as BERT score while still achieving higher accuracy.

Questions

1. Obtaining $y_{\text{test}}$: We wish to clarify that $y_{\text{test}}$ is part of $z_{\text{test}} = (x_{\text{test}}, y_{\text{test}})$ (line 103) which refers to the ground truth label of the query sample. $y_{\text{test}}$ is provided as part of the test dataset.

2. Incorporating BERT embedding in $m(x)$: Thank you for providing the insightful suggestion to incorporate BERT embedding in calculating $m(x)$. We believe it is an exciting direction for future research to consider a combination of BERT and transformer embeddings to improve performance. We have conducted a preliminary experiment with three different $m(x)$ computation tabulated below. For the transformer embedding, we use $d=768$, the same size as the BERT embedding. When using BERT embedding only, the performance is worse than DETAIL. Interestingly, using an equal-weighted average results in SOTA test accuracy on 3 out of 4 datasets.

Thank you again for providing a constructive review and offering insightful suggestions. Please let us know if there are any remaining questions. We are glad to offer further clarifications.

Response to the review of reviewer ixwr

Dear Reviewer ixwr,

Thank you for reviewing our paper and highlighting the novelty of our application of 'influence' for ICL which has the potential to inspire better instruction-tuning algorithms.

We would like to address your concerns in the following.

1. Identifying adversarial attacks for LM

We thank you for suggesting this exciting potential use case. In this work, we have demonstrated how DETAIL can identify corrupted demonstrations, which is one form of adversarial attack where an attack is performed by perturbing the ICL demonstration label (Table 1 and Table 3). We believe it is an interesting area to research how DETAIL can be applied to other types of attacks. For example, DETAIL might be used to defend against attacks that attempt to steal private ICL demonstrations by filtering out or rephrasing queries that cause excessively high DETAIL scores.

2. Additional comment on how DETAIL can help with LLM training

We thank you for the question. While we primarily focus on in-context learning settings in this work, the DETAIL score can also potentially be used for selecting demonstrations for instruction fine-tuning. One way to perform the selection is to compute the DETAIL score of each ICL demonstration on a validation set and then use the demonstrations with the highest DETAIL scores for fine-tuning.

We will incorporate the mentioned points in the future work section in our work.

Once again, we would like to thank you for the insightful suggestions. We hope our response has been helpful in addressing your questions.