Mitigating Copy Bias in In-Context Learning through Neuron Pruning

We thank the reviewer for their comments.

Q1: Is the copying bias merely an artifact of having a limited number of examples (e.g., <= 4)? For instance, what happens in a 64-shot setting? Could this bias indicate that the copying issue arises from an ill-posed in-context learning (ICL) problem rather than a general model flaw?

Our additional experiments with up to 64 shots (reported in Appendix G) directly address this concern. While copying bias is indeed more pronounced in few-shot settings, our results show it persists even with increased examples, albeit to a lesser degree. The gradual narrowing of the performance gap between pruned and unpruned models as shot count increases suggests that copying bias is not merely an artifact of limited examples but rather a fundamental behavior that models naturally rely on. This behavior diminishes as models receive more examples and become better equipped to learn underlying patterns, yet our pruning method continues to yield consistent improvements even in many-shot settings. This indicates that while the problem is more acute in few-shot scenarios, copying bias represents a general model tendency that our method effectively mitigates across different shot regimes.

Q2: Could the authors provide additional details about the proxy dataset? For example, how many shots are used, and could a few more examples be included for completeness?

In our implementation, we have used n=2 examples per prompt which allows us to study copying behavior. Our synthetic dataset construction addresses the concerns about label repetition: we ensure that each prompt's examples (Sp) contain distinct vowel counts, and the target word's vowel count (ŷ) is always different from those in Sp. We sample words with vowel counts ranging from 1-8, providing sufficient diversity while avoiding repetition within prompts. These constraints are applied during dataset generation. Examples from our proxy dataset:

• "bed → 1"
• "school → 2"
• "octopus → 3"
• "penguin → “3"
• "zebra → 2"

We generate 400 prompts, we use 300 for the detection process and 100 for the meta-optimization ( a validation set to determine optimal block and pruning percentage). We have updated the revised to include these details. We also refer the reviewer to the newly added Appendix I, where we explore alternative proxy tasks, demonstrating that the specific choice of the proxy task may not be as critical.

Q3: How does pruning of these neurons affect the overall model performance beyond ICL tasks? For example, how do the pruned LLMs perform on more general benchmarks such as Live-Bench or other reasoning benchmarks (e.g., MMLU or GSM8K)? Or, what’s the performance on the indirect object identification (IOI) task? The authors suggest that their method “could be broadly applicable, potentially becoming a standard post-processing step in LLM deployment pipelines” (l. 497-498). However, it remains unclear if this would be a sensible approach for LLMs deployed “in the wild.”

Q3: How does pruning of these neurons affect the overall model performance beyond ICL tasks? For example, how do the pruned LLMs perform on more general benchmarks such as Live-Bench or other reasoning benchmarks (e.g., MMLU or GSM8K)? Or, what’s the performance on the indirect object identification (IOI) task? The authors suggest that their method “could be broadly applicable, potentially becoming a standard post-processing step in LLM deployment pipelines” (l. 497-498). However, it remains unclear if this would be a sensible approach for LLMs deployed “in the wild.”

To assess potential tradeoffs introduced by our pruning method, we evaluate the pruned models on perplexity (WikiText) and MMLU benchmark. As shown in the following two tables, our method has minimal impact on these capabilities, while MMLU performance remains stable or even improves in some cases, indicating that our targeted neuron pruning successfully mitigates copying behavior while preserving general capabilities. We have updated the draft to include these results with additional models in Appendix F.

Perplexity on WikiText test (lower is better) :

MMLU (higher is better):

Q4: Are the minimum and maximum values of the normalizing functions calculated across the dataset alone, or are they based on both the dataset and all neurons?

For each neuron, we first compute its attribution scores using integrated gradients across all prompts in our detection synthetic dataset. We then normalize these attribution scores for each neuron independently by applying min-max normalization across the scores of the different prompts (using the minimum and maximum values of that neuron's scores across different prompts). Finally, we average these normalized scores to obtain the final relevance score for the neuron. This ensures we get comparable scores across different samples before aggregation. We have updated Equation 7 in the revised version to make it more clear.

Q5: What’s the significance of the meta-optimization (l. 319-369)? Why not greedily remove the neurons based on the “copying score” from Eq. 7 (this may be related to the question in Q4)?

The meta-optimization process (using a validation set to determine optimal block and pruning percentage) is crucial rather than simply removing neurons based on the copying score for several reasons. First, while the copying score identifies neurons that contribute to copying behavior, it doesn't directly tell us the optimal proportion to remove - pruning too many neurons could impair the model's general capabilities, while pruning too few might not sufficiently mitigate the copying bias. Second, different model architectures may have different optimal pruning points due to their varying architectures and redundancies. The meta-optimization helps us find this balance automatically.

W2: It is unclear to me whether the performance improvement results from the reduction of copying neurons or from the pruning of neurons exhibiting different behavior. For example, why would performance improve on copying tasks such as “list first” or “list max” in the ICL benchmark of Hendel et al [2], which require copying from the in-context examples? Tabs. 3-6 consistently also show improvements on these copying tasks. If the identified neurons indeed suppress the copying behavior, we would expect to see drops in performance on such tasks. This seems to contradict one of the authors’ main claims that they “identify specific neurons responsible for triggering the copying behavior (copying neurons)” (l. 71-73).

1. Clarifying "Copying Errors" vs. "Valid Pattern Matching":

Our work targets inappropriate copying where the model reproduces an promt example answer instead of applying the learned pattern This is different from tasks like "list first" or "list max" which require understanding and applying a consistent pattern that may involve selecting/copying elements from the example itself according to a rule. When we say "copying neurons," we specifically mean neurons that trigger direct reproduction of example outputs (and not example content) without pattern recognition.

2. Evidence for This Distinction: For tasks like "list first," the model must: Understand the positional pattern (select first element). Apply this pattern to new inputs. "Copy" the correct element based on the learned rule. In contrast, copying errors occur when the model: Fails to learn the underlying pattern. Instead directly copies an example output regardless of whether it follows the task pattern

The neurons we identify through ΔL specifically target cases where probability mass shifts toward example outputs that don't match the ground truth. This focuses on inappropriate copying while preserving the model's ability to learn and apply patterns that may involve selective copying. The improvement on "copying tasks" suggests our method is successfully distinguishing between: Valid pattern-based selection/copying, Invalid memorization-based copying.

If we were indiscriminately suppressing all copying behavior, we would indeed expect degradation on tasks requiring valid copying. The consistent improvements across both pattern-matching tasks and other tasks suggest we're successfully targeting problematic copying while preserving valid pattern learning.

W3: It is unclear whether the copying bias is merely an artifact of the ill-posed nature of few-shot ICL problems, and whether it remains relevant for many-shot ICL problems. Does this copying bias persist beyond the few-shot setting? (This point is also raised in the questions below.)

Our additional experiments with up to 64 shots (see Appendix G) reveal that while copying bias is most prominent in few-shot settings, it doesn't completely disappear with more examples. As the number of shots increases, we observe a smaller but still meaningful gap between pruned and unpruned models. This pattern suggests that copying isn't just a fallback strategy when examples are scarce - it's an inherent model behavior that gradually gives way to pattern learning as more examples become available. The persistent improvements from our pruning method, even with many shots, confirm we're addressing a fundamental aspect of how these models process in-context examples.

W4: The writing could be improved. There are several typos throughout, and clarity and reproducibility would benefit from including all necessary details, such as those regarding proxy dataset generation. Fixed and updated at the relevant places in the paper draft.

W5: Code is not provided, making reproducibility challenging. Code will be made publicly accessible

In Figure 2 and 3, it seems that only models up to 1.3B are evaluated. In Figure 2 it seems that a large proportion of error comes from the copying error. However, for larger models (e.g., 8B, 14B) and for the singular to plural task, I suspect that larger model will not have this high copying error if each in-context example is unique. For 1-shot and 2-shot setting, the performance might simply be improved by adding a task description at the very front to eliminate task ambiguity.

Following the review, we have experimented with using task descriptions. In our experiments, we compare three settings: (1) standard ICL without task descriptions (baseline), (2) ICL with added task descriptions explaining the required transformation (e.g., "Convert verbs from present to gerund form"), and (3) our pruning method. While adding task descriptions improves over the baseline, our pruning method achieves substantially better results across all settings , suggesting that copying bias persists even with explicit task specifications and requires targeted intervention through neuron pruning. Interestingly, combining task descriptions with our pruning method (TD+Pruning) yields the best performance across all models and shot settings, and maintains consistent gains even for larger models like Llama3 8B, suggesting these approaches are complementary in addressing different aspects of the copying bias problem. The results below are averaged across the following tasks : (linguistic present simple to gerund, linguistic present simple to past simple, linguistic present simple to past perfect, linguistic singular to plural) over 5 different seeds. (TD refers to adding Task Description). We have updated the revised version to include this analysis, at Appendix J.

Pruning the neuron is essentially "training" the model on the ICL task. It would be convincing if the author presents other training-based method as baselines (with a sweep of hyperparameter on the validation set) in Figure 2 and 3.

We appreciate the reviewer's suggestion to compare our pruning approach with training-based methods. To address this, we conducted experiments comparing our pruning method with fine-tuning on the vowel mapping task. Specifically, we fine-tuned models using the same synthetic vowel mapping dataset used for neuron identification, performing a thorough hyperparameter sweep (learning rates: [1e-5, 1e-4, 1e-3], batch sizes: [4, 8, 16], epochs: [1, 3, 5]) and selecting the best configuration using the validation set. Results are averaged across all of the linguistic tasks for 5 different seeds, the left number in each cell presents the baseline model without any modification, and the right number in each cell presents the result of the finetuned/pruned model. We have updated the revised version to include this analysis in Appendix J.

These results demonstrate that finetuning on the vowel mapping task often degrades or maintains baseline performance. This may arise from an overfitting to the synthetic task caused by the finetuning process. In contrast, as shown in our paper our pruning method consistently yields substantial improvements across all models and shot settings.

For Figure 5, while there is improvement, the author does not show how much improvement comes from less copying bias. The author has larger models like Llama-3-8B in Figure 4 experiment but there lacks evidence that the pruning decreases the copying error.

Task vectors, as introduced by Hendel et al., represent the model's learned understanding of a task from in-context examples - essentially capturing the transformation needed to solve the task. In Figure 5, we compare two types of task vectors: "Task-Vectors" refers to task vectors obtained from the original unpruned model which exhibits more copying errors, while "Task-Vectors-Pruned" refers to task vectors obtained from our pruned model which reduces these copying errors. The results demonstrate that Task-Vectors-Pruned significantly outperforms Task-Vectors across the different shots and tasks. providing strong evidence that our pruning strategy enhances the model's ability to distill task-relevant information into high-quality task vectors.

In the following, we show the percentage of the copying errors among the total errors for the standard ICL method compared to the case with pruning the copying neurons detected by our method over the BBH ICL task. Evidently, our method indeed decreases the copying error also on larger models like Llama3 8B (We have updated the revised version to include this analysis in Appendix C):

Example in the introduction can be ambiguous. it could also be the case to count the number of words and this case the correct answer for "Florida" is 1.

Thank you for this comment. Following this concern, we have replaced this example in the revised version.

Line 063-064: the first quotation mark of "Copying Neurons." is not correct. Thank you for pointing this. Following this concern, we have fixed it along with many other typos.

We are grateful for your detailed review, which has guided us in improving our paper.

Lacks some implementation detail: in Section 3.3, what is the size of n? The author assumes that y^∉Sp and therefore if n is large, there will be many repetitions in Sp since labels in Sp are the count of vowels, which are integers. Also, does the author explicitly held out an integer so that when constructing Sp, the words cannot have the number of vowels equal to this held-out integer?

In our implementation, we used n=2 examples per prompt, as this setting more clearly exposes the copying tendency in models, making it ideal for detecting the relevant neurons. When constructing prompts, we ensure each prompt's examples (Sp) contain distinct vowel counts, and the target word's vowel count (ŷ) differs from those in Sp, sampling from words with 1-8 vowels.

Universality of copying neurons detection: This paper only studies how to detect copy neurons on the vowel counting task. What if this proxy ICL task is switched to other tasks? Will the copying neuron detected still work, and if not (e.g., if all elements in are unique), what makes vowel counting special that it works as a proxy ICL task?

Our results show that neurons identified using any of these tasks led to similar improvements in performance across the broader set of evaluation tasks. The key requirement for an effective proxy task appears to be that it: 1) has clear, deterministic rules that require pattern recognition rather than memorization, 2) can elicit copying behavior from the model, and 3) has a sufficiently constrained output space to detect copying errors reliably. While vowel counting meets these criteria particularly well and is easy to generate synthetic data for, it is not uniquely special - other structured tasks with similar properties could serve as effective proxies for copying neuron detection. To verify this, we conducted additional experiments using different proxy ICL tasks beyond vowel counting, mainly the two linguistics tasks: linguistic antonyms (happy → sad) and knowledge person language (Muhammad → Islam).

The results below are averaged across the following tasks : (linguistic present simple to gerund, linguistic present simple to past simple, linguistic present simple to past perfect, linguistic singular to plural) over 5 different seeds. In each cell of the table, on the left side, we report the result of the unpruned version of the model, and on the right side, we show the result of the pruned version using our method. We have updated the draft to include this analysis in Appendix I.

For the vowel counting task:

For the linguistic antonyms task:

For the knowledge person language task:

Our experiments with different proxy tasks (vowel counting, linguistic antonyms, and knowledge person language) demonstrate comparable improvements across models and shot settings, with vowel counting showing slightly stronger gains, suggesting that while multiple structured tasks can effectively identify copying neurons, tasks with clear deterministic rules and constrained output spaces like vowel counting may be particularly well-suited for this purpose.

Continuation of the previous point: method in 3.3 assumes that the output has single token. This is probably fine for the vowel counting task as single-digits are single-tokens. What if we have other ICL tasks as proxy ICL task and the outputs are multi-tokens?

We demonstrate that our method, despite using single-token vowel counting as a proxy task, effectively generalizes to multi-token outputs. As empirical evidence, we show consistent improvements on the country-currency task where outputs are typically multi-token (e.g., "Benin" → "CFA Franc (XOF)") and national parks ("Badlands National Park": "South Dakota")., indicating that neurons identified through single-token analysis still effectively mitigate copying bias in multi-token scenarios. In each cell of the table, on the left side, we report the result of the unpruned version of the model, and on the right side, we show the result of the pruned version using our method.

national parks:

Country-currency:

Our detailed analysis of multi-token tasks (national parks and country-currency pairs) shows that our method improves performance specifically on multi-token predictions. For example, on the country-currency task, improvements are observed on multi-token currencies like 'New Zealand Dollar' and 'Saudi Riyal'. This demonstrates that while our proxy task uses single-token outputs for neuron identification, the benefits effectively generalize to multi-token prediction scenarios, suggesting the identified copying neurons play a role in sequence-level prediction behaviors.

While our current implementation uses single-token proxy tasks for simplicity, the method could be naturally extended to explicitly handle multi-token outputs by aggregating prediction shifts across the sequence: ∆L = Σt(l_v^t - l_u^t), where t indexes token positions. This approach aligns with standard practices in applying integrated gradients to generation tasks. However, our results suggest that such an extension may not be necessary in practice, as the current approach already generalizes well to multi-token settings. We have updated the revised version to include these results in Appendix E.

How are errors being calculated? There are two ways: 1) probability-based: all probability summed for labels not equal to the true label and averaged across all inputs; 2) accuracy-based: 1 - (accuracy for the correct output) with greedy decoding.

In our evaluation, we use accuracy-based error calculation with greedy decoding (method 2). A prediction is counted as an error if the model's output with the highest probability doesn't match the ground truth. For multi-token outputs, we require an exact match of the complete sequence. While this approach provides a straightforward way to measure performance, we note that it may be conservative in cases where multiple valid answers exist.

Dear Reviewer ZJtA

As we approach the end of this discussion period, we want to thank you for your thoughtful feedback, which has significantly improved our paper. We believe we’ve addressed most of the concerns raised by you and the other reviewers. In particular, we have incorporated:

• Additional implementation details.
• A fine-tuning baseline.
• A Task Description baseline.
• Evaluations across a broader range of tasks (e.g., MMLU, AlpacaEval2) and retrieval challenges like Needle IN A Haystack
• Experiments demonstrating stability to the choice of the proxy task
• Experiments demonstrating the effectiveness of our method over multi-token output tasks.

These experiments demonstrate that our approach consistently mitigates copy bias without compromising performance on tasks where copying is beneficial (e.g., certain retrieval tasks).

We would appreciate it if you could let us know whether our responses properly address your concerns.

Sorry for the late response. I thank the author for their reply. I think the paper has improved, and the new experiments added show the effectiveness on simple tasks like vowel counting and on smaller models. However, I still have some concerns about the paper.

• This paper studies copying bias of ICL. The paper can fall into one of the two categories, each with different criterion to evaluate the contribution of the paper:
  • (1) "Methodology" paper, where the contribution is evaluated based on how applicable / useful the new methodology can be, or
  • (2) "Finding" paper, where a detailed analysis need to be conducted to study copying bias (e.g., mechanistically; Reviewer 4ESn also mentions this).
• It seems that the paper falls more into a "methodology" paper as the author claims that "consistently improves performance across a diverse set of ICL tasks". However, I am not sure this claim is fully supported with current evidence in the experiments and the applicability is limited.
  • As I mentioned in the review, "I suspect that larger model will not have this high copying error if each in-context example is unique". While the author provides table for experiments in the rebuttal, looking at the results of Llama-3 8B, the improvement is quite marginal (also I'm not sure how much percentage is from mitigating copying bias and whether there are cases where, after pruning, get worse).
  • While one can argue that there is 3 percent improvement in the 1-shot setting, the performance in 2-/3-/4- shot setting is very close to (or the same as) the TD case. The additional cost from adding additional ICL samples is small (so one can choose to switch to 4-shot ICL). Also, there is an extra cost of applying the pruning on the model.
  • The tasks are relatively simple.

For these reasons, I will keep the current score. I would encourage the author to do a more detailed analysis on copying bias.

Dear Reviewer ZJtA,

Thank you for your detailed feedback and for engaging with our rebuttal and additional experiments.

We would like to respond to the concerns you raised. We believe our contribution bridges both categories you outlined – Findings and Methodology. On the methodological side, we propose an approach to mitigate copying bias effectively. Concurrently, we aim to provide insights into the behavior of ICL approaches regarding copying bias. While "mechanistic" might not correctly capture the nature of our findings, we do establish a connection between copying bias and inaccuracies in task vector predictions, demonstrating that pruning improves inference by addressing these inaccuracies. Furthermore, we find it compelling that this shortcut is largely task-agnostic; that is, detecting neurons on a single task allows us to influence other tasks in a functionally isolated and effective manner.

On the methodological front, we highlight the consistent reduction in copying error. While larger models and multi-shot ICL settings are inherently more accurate, leading to smaller absolute reductions, the relative impact in reduction of copying of our approach remains substantial, and the proportion of copy errors remains high or even increases. For instance, as highlighted in Table 5 (Appendix C),: 15% of errors in LLaMA3 (8B) with 4-Shot ICL on BBH are copy errors, compared to only 4% with 1-Shot ICL. This highlights the continued relevance of our approach, even in advanced settings. Furthermore, the pruning process is computationally inexpensive and only needs to be performed once, making the method practical for real-world scenarios. We also wish to point out that these new concerns regarding the magnitude of improvement and the positioning of the paper were not part of your initial review. Nevertheless, we have made extensive revisions to address all 7 weaknesses you originally highlighted. The paper has substantially improved in clarity and experimental support as a result of your valuable input.

1. Lack of implementation details: We’ve added detailed explanations (see lines 185 - 193).
2. Universality of copy neuron detection: We’ve demonstrated stability with respect to the choice of the proxy task (see Appendix I).
3. Only single-token tasks: The model was already evaluated on multi-token tasks. We clarified this and emphasized the improvements in multi-token setups (see Appendix E).
4. Error calculation unclear: We clarified that the standard metric, exact match is used
5. Results in Fig 2 and 3 limited to 1.3B model & Task Description: We incorporated results with LLaMA 3 (8B model) (see Appendix J).
6. Comparison with standard training-based baselines missing: We added these comparisons and showed that the baseline performs poorly (see Appendix J).
7. Source of improvement (reduced copying bias): We added a new table confirming that the improvement is indeed due to reduced copying bias (Appendix C).

We greatly appreciate your careful evaluation and hope this response helps clarify the significance of our work and its positioning. We respectfully ask you to reconsider the score in light of the substantial improvements and alignment with your original feedback.

Thank you once again for your time and effort.

Sincerely,

The Authors

Thank you for your effort and for taking the time to help us improve our paper.

In Figure 2, it’s clear that the copying error rate has dropped consistently across different models and tasks, contributing to improved accuracy. However, I’m surprised that copying errors still dominate the error rate in the unpruned model. Do the authors have any insight into why this phenomenon occurs? Additionally, how does the copying error rate compare for the vowel-counting task specifically? I’m curious about the extent to which copying errors can be reduced on the in-distribution task (vowel counting) versus out-of-distribution tasks (the other ICL tasks).”

We believe copying errors dominate because it's a "safer" failure mode for the model - repeating a previously seen answer rather than generating a potentially incorrect novel response. This aligns with the training objective of minimizing unlikely outputs. For the vowel-counting task, copying errors occur at a similar rate to other tasks, suggesting that copying behavior is a general phenomenon rather than task-specific. This consistency between in-distribution and out-of-distribution tasks supports our hypothesis that we're identifying fundamental copying mechanisms in the model architecture rather than task-specific patterns.

In the following table, we report (Copying Error Rate / Total Error Rate) over the vowel counting for a single run. (we have updated the revised version to include these results in Appendix C)

Why are all the tasks restricted to 4 shots? My intuition is that with more few shot examples, the model is more likely to pick up the actual pattern and less likely to copy, therefore, the method will become less effective. This analysis is not to weaken the method — as authors have pointed out that some task might have limited ICL examples as a restriction, and the proposed method is a great way to mitigate the bias. I am trying to get a better understanding on the limitation of the proposed method.

To address this, we conducted additional experiments beyond 4 shots, testing performance with up to 64 shots (updated and reported in Appendix G). The results confirm the reviewer's intuition: as models receive more examples, they indeed become better at learning the underlying patterns rather than relying on copying behaviors, leading to a gradually narrowing performance gap between pruned and unpruned models. However, our pruning method continues to yield consistent improvements even in many-shot settings, albeit with diminishing returns. This pattern suggests that while our method is particularly powerful in few-shot scenarios (where copying bias is most problematic and many real-world applications are constrained), it remains beneficial across different shot regimes by promoting better pattern recognition over copying strategies. We believe this analysis provides a clearer understanding of our method's scope and limitations while reinforcing its practical value in scenarios where example availability is constrained.

For the BBH object counting task, my understanding is that these are complex reasoning tasks that benefit from Chain-of-Thought (CoT) reasoning, which ICL examples could potentially demonstrate. Can the authors clarify if CoT prompting is employed in either the baseline models or the copy-error-pruned method?

In our experiments, we used simple input-output format for all methods to isolate the impact of our pruning approach. While CoT prompting is indeed powerful for complex reasoning tasks like BBH, we observed that copying errors still occur even with CoT - for example, in our vowel counting task, models sometimes copy incorrect vowel counts from demonstration examples even when chain-of-thought reasoning is provided. In additional experiments with Llama3, we found that while Chain-of-Thought prompting improved standard ICL performance by 1.5% in the 1 Shot settings for BBH, our pruning method achieved a more substantial 6% improvement. Future work could explore how our pruning method complements CoT and other advanced prompting techniques.

Similarity of Tasks and Limited Generalizability: The 18 tasks studied in this paper share notable similarities with vowel counting, meaning that the copying neurons identified through Integrated Gradients primarily apply to a set of very related and rather synthetic tasks. While the authors also attempt to show improvements on non-synthetic tasks, I am not sure how to interpret "copying error" on datasets like SST2 and SST5, which are multiple-choice tasks with a limited set of choices (e.g., 2 or 5 options)? In such cases, does the model exhibit a genuine copying bias or if the improvements are driven by other factors? Would the copying bias be easily mitigated by using unbiased ICL samples (for example, one sample for each answer choice A/B/C/D).

Task Diversity Beyond Syntactic Similarity:

While some tasks share surface-level similarities with vowel counting, they require fundamentally different reasoning: (1) Vowel counting: Character-level pattern recognition, (2) Linguistic tasks: Understanding morphological rules (e.g., singular-plural) and (3) Knowledge tasks: Retrieving factual relationships (e.g., country-capital)

The success across this diverse range suggests we're capturing a general copying mechanism rather than task-specific patterns. SST and Multiple-Choice Tasks: We can observe copying bias in sentiment analysis even with limited choices: Models may copy ratings from examples with similar phrases rather than analyzing sentiment, Example: Given a prompt with:

"The movie was okay but boring": Negative

"This film bored me to tears": Negative

"The ending was boring but had good moments": ???

A copying-biased model may predict "Negative" based on the word "boring" appearing in previous examples, rather than analyzing the nuanced sentiment

Our pruning improves the model's ability to consider the full context rather than copying based on lexical overlap. To verify this, we conducted additional experiments on SST2 using balanced ICL demonstrations (equal number of positive and negative examples), results are averaged across 5 different seeds:

These results demonstrate that our method's improvements persist even with balanced demonstrations, suggesting we are addressing a fundamental copying mechanism rather than simply correcting for dataset biases. (We have updated the revised version to include this analysis in Appendix J).

Minor Reproducibility Concerns: The paper lacks details on certain experimental settings, such as the dataset specifications (e.g., training/validation sizes, number of shots) for the vowel-counting data, as well as the hyperparameter settings used for pruning. Including these details would enhance reproducibility and provide greater clarity on the experimental setup.

We appreciate the reviewer's request to add more details on the experimental settings, to addres this, we have added:

Vowel-Counting Dataset Specifications:

Detection dataset: total of 400 ICL examples.

Each example uses 2-shot format.

Examples are constructed to ensure test answers never appear in prompt responses.

Split: 300 examples for computing neuron importance scores, 100 for validation to determine the optimal pruning configuration.

Words sampled randomly from an English dictionary, filtered to ensure diverse vowel counts (0-9)

We carefully design prompts so that the test words have different vowel counts than example words. (2) Hyperparameter settings used for pruning are updated in the revised version (Table 3, Appendix B), including: Pruning rates explored per layer. Layer selection criteria.

Thank you very much for the author’s response. I believe that the experiments you provided are quite comprehensive. However, I still have a few questions that I would appreciate your response to:

• In the reply addressing Q1, the author presented results related to perplexity (PPL) and downstream tasks (MMLU). On WikiText, it appears that the PPL of the model has shown a certain degree of degradation, yet the MMLU score of the model has maintained its original level to some extent. I am somewhat skeptical that the MMLU dataset is “favored” by pruning. For example, in earlier pruning works like ShortGPT or The Unreasonable Ineffectiveness of the Deeper Layers, MMLU has often been a dataset that preserves relatively good performance. My hypothesis is that MMLU-related tasks are among the least sensitive to pruning, whereas other tasks, such as associative recall or safety issues, seem to be less effectively addressed by pruning. Therefore, I suggest the author also provide experiments related to these types of tasks (considering that recall is a commonly used LLM application scenario, such as locating and copying large sections of text from PDFs).
• Thank you for your response to Q2, which has partially resolved my doubts. I recommend testing with datasets such as AlpacaEval2. Such datasets seem to be closer to typical usage scenarios.
• Thank you for addressing Q3. However, I am still very interested in whether data engineering can address this issue. Data engineering appears to be a simpler approach. For example, most commercially available LLMs have not undergone pruning.

Thank you for taking the time to provide such insightful feedback.

How does the pruned model perform on downstream tasks, such as reasoning, math, long-context understanding, instruction following, and safety? While it may not be necessary to conduct an exhaustive evaluation due to computational resource limitations, at the very least, relevant experiments should be included.

To assess potential tradeoffs introduced by our pruning method, we evaluate the pruned models on perplexity (WikiText) and MMLU benchmark. As shown in the following two tables, our method has minimal impact on these capabilities, while MMLU performance remains stable or even improves in some cases, indicating that our targeted neuron pruning successfully mitigates copying behavior while preserving general capabilities. We have updated the draft to include these results with additional models in Appendix F.

Perplexity on WikiText test (lower is better) :

MMLU (higher is better):

Does it matter if the model is a base model or an instruct model? For example, if Bloom-560M is turned into an instruct model or fine-tuned with DPO, will the experimental results change or remain the same?

Following the reviewer request, we report below the results of OPT1.3B base and OPT1.3B Instruct [1]: These results are averaged across the following tasks : (linguistic present simple to gerund, linguistic present simple to past simple, linguistic present simple to past perfect, linguistic_singular_plural, linguistic_antonyms), and across 5 different seeds. (we have updated the draft in Appendix J to include this analysis).

In each cell of the table, on the left side, we report the result of the unpruned version of the model, and on the right side, we show the result of the pruned version using our method. The results demonstrate that our pruning method effectively improves performance for both base and instruction-tuned models, with similar relative improvements observed across different shot settings, suggesting that copying bias and the effectiveness of our mitigation strategy persist regardless of the model's fine-tuning regime.

[1] : https://huggingface.co/facebook/opt-iml-1.3b

Can data engineering help solve this problem? If it can, why do we still need an additional step to prune the model?

We agree that an alternative approach could involve incorporating data or a learning objective designed to prevent this reasoning shortcut from emerging, although this may prove to be nontrivial. One of the key contributions of our submission is precisely the identification and demonstration of the significance of this shortcut, which may be then tackled from various angles. In any case, we believe, our approach remains valuable because many existing models exhibit this behavior. Furthermore, we believe our method serves as a compelling example of how neuron-level editing can effectively discourage shortcuts, with potential applications in other settings — for instance, addressing the ‘lost in the middle’ issue in long-context generation.

Experiment on Alpaca Eval2: This is the experiment that confuses me the most. I am curious about how the authors managed to achieve such a significant improvement in the model’s performance on this dataset through neural pruning. My initial motivation for testing this issue was that if neural pruning could maintain the model’s capabilities without loss (or with minimal loss, which could be mitigated through fine-tuning), it would prove the reasonableness of this method. However, what is most intriguing is that this method not only maintained but also enhanced the model’s capabilities, achieving an improvement close to a percentage point.

I believe Alpaca Eval2 does not contain many experiments that are particularly sensitive to copy bias, so such a significant improvement is quite puzzling. I suggest the authors provide more details about this experiment, such as whether the sampling parameters of the model were aligned and the number of inference runs. Additionally, specifics on the problems and answers where the pruned model outperformed the unpruned model should be included. I strongly suspect this improvement might be due to certain experimental variables, such as randomness caused by an insufficient number of runs.

Also, is the "LLaMA-2" model a base one? I think the authors should use LLaMA-2 instruct version.

Thank you very much for the prompt response.

I recommend testing with datasets such as AlpacaEval2.

We tested on AlpacaEval2 (https://github.com/tatsu-lab/alpaca_eval). Due to computational constraints, we evaluated on half of the test set. For reference, the original Llama2's performance on the full test set achieves 5.4% LC win rate and 5.0% win rate. On our half-set evaluation, the unpruned Llama2 achieves 4.84% LC win rate and 2.69% win rate, while our pruned version improves to 5.26% LC win rate and 3.21% win rate. This (along with results on MMLU) suggests that our pruning method preserves the model’s ability to handle both general language tasks. (The results of this experiment are updated in the revised version).

Therefore, I suggest the author also provide experiments related to these types of tasks (considering that recall is a commonly used LLM application scenario, such as locating and copying large sections of text from PDFs).

We considered the Needle-In-A-Haystack benchmark (https://github.com/gkamradt/LLMTest_NeedleInAHaystack), which evaluates the retrieval capabilities of LLMs. This benchmark asks the LLM to locate a specified key within a long context and accurately reproduce the corresponding value from the identified location. The results (reported in the revised version at Appendix F.2 and below) show only a slight difference in performance between pruned and unpruned versions of the tested model across varying context lengths. The high consistency across context and depth length demonstrates that while our pruning effectively mitigates copying bias in ICL settings, it does not compromise any model’s fundamental ability to process and copy information from long contexts.

In the following table we show the averaged fact retrieval scores (in percentages) for pruned and unpruned Llama2 model for the needle-in-a-haystack benchmark using PaulGrahamEssays as a context ‘high-stack’, and the following sentence as the ‘needle’ : “The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.” tested at different context window sizes ranging from 1000 to 6000 tokens (shown at even intervals), where higher scores indicate better performance at finding the relevant information in the given context length.

Llama-2 7B:

I am still very interested in whether data engineering can address this issue.

While data engineering during pretraining or alignment could in principle help address copying bias, reliably testing this hypothesis would require extensive modifications to training data and procedures - a significant undertaking that warrants its own dedicated investigation. However, we have concretely tested the post-hoc data engineering approach through fine-tuning experiments on our proxy task data (see Table 18, Appendix J). Our results show that fine-tuning fails to mitigate copying bias effectively, with performance actually degrading in some cases compared to the base model. This suggests that simply retraining on carefully engineered data may not be sufficient to address the fundamental copying mechanisms in these models. We welcome suggestions for specific data engineering approaches that could be empirically evaluated against our method. If we won’t be able to implement them during the discussion period, we will at least try to incorporate them in the final version of the paper.