We thank the reviewer for the positive feedback and thoughtful questions. We address them below:

1. "The current scope of the 'knob' is limited to toggling between context and parametric knowledge, but the study doesn’t investigate whether this mechanism might extend to other desirable behaviors, such as controlling for safety, reducing bias, or maintaining consistency. Does the subspace affect other behavior of the LLM which may or may not be desirable?"

This is a very insightful point. We have not yet tested whether the identified subspace influences behaviors beyond the context-parametric knowledge tradeoff. Investigating these potential effects would indeed be fascinating and could open up new research directions. We agree that understanding the broader implications of the subspace on other behaviors could yield valuable insights, and we plan to explore this in future work.

2. "A comparison against previous techniques, mentioned in Section 2, would showcase the usability of the proposed technique."

We acknowledge that a direct comparison with previous techniques could help illustrate the practical advantages of our method. However, we believe that such an analysis would be beyond the scope of our current paper. The methods mentioned in Section 2 often differ fundamentally in both approach and goals, and a comprehensive comparison would require a full interpretability analysis for each technique. This would divert focus from our core contribution, which is to provide insights into the model’s internal mechanisms related to context and parametric knowledge reliance. Nonetheless, we recognize the importance of contextualizing our work within the broader landscape of interpretability research and added a brief elaboration in the last paragraph of section 2 on other mechanistic studies of this behavior in the revised paper.

We thank the reviewer for their thoughtful feedback and for acknowledging the clarity and quality of our writing. We have carefully considered each point raised and addressed them in detail below.

Weakness 1: "While the paper is well-written and clear, the framing around main contributions can be made more precise (see questions to authors)."

We agree that the framing of our main contributions could be more precise, and we have revised the introduction to make our research goals clearer and more explicitly defined. We appreciate this constructive suggestion.

Responses to Specific Questions

1. “These findings suggest a step forward toward developing more robust language models with controllable levels of reliance on context and prior knowledge.”

It would be great to describe in more detail why practitioners might want a knob to control context sensitivity if it depends on individual samples. You raise an important point. First, we acknowledge that dynamically setting this "knob" is an open research area that we envision for future work. Our main contribution is focused on understanding how context sensitivity operates within the model, rather than providing a fully automated solution. However, we can indeed propose potential practical scenarios: The "user" who adjusts this knob does not have to be a human operator—it could be an automated system. For instance, imagine a classifier that determines, on a sample-by-sample basis, whether a given prompt would benefit from more reliance on external context or the model’s parametric knowledge. In this way, the knob could be dynamically adjusted to optimize performance without human intervention. This opens avenues for systems that intelligently and adaptively manage the model’s behavior.

2. "Similarly, why not just fine-tune for controllability over context reliance/parametric knowledge reliance if it’s already a step of the procedure? Why try first fine-tuning, then identifying the 1-d subspace, then setting its value?"

This is also an excellent question. These steps: first fine-tuning, then identifying the 1-d subspace, then setting its value, are part of our novel interpretability technique that we introduce in this paper. The key idea underlying our technique is to exploit the similarity between base and fine-tuned models in order to draw insights about the base model. In other words, the value of our approach lies in uncovering information about the base models through fine-tuning. In order to find a subspace which governs the model’s decision, we first need a model which can controllably pick between context and prior knowledge, so that we can compare the model’s mechanisms in each case. Fine-tuning is one such approach to achieve this model capability. We acknowledge that, as ICL also performs very well, we could probably also use the same method to find the subspace in the ICL model (however, fine-tuning tends to outperform ICL and this would shift compute costs from fine-tuning to inference-time ICL examples when learning the projection). Once we have a controllably context sensitive model via fine-tuning, we can discover subspaces within the model that govern how it balances context and prior knowledge. These discovered subspaces generalize to the non-fine-tuned models and we show that by going back to the base model and intervening on our subspace there.This offers a novel interpretability advantage, as it reveals inherent properties of the base/instruct model that are not obvious without this fine-tuning. Even though this is not at all the goal of the paper, we can also present you an application scenario, where such subspaces might be preferred to fine-tuned versions of a model: In real-world deployments of LLMs, where context sensitivity must be adjusted on the fly, traditional methods like injecting in-context examples or loading fine-tuned models have serious drawbacks. ICL is computationally expensive and struggles with base models, while loading fine-tuned models creates memory and batch-processing inefficiencies. For example, if we have 2 user requests, one of which requires context sensitivity and one of which does not, we would need to process this in 2 batches on two models (the finetuned model and the normal, non-finetuned model). Our steering approach, by contrast, requires minimal overhead (just storing d_model additional floats) and allows flexible, per-batch-element adjustments. This could prove especially useful in a future in which model providers have multiple such subspaces that they can control without changing the model itself. We also want to emphasize that we did not pursue any research into this area and have only looked at single word/number answers given a query.

We thank the reviewer for their questions and comments. We understand the concerns raised and believe there is a misunderstanding in the goals of this paper. As such, we would like to clarify the motivation and positioning of our work in the research landscape. Consistent with the interpretability community, our aim is not to propose a method that surpasses fine-tuning in terms of performance metrics but rather to improve understanding of the underlying mechanisms behind resolving knowledge conflicts. This is an important first step toward understanding language model behavior in more complex settings integrating context and prior knowledge, such as retrieval-augmented generation, in-context learning, and combating misinformation. We now address each of the reviewer's points in detail:

Weakness 1: "The proposed steering methodology doesn’t seem to outperform the fine-tuning baseline, so why would practitioners adopt this more complex approach?"

At its heart, our work is focused on interpretability, i.e., improving our understanding of how and why models choose to follow context or prior knowledge at a mechanistic level. Importantly, the goal of our study is not to find a steering method that is more effective at controllable context sensitivity than a model fine-tuned on the task. The purpose of the evaluation comparing steering performance to fine-tuned model performance is primarily as a point of comparison for faithfulness: that is, how faithfully can this subspace explain the model’s behavior between context and prior knowledge? If the steered model can achieve 100% PairAccuracy, this suggests that the subspace fully determines the model’s behavior and would represent the highest level of faithfulness. We show that in-domain, the steered model achieves 83% PairAccuracy, which is relatively high (especially compared to prior works which seek to identify mechanisms facilitating the context vs prior knowledge behavior, such as https://arxiv.org/abs/2310.15910 and https://arxiv.org/abs/2402.11655; while those works do not use a formal intervention-based evaluation of faithfulness, we can infer that the PairAccuracy of their methods are both at most ~50%). Then, figure 3 shows that steering with this projection is highly faithful (between 73-88%) on many model configurations despite being learned on just one, and that steering outperforms the baseline on many of those model configurations. These results serve the interpretability focus of our study by validating that the setting of this subspace has high faithfulness to the model’s observed behavior.

There might still be practical benefits to a steering method, although we neither investigated this nor was this the goal of our paper. Consider the case of serving LLMs in real-world applications where the sensitivity of model responses needs to be adjusted dynamically based on user requests (users might want to control sensitivity or just use the normal model). There are two basic solutions: injecting in-context learning (ICL) examples or loading fine-tuned models. Both have drawbacks. ICL often requires a significant increase in computational resources due to an expanded context length, and it generally performs poorly on base models. Loading a fine-tuned model on demand involves substantial overhead, as it requires either loading or keeping the PEFT updates in memory. When mixed requests arise, this would require small batch sizes, as we can only activate or deactivate the PEFT update. In contrast, our steering approach incurs minimal overhead (you have to store d_model additional floats) and allows for flexible, per-element steering within a batch. This capability becomes especially appealing when envisioning future model deployment scenarios, where multiple steering projections could be present for efficient, user-specific adjustments.

Thanks for your thorough engagement with our paper! We appreciate that you highlighted that our work is “well-motivated and of substantial interest”, “well-written/easy to follow”, with methods that are “particularly creative and interesting” and “generally comprehensive” experiments. We also appreciate your thoughtful feedback and questions; in this rebuttal, we hope to bring us toward the same understanding on the stated weaknesses.

Weakness 1 “it overstates the novelty of proposing and providing evidence for the research question that there is a simple fundamental mechanism in LMs that modulates whether LMs defer to information in context vs in weights.”:

We thank the reviewer for raising this important point and helping us better situate our work in the existing literature. We completely agree that it’s important to acknowledge their contributions towards understanding the mechanism for balancing context and prior knowledge. We have rephrased the relevant sections in the introduction to refer to the mentioned works and better reflect our contribution.

We further want to acknowledge that we as a field are still far from fully understanding how exactly the underlying mechanism works. For example, it is not self-evident that “the same” mechanism always forms (even in different model architectures, trained on different datasets). We see our work as an important addition to the existing body of literature and understanding of the underlying mechanism by emphasizing that (1.) the mechanisms underlying the decision between context and prior knowledge in our CCS setting can be elegantly controlled in both directions with our single one-dimensional subspace, (2.) that we find similar mechanisms as the related work that mostly studied Pythia, GPT2, GPT-J, and Llama 2 also in a range of modern LLMs like Llama 3, Mistral and Gemma, (3.) that our one dimensional subspace transfers between base and fine tuned models.

In summary, we thank you for your valuable input, which has led us to refine the wording of our contribution and our position in relation to related work. We believe this has improved the quality of our work and your feedback is greatly appreciated. Please let us know if the rewritten contributions and their novelty fits your understanding of the field now. Further, we propose rephrasing our focus as a search for a “fundamental mechanism in the form of a subspace”, to clearly differentiate our work from prior literature, but are also open to incorporating any alternative phrasing the reviewer might suggest here.

Weakness 2 “Missing Related work”:

Thank you for suggesting many related works. We have now (a) referenced your suggested works in the introduction and section 2, and (b) expanded on Yu et al 2023 and Ortu et al 2024 (and their shortcomings) to contextualize where our work fits in.

Weakness 3 “Missing Investigation of MLPs and direct residual stream patching”:

Thank you for your detailed feedback. We appreciate your suggestion to broaden our approach beyond just focusing on MHSA outputs, especially given the significant role that MLPs are thought to play in encoding factual knowledge. To clarify, our decision to concentrate on MHSA was guided by recent research (e.g. https://aclanthology.org/2023.emnlp-main.751.pdf, https://arxiv.org/pdf/2402.18154) which provides evidence that contextual answers are often processed at earlier token positions and later transported to the final token by attention heads. In fact, during our initial explorations, we did consider the role of MLPs but found that they didn't have a strong influence on the last token’s forward pass in our case. To clarify this, we’ve included more experiments on patching the MLP when investigating the prior answer in Appendix B that illustrate this limited impact. Regarding the lack of localization of the contextual answer mechanism: We believe our results do localize where the contextual answer is integrated into the residual stream, in particular in post layer 24 layers. The MHA prior to layer 24 do not seem to integrate information about the contextual answer. In Appendix Figure 7b, we highlight how layer 24 is pivotal, underscoring the emergence of the answer mechanism at this stage. This is especially interesting as we show that the intent is imported much earlier (12-16). The intent seems to be decoupled from how the context answer is integrated. Lastly, we acknowledge the appeal of interventions at the residual stream. While we tried this earlier on, it has some drawbacks. Intervening directly at the residual stream level overwrites existing information, which makes it difficult to isolate the contributions of specific components like MHSA or MLPs. By targeting these components individually, we gain finer insights into what each layer contributes. Nonetheless, we have put some additional experiments with patching the residual stream and showed that patching the MHA leads to a significantly better understanding.

We wanted to ask reviewer 1YMS if they have any questions about our response or updated submission. Please let us know if there is anything else that needs to be clarified or elaborated on to help with your concerns.