We appreciate your recognition of the value in our study, particularly in uncovering the instruction-following direction as a practical tool for steering models toward better adherence to instructions. Your acknowledgment of its potential to detect instruction-following failures before response generation underscores a key application of our findings. We are also grateful for your recognition of our contribution to advancing the scientific understanding of LLM behavior and for highlighting the robustness and reproducibility of our experiments across multiple models.

---

W1. Comparing with few-shot prompting

Thank you for this insightful suggestion. Our study specifically focuses on simple and non-ambiguous instructions to isolate the instruction-following dimension under controlled settings. Prompting methods like few-shot prompting or Chain-of-Thought (CoT) are designed to address more complex scenarios, such as ambiguous instructions [1] or tasks requiring multi-step reasoning. Since the instructions in the IFEval dataset are clear and straightforward, we did not include these advanced prompting techniques as baselines.

That said, comparing representation engineering (RE) with few-shot prompting could indeed be an interesting direction for future work. Such experiments might uncover unique benefits or limitations of RE, even within the context of simple, clear instructions. We appreciate your suggestion and will highlight this as a potential avenue for further exploration in our revised manuscript.

[1] Mann, Ben, et al. "Language models are few-shot learners."(2020).

W2. Clarification

I don’t see how this helps explain why models sometimes fail to follow clear instructions or why prompt engineering helps.

Thank you for highlighting this important point. Let us clarify how our findings connect to these observations. Our study uncovered a linearly separable geometry in LLM representations for success and failure cases, with a sensitivity to phrasing modifications. This sensitivity suggests that the LLM’s internal representation space is not robust to small changes in phrasing, similar to how deep neural networks without adversarial training are sensitive to small perturbations on images.

This sensitivity explains two phenomena:

1)Why LLMs fail to follow clear instructions Even when instructions are clear and simple, slight differences in phrasing in the input space can shift the internal representations to a failure class in the representation space. This arises from the limitation of LLMs; non-robust geometry of their representation space.

2)Why prompt engineering helps Since the instruction-following dimension is sensitive to phrasing, when prompt engineering changes phrasing, as a result, it can effectively move a failure example into the success class in the representation space.

While our work does not fully explain why this sensitivity exists or how it originates during training, it provides a foundation for understanding and addressing this limitation.

W3. Clarification

The authors ask the question “Do ...

We appreciate your observation and we will clarify this earlier in the revision.

W4. Question on Larger models

My impression is that more recent LLMs are much more reliable ...

Thank you for raising this point. We agree that larger foundation models (e.g., recent GPT-4, Anthropic models) tend to be more robust to phrasing modifications, likely due to their extensive training on massive and diverse datasets that include a wider variety of instruction forms, improving both robustness and generalizability [1]. Our analysis focuses on models with comparably smaller models, which are widely adopted for on-device applications and fine-tuning. However, we acknowledge that insights from these models may not fully capture the behavior of larger, state-of-the-art models. Extending our analysis to larger models is indeed a promising direction for future work, and we thank the reviewer for emphasizing this important area of inquiry.

[1] Zhu Kaijie, et al. "Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts." (2024).

Q2. Define “know internally”

Yes, your interpretation is correct. By “know internally,” we mean that the model encodes information in its representations that correlates with whether its response will follow the given instruction. This terminology aligns with prior work [1] analyzing hallucination, which refers to similar separability in the representation space for truthful versus hallucinated outputs. We will clarify this definition explicitly in the paper to avoid confusion. Thank you for highlighting this point.

[1] Amos Azaria et al. The internal state of an llm knows when it’s lying. 2023.

---

We sincerely thank the reviewer for their thoughtful feedback and constructive suggestions. We hope our responses have addressed your questions and clarified the scope and implications of our work.

W1. I still think few-shot would be a good comparison to run.

W2.

While our work does not fully explain why this sensitivity exists or how it originates during training, it provides a foundation for understanding and addressing this limitation.

The first part of this sentence is crucial. I think more work is needed to explain these things.

W4. I think frontier models like Claude Sonnet 3.5 are much more robust to paraphrasing prompts. This undermines the explanatory potential of this work -- which finds representations that are sensitive to prompt phrasing.

While some of the rebuttals were helpful, they didn't resolve the most important issues raises in the review. Hence I keep the same score.

We sincerely thank the reviewer for their thoughtful assessment of our paper and for highlighting several strengths. We are glad that you found our work novel and impactful.

---

W1. Data scope

Thank you for your thoughtful comments. We kindly refer you to the general response section.

W2. Reverse RE

Thank you for your insightful question about reverse representation engineering. We conducted initial tests on two models, Phi-3-mini-128k and Mistral-7B-inst-v0.3, using reverse RE to move representations toward the failure class ($-\alpha \times D$).

We used conservative $\alpha$ values to maintain a similar quality ratio (QR) for reverse RE and random directions (QR: 0.86 for Mistral, 0.77 for Phi). As expected, reverse RE yields worse success rates (SR) than random directions, but the difference is not yet significant. We believe optimizing $\alpha$ on a validation set will further emphasize the difference between reverse and random directions. Additional experiments are planned to refine $\alpha$ and better assess reverse RE’s impact on instruction adherence.

W3. Suggestion on finding multiple directions

Thank you for raising this insightful point. Considering multiple directions related to instruction-following is indeed a promising avenue for future research. In our current work, we focus on a single direction associated with success in instruction-following as an initial exploration of the geometry of LLM representations in this context. This approach provides a foundational understanding but does not preclude the possibility of additional meaningful directions being present.

Future work could extend this analysis to multiple directions using techniques such as dictionary-based learning to uncover latent factors in an unsupervised manner, or by incorporating human-defined directions, as demonstrated in the paper you suggested. Expanding RE to consider multiple directions could reduce the risk of task degradation or safety concerns. Also, visualizing these directions using methods like sparse autoencoders (SAE) or other interpretability techniques could yield further insights into the nuanced geometry of instruction-following in LLMs. We appreciate your suggestion and will include these points in the discussion section of our revised paper.

W4. Question on parameter alpha

Thank you for your observation. First, we want to clarify that in our current implementation, we require a single $\alpha$ parameter for each model, which is then applied uniformly across all instruction types. This ensures consistency without needing separate parameter fitting for each instruction type.

Our approach uses a basic form of representation engineering (RE) that relies on $\alpha$ to adjust the representations. However, we believe that integrating more advanced RE methods, such as those proposed in [1], which eliminate the need for explicit parameter search, could address this limitation and improve the practical applicability of RE. We discuss this direction as a promising avenue for future work in lines 534–535.

[1] Jinqi Luo, et al. Pace: Parsimonious concept engineering for large language models. arXiv preprint arXiv:2406.04331, 2024.

W6. GPT-4 evaluator

Thank you for pointing this out. As you pointed out, we acknowledge that the interpretation of raw task quality scores from GPT-4 evaluator may be not clear. Thus, rather than directly using the raw task quality scores, we employ a Quality Ratio (QR) metric; the proportion of responses scoring above a threshold 7, where this threshold is decided based on the distribution of GPT-4 quality scores.

W7. Suggestion

Thank you for your suggestion regarding uncertainty estimation. We agree that exploring additional methods, such as bootstrap resampling, could provide a more comprehensive understanding of the underlying variability. As mentioned in our future work section (line 533-534), we plan to apply alternative linear probing techniques to enhance AUROC performance and improve uncertainty quantification. This will further strengthen the robustness of our findings in future iterations of this work.

Q1. Clarification

Thank you for pointing this out. The middle token’s representation, $LLM(x_1, x_2 …, x_n, y_1, y_2 …,y_{m//2})$, where ${x_1, x_2, …, x_n}$ are the  n  tokens in the input prompt and ${y_1, y_2, …, y_m}$ are the m tokens in the response. The middle token’s position varies depending on the length of the response for each specific prompt, as it is determined dynamically as m//2 . We will clarify this in the revision.

---

We sincerely thank the reviewers for their thoughtful feedback and constructive suggestions. We hope our responses have addressed your questions and provided greater clarity on our work.

Thank you very much for your thoughtful review and valuable feedback. We appreciate your recognition of the novel insights provided by our study into instruction-following in LLMs. We are also grateful for your positive feedback on our experimental design and the clarity of our writing.

---

W1.

Some of the conclusions in this paper are as expected, e.g., results from Table 1 and Table 2 are related to the “Lost in the Middle” phenomenon. [1: Lost in the Middle: How Language Models Use Long Contexts]

The “Lost in the Middle” phenomenon observed by [1] primarily discusses performance degradation in models due to the position of relevant information in long contexts. In contrast, our prompts are short—typically just two sentences (examples are provided in Tables 6 and 7 in the Appendix). Also, our analysis focuses on token representations during different stages of ‘response generation’ (before, during, and after), rather than different stages of the input prompt (noted in lines 157-158, Section 2.2). Given these differences in focus and context length, we believe our observation in Table 2 addresses distinct aspects of representation use in LLMs. Please let us know if there is an additional aspect of [1] that you believe would be relevant to our work.

W2. Question on RE results

The performance improvement brought by this method is limited, as shown in Table 4.

We want to highlight that our primary aim is to analyze and understand the internal states of LLMs in relation to instruction-following, rather than to introduce a method for improving instruction-following accuracy. In this context, representation engineering (RE) serves as a validation step to assess whether the instruction-following dimension we identified significantly impacts model behavior, as described in Section 3, rather than aiming to improve instruction-following success accuracy. In our paper, we apply basic form of RE, thus we believe that integrating more advanced RE methods could further improve instruction-following performance and that our findings lay a foundation for this exploration, as we discuss in future work (Section 6.3, lines 534-535).

W3. Data scope

The IFEval dataset is small, and this limitation may be addressed by extending instruction-following data.

Thank you for your thoughtful comments. We kindly refer you to the general response section, where we address the rationale for choosing IFEval as our primary dataset instead of using real-world dataset and the potential to extend analysis to more complex, real-world tasks in future studies.

Q1. Reverse RE

Have you tried reverse representation engineering? Will it be worse than random?

Thank you for your insightful question regarding reverse representation engineering and its impact on instruction-following. We conducted initial experiments on reverse representation engineering with two models: Phi-3-mini-128k and Mistral-7B-inst-v0.3. In these tests, we try to move representations towards the failure class by flipping the adjustment vector ($-\alpha \times D$).

Notably, we set the $\alpha$ values conservatively to keep the quality ratio (QR) of reverse RE remains similar to that of random directions (0.86 for Mistral and 0.77 for Phi). The results indicate that the success rate (SR) for reverse RE is worse than random directions, as expected, but the difference is not significant. We anticipate that finding $\alpha$ on a validation set will amplify the difference between reverse and random directions. We plan to conduct additional experiments to refine \alpha and better evaluate the effectiveness of reverse RE in disrupting instruction adherence.

Q2. Clarification

Why did you focus on the representation of the first token in the last layer? Any insights?

Thank you for asking about our methodological choices. We discuss this in Section 3.1 (lines 306–311), and for convenience, we include the explanation below: “we applied adjustments to the representations in the last layer, as it was found to be more robust to variations in $\alpha$. The first token represents the model’s initial input embedding, capturing the model’s state before response generation begins. Since RE aims to adjust internal states early to improve instruction adherence, focusing on this initial representation aligns with our objective.” We hope this clarifies our choice.

---

Thank you once again for your thoughtful feedback and for recognizing the value and clarity of our work. We believe your suggestions will help further enhance the manuscript, and we look forward to incorporating these improvements.

We really appreciate for your positive and constructive feedback and recognizing the strengths of our work. We are pleased that you found the paper well-structured and appreciated our in-depth follow-up analyses using representation engineering, which go beyond identifying a latent dimension to demonstrate its practical implications. We also value your recognition that our claims are robustly supported by experiments and evidence, underscoring the novelty and significance of our findings.

---

W1. Clarification on the direction $D$

In section 3, I'm confused what the direction $D$ taken is for different instructions, since the previous section states that linear probes do not generalize well across instructions. Does this mean representation engineering promotes different directions for different instructions, or is there a universal direction that enhances all instructions generally? It would be helpful to clarify this in the paper.

Thank you for this thoughtful question. In Section 3, we use a universal direction $D$, derived from training on all IFEval-simple data, to enhance instruction-following performance across all instruction types. While $D$ is evaluated on unseen instruction types in Section 2 to test generalizability, in Section 3, it is applied to the same instruction types seen during training, ensuring consistent performance improvements within this scope. We will clarify this distinction in the revised version to ensure greater clarity and address any potential confusion.

W2. Clarification on terminology

For section 4 as well, one instruction type is chosen to perform the analysis on correlations. Could you elaborate on how much the findings in this section generalizes to other instruction types? Or could there be instruction types whose representation is more correlated with task familiarity rather than phrasing?

Thank you for raising this insightful point. Exploring how sensitivity to different interventions, such as phrasing modifications, task familiarity, or instruction difficulty, varies across different instruction types is indeed an interesting direction. We will note this as a promising avenue for future work.

Q1 & 2. Suggestion

At the start of section 2, it would be helpful to elaborate what the IFEval dataset is used for and Make text in Figure 1 bigger

We appreciate this suggestion. We will add a brief explanation of the IFEval dataset’s role at the start of Section 2 to provide better context. Additionally, we will adjust the figure to improve readability.

---

Once again, thank you for your thoughtful feedback and positive evaluation of our work. We look forward to incorporating your comments to enhance clarity and presentation.

Thank you very much for your thoughtful review and valuable feedback. We appreciate your constructive comments, highlighting that our paper provides valuable, actionable insights into the internal representations of LLMs and effectively builds a coherent narrative, from identifying the instruction-following dimension to manipulating and interpreting it.

---

W1. Notation

The methodology description lacks clarity due to insufficient mathematical notation. It is especially difficult to follow the procedures outlined in Sections 3 and 4. The authors should clearly define mathematical objects, such as "input representation," and provide precise equations for the computations (e.g., lines 370-374). Specifically, given a prompt ( x_1, x_2, ..., x_n ) of n words and an output ( y_1, y_2, ..., y_m ) of m words, what is the "input representation" as a function of these tokens? It appears to be computed solely from the output words.

Thank you for pointing this out. To clarify, the “input representation” refers to the model’s representation after processing only the input prompt, denoted by $LLM(x_1, x_2 …, x_n)$. Here is more details about three representations we analyze in lines 154-161: (1) the first token’s representation, which is before generating the response $LLM(x_1, x_2 …, x_n)$; (2) the middle token’s representation, $LLM(x_1, x_2 …, x_n, y_1, y_2 …,y_{m//2})$, representing the model state halfway through response generation; and (3) the last token’s representation, $LLM(x_1, x_2, …, x_n, y_1, y_2 …,y_m)$, representing the model state after completing the response.

In Section 4 (lines 370-374), we conduct a sensitivity analysis to understand the impact of various prompt modifications on these internal representations. This analysis evaluates how different perturbations (phrasing, task familiarity, instruction difficulty) affect the LLM’s instruction-following capability, described in lines 375-418. We will clarify these definitions and notations in the revision to make these concepts more accessible.

W2. Terminology

The terminology is confusing. The term "instruction" would be more accurately described as "constraints." It’s challenging to distinguish "instruction" from "task," as "instruction" could imply the entire command given to the model. Additionally, the meaning of "dimension" is unclear. Does it refer to a hypothetical concept rather than an actual dimension of a hidden vector computed by the model?

We use the term “instruction” to align with the IFEval dataset terminology, referring to specific guidelines or constraints (e.g., “please do not use keywords”), while “task” refers to the broader context or content (e.g., “please write a resume”) within which the instruction is applied (lines 51–78). However, we understand that “instruction” could imply the entire command given to the model, which might lead to confusion. To address this, we will revise the text to clarify that the term “instruction” specifically refers to constraints applied within a task context.

The “instruction-following dimension” refers to an actual direction in the model’s representation space, identified through the weights of a linear probe trained to distinguish between successful and unsuccessful instruction-following instances. To make this clearer, we will adjust the terminology and provide explicit definitions in the revision. Thank you for pointing this out.

W3. Data Scope

The experiments were conducted on a single, relatively small dataset, raising questions about whether the findings generalize.

Thank you for your thoughtful comments. We kindly refer you to the general response section, where we address the rationale for choosing IFEval as our primary dataset instead of using real-world dataset and the potential to extend analysis to more complex, real-world tasks in future studies.

Q. Clarification on motivation

Could you explain the motivation and intuition behind the update in lines 303-304?

Thank you for the question. The update in lines 303-304 involves applying representation engineering to shift each input representation towards the instruction-following direction $D$. Here, $D$ is a vector that points toward successful instruction-following cases in the model’s representation space, while -D points towards failures. By adding $\alpha \times D$ to the original representation $R_{original}$, we effectively shift $R_{original}$ closer to the success class, improving the model’s adherence to instructions. This update leverages the instruction-following direction identified through linear probes, providing a mechanism for improving performance in instruction-following tasks.

---

We sincerely thank the reviewers for their thoughtful feedback and constructive suggestions. Your insights have helped us refine the clarity and presentation of our work. We hope our responses address your concerns.