Emergent Response Planning in LLMs

We sincerely appreciate your detailed and constructive feedback. In response to your inquiries:

Q1. Detailed illustration of probing strategies.

Thank you for the helpful suggestion. To improve clarity, we will add this explanation to the final version: Probing trains auxiliary models (e.g., MLPs) to predict attributes (e.g., confidence, truthfulness) from LLM hidden states, revealing what they encode. Our method involves: (1) generating and labeling LLM responses w.r.t target attributes, (2) extracting hidden states at the first token, and (3) training MLPs to predict labels from these states. Accuracy reflects encoding strength. Implementation details are in the paper.

Q2. If probing demonstrates that LLM hidden states encode sufficient information for self-prediction, why do models fail to leverage this capability during standard inference?

We really appreciate your thoughtful insights. Previous work has shown that internal knowledge in LLMs is not always directly reflected in outputs [1]. Section 5.3 of the main paper supports this hypothesis in the context of response planning, showing that simple prompting fails to predict LLM response attributes. We expand on this with additional tasks, showing a consistent probing–prompting gap (Table 1). LoRA fine-tuning helps reveal this internal planning capability during standard inference (Table 2). Since LoRA does not alter LLM representations grastically, it likely elicits hidden states already present in the original model, supporting our argument that LLMs possess internal planning abilities.

[1] Li, Kenneth, et al. "Inference-time intervention: Eliciting truthful answers from a language model." Advances in Neural Information Processing Systems 36 (2023): 41451-41530.

Table 1. LLMs possess an internalized but unelicited capability for self-prediction, as evidenced by consistent probing-prompting gaps across 6 tasks and 8 model variants (see table below). Each cell shows [prompting / probing / gap], with bold gaps highlighting unelicited potential.

Value Format: self-estimate / probing / GAP (probing - self-estimate, bold)
Models: M=Mistral, Q=Qwen2, L2=Llama-2, L3=Llama-3, -I=Instruct, -C=Chat
Metric↑: ↑=Higher is better • Random: 0.25=4-class, 0.33=3-class, 0.50=2-class baseline

Table 2. This capability can be elicited through parameter-efficient fine-tuning. As shown above, LLMs may fail to leverage internal representations for self-prediction. To address this, we fine-tune models (via LoRA, rank 64) on the 3K-sample UltraChat dataset for token-length prediction. Post-tuning results across three seeds confirm: (1) acquired self-prediction, (2) in-distribution and cross-dataset generalization, and (3) preserved token-generation behavior (consistent response lengths). This demonstrates that minimal fine-tuning (2% parameters) enables generalizable self-prediction.

Q3. Further exploration of the LLMs' internal emergent response generation mechanisms.

Thank you for your insightful advice. We fully agree that understanding the internal mechanisms behind LLMs’ emergent response planning is essential for advancing interpretability. However, our current focus is on establishing the existence and characteristics of emergent response planning. We leave a deeper exploration of the underlying interpretability mechanisms to future work.

We are grateful for your comprehensive remarks and thoughtful advice, and we are really glad that you find our paper interesting.

Q1. Defining "planning" and designing experiments to prevent spurious correlations.

We really appreciate your thoughtful insights. Indeed, results may be biased by "first-token shortcuts" when the first token correlates with the target response attributes, as your example demonstrates. Our approach addresses this via:

(1) Defining planning as independent encoding of next-token and long-term attributes: We define planning as hidden representations at the first token encoding both next-token information and long-term attributes, with these two types of information being independent—i.e., long-term attributes must not be reflected by the immediate next token. In practice, we ensure such independency through careful prompt enigneering, as we will discuss in the next paragraph.

(2) Prompt engineering to block shortcuts in experiment design: We design prompts to ensure initial tokens cannot reveal target attributes. For example, in multiple-choice tasks, we require models to analyze the question before answering, ensuring first tokens (analysis) do not leak the answer. This isolates true planning from shortcut correlations.

We will include a detailed discussion of this important methodological consideration in the final version of our paper.

Q2. Performance gaps between probing and LLM self-prediction is less thoroughly explored.

Thank you for your valuable suggestion. We extend our analysis of probing vs. self-prediction to 6 tasks and 8 model variants (see table below, with each cell shows [prompting / probing / gap]). Prompting occasionally underperforms random baselines, likely due to LLMs' overestimating their capabilities (e.g., overconfidence in correctness) and biased self-evaluation. The results further support our claim that models encode more planning information in their hidden representations than they can explicitly access during token-by-token generation.

Value Format: self-estimate / probing / GAP (probing - self-estimate, bold)
Models: M=Mistral, Q=Qwen2, L2=Llama-2, L3=Llama-3, -I=Instruct, -C=Chat
Metric↑: ↑=Higher is better • Random: 0.25=4-class, 0.33=3-class, 0.50=2-class baseline

Q3. Suggestions on enhancing the clarity of Figure 4.

Thank you for your constructive advice. We will carefully incorporate the following edits in the final version:

1. Adding a color legend to clarify grid values;
2. Replacing "deep to shallow →" with "early to late layers →" to resolve ambiguity about layer ordering;
3. Explicitly stating the model order (top to bottom): Mistral-7B-Instruct, Llama-2-7B-Chat, Llama-3-8B-Instruct, Mistral-7B, Llama-2-7B, Llama-3-8B, Qwen2-7B-Instruct, Qwen2-7B (first six: 32 layers; last two: 28 layers).

Thank you for the time, thorough comments, and nice suggestions. We answer the comments/questions point-by-point:

Q1. Adding annotations on model orders of Figure 4.

Thank you for your constructive feedback. We agree that explicitly annotating model orders will improve clarity, and we greatly appreciate your attention to this important detail. In the final version, we will clearly annotate the model order in Figure 4 (top to bottom) as: Mistral-7B-Instruct, Llama-2-7B-Chat, Llama-3-8B-Instruct, Mistral-7B, Llama-2-7B, Llama-3-8B, Qwen2-7B-Instruct, and Qwen2-7B. For clarity, we will also note that the first six models (Mistral and Llama variants) use 32-layer architectures, while the Qwen2-7B variants employ 28 layers.

Q2. Explaining Qwen2-7B-Instruct's divergent confidence prediction patterns.

Thank you for your insightful observation. The divergent behavior of Qwen2-7B-Instruct (second-to-last row in figures) arises from two factors: (1) its shallower 28-layer architecture (vs. 32 layers in Mistral/Llama models), which alters attribute encoding dynamics across layers, and (2) instruction-tuning—evident when comparing Qwen2-7B-Instruct with its base model (Qwen2-7B)—introduces shifts in how attributes like confidence are encoded. We observe that Qwen models generally encode attributes in later layers compared to Mistral/Llama. While the base Qwen2-7B exhibits uniform layer-wise behavior across attributes, its instruction-tuned variant shows pronounced divergence—a general trend that is most salient in confidence prediction tasks. We will clarify these architectural and training effects in the final paper.

Q3. Explanation for layer-wise performance analysis on behavioral attributes tasks.

Thank you for your thoughtful and detailed observation. The interpretation of Figures 4e and 4f hinges on the distinction between relative saliency dynamics (emphasized by our layer-wise normalization) and absolute encoding strength (which appears subtler in the current normalized presentation). While early layers in Figures 4e/f do exhibit higher absolute values (indicative of early-stage encoding), the normalized visualization prioritizes cross-layer trends, which can attenuate static contributions. For instance, Figure 4c shows comparable relative trends in initial layers, but its absolute activation levels (not explicitly highlighted here) differ from those in 4e/f. This nuance suggests that while normalization reveals functional dynamics, it may not fully capture the magnitude of early encoding. We sincerely appreciate your feedback, which has prompted us to refine the figures to better distinguish absolute vs. relative patterns in the final version.