Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models

Thank you very much for your valuable feedback, constructive comments, and appreciation of our experimental setup. In response, we have added an additional examination of LLM judge reliability involving human and cross-model annotation (see Answer 1). Below, we carefully address each of your concerns.

Answer 1 (Weakness 1 & Question 2): The LLM-as-a-judge approach may introduce biases or inaccuracies, as it lacks validation against human judgment.

We appreciate your concern regarding the reliability of using LLMs as evaluators. To rigorously verify the reliability, we conducted additional experiments involving three human annotators and multiple judging models and measure their agreement rates (i.e., the percentage of matching annotations between two annotators) and Cohen’s Kappa coefficient:

• Step Labeling: 3 annotators evaluated 200 randomly sampled reasoning steps (100 self-reflect, 100 continue reasoning) from MMLU Deepseek-R1 traces.
• Behavior Classification and Trace Answer Extraction: 3 annotators evaluated 200 reasoning traces from Qwen3-14B (100 explicitly corrected, 100 consistently followed) across all intervention types on MMLU for Intra-Draft Faithfulness.

Human-Human Agreement Rates:

Human-LLM Agreement (using Majority Vote from human annotations):

We also calculate Cohen's Kappa coefficient between the majority vote of human labels and the LLM for both step labeling and behavior classification. Step labeling yields a coefficient of 0.83, indicating almost perfect agreement. Behavior classification, which is more challenging for humans due to the extended content in the reasoning trace, still yields a coefficient of 0.62, reflecting substantial agreement.

Furthermore, we assessed reliability across multiple LLM judges (Qwen2.5-32B-Instruct, Llama-3.2-70B-Instruct, Qwen3-32B) for behavior classification on three additional models: Deepseek-R1-Distill-1.5B (DS-1.5B), Skywork-OR1-7B (OR1-7B), and Qwen3-14B (faithfulness rate reported in Answer 5 to Reviewer hiiz) over all generated traces on both MMLU and GPQA:

Cross Model Agreement Rates:

These high agreement rates, comparable to human-level consistency, strongly support the reliability of our LLM-based evaluation method.

Answer 2 (Weakness 2 & Question 1): The detailed construction (e.g., content, structure, selection criteria) of counterfactual reasoning steps is omitted.

Thank you for raising this point. We would like to respectfully clarify that these details are, in fact, provided in our paper. Specifically:

• Figure 2 (Section 4.2.1) and Figure 4 (Appendix B.1) provides clear examples illustrating the inserted counterfactual reasoning steps.
• Appendix B.1 explicitly present comprehensive details on selection criteria for insertion locations, the specific content and structure of inserted counterfactual steps, and the precise prompts employed to generate corrupted options.

We acknowledge, however, that including these details primarily in the Appendix may limit visibility and immediate accessibility. Thus, we will emphasize these methodological specifics more explicitly in the main manuscript to enhance reproducibility and transparency with later revision.

Answer 3 (Question 1): What criteria ensure inserted intervention steps are “meaningfully tied to the reasoning process” and “verifiable by an external LLM-based evaluator”?

Thank you for this important question. To satisfy the two criteria, our interventions are carefully designed to mimic naturally occurring reasoning steps—specifically, option restatements (e.g., “Let me verify the options: A)... B)...”)—which are frequently observed in the thinking drafts of multiple-choice reasoning tasks. We manipulate the inner options to meet the criteria as follows:

• Meaningfully tied to reasoning: Since multiple-choice questions inherently require explicit consideration and evaluation of provided options, our interventions directly engage critical reasoning content. By altering these option statements, we ensure that the inserted counterfactuals integrate seamlessly into the logical reasoning flow and can have global implications for the model’s reasoning trajectory.
• Verifiable by external evaluators: Since options are central to the reasoning process, model responses to these interventions are clearly observable in the generated drafts. Models either explicitly correct the manipulated options or consistently follow (or ignore) the alterations, allowing behavior to be reliably classified into two categories: explicit correction or consistent following. This clear dichotomy enables robust external verification using carefully designed LLM-based classification prompts, as detailed in Appendix B.1.

Answer 4 (Weakness 3): The paper does not introduce novel techniques, reducing its contribution to advancing evaluation methodologies.

Thank you for raising this point about technical novelty. We agree that counterfactual interventions have been widely used in prior work on CoT faithfulness. However, our core contribution lies in clearly defining and operationalizing the novel concept of thinking draft faithfulness, and rigorously applying counterfactual interventions to systematically investigate this property in emerging LRMs.

Specifically, we introduce:

• A precise conceptualization of faithfulness within thinking drafts, formalized through two new perspectives: Intra-Draft Faithfulness and Draft-to-Answer Faithfulness.
• Tailored counterfactual interventions specifically designed to address the inherent complexity and multi-path exploratory nature of reasoning drafts in LRMs. To our knowledge, no existing interventions in prior work effectively capture the intricate causal dependencies that emerge within the thinking drafts of recent LRMs.

Thus, while our work builds upon the general idea of counterfactual intervention, the adapted definitions, novel metrics, and customized intervention and evaluation framework constitute meaningful methodological advances. These components are essential for rigorously assessing faithfulness in this emerging context. We will clarify and emphasize this contribution more explicitly in the revised manuscript.

We sincerely appreciate your feedback, which significantly helps us improve the clarity, rigor, and impact of our work.

We appreciate your insightful and constructive feedback. In order to address your questions, we have added the following evaluations:

• An additional examination of LLM judge reliability involving human and cross-model annotations (Answer 1).
• An additional evaluation method using internal model calculations (perplexity) (Answer 3).
• An expanded benchmark that includes a mathematical reasoning dataset (MMLU college math) and three additional models (Answer 5).

We further address each of your concerns in detail as follows:

Answer 1 (Weakness 1 & Question 3): Reliability of using LLMs for judgment and modification.

We appreciate your concern regarding the reliability of using LLMs as evaluators. To rigorously verify the reliability, we conducted additional experiments involving three human annotators and multiple judging models and measure their agreement rates (i.e., the percentage of matching annotations between two annotators) and Cohen’s Kappa coefficient:

• Step Labeling: 3 annotators evaluated 200 randomly sampled reasoning steps (100 self-reflect, 100 continue reasoning) from MMLU Deepseek-R1 traces.
• Behavior Classification and Trace Answer Extraction: 3 annotators evaluated 200 reasoning traces from Qwen3-14B (100 explicitly corrected, 100 consistently followed) across all intervention types on MMLU for Intra-Draft Faithfulness.

Human-Human Agreement Rates:

Human-LLM Agreement (using Majority Vote from human annotations):

We also calculate Cohen's Kappa coefficient between the majority vote of human labels and the LLM for both step labeling and behavior classification. Step labeling yields a coefficient of 0.83, indicating almost perfect agreement. Behavior classification, which is more challenging for humans due to the extended content in the reasoning trace, still yields a coefficient of 0.62, reflecting substantial agreement.

Furthermore, we assessed reliability across multiple LLM judges (Qwen2.5-32B-Instruct, Llama-3.2-70B-Instruct, Qwen3-32B) for behavior classification on three additional models: Deepseek-R1-Distill-1.5B (DS-1.5B), Skywork-OR1-7B (OR1-7B), and Qwen3-14B (faithfulness rate reported in Answer 5) over all generated traces on both MMLU and GPQA:

Cross Model Agreement Rates:

These high agreement rates, comparable to human-level consistency, strongly support the reliability of our LLM-based evaluation method.

In addition to evaluating the judge’s reliability, we will include more qualitative examples of LLM-generated corrupt options and misleading answers, and we will open-source the full evaluation code and data in a later revision. Unfortunately, the current rebuttal phase does not allow external links to share these examples. Notably, similar approaches—using LLMs to inject mistakes—have been widely adopted in prior work [2], further supporting the reliability of this method for content modification

Answer 2 (Weakness 2): Complexity of Faithfulness Definition.

We acknowledge that the definitions of faithfulness may initially appear complex. This complexity arises from the nature of faithfulness itself, which is not directly observable. As in prior work [1,2], we adopt a decomposition strategy to make this concept operational.

Our decomposition into Intra-Draft Faithfulness and Draft-to-Answer Faithfulness captures distinct, practically important dimensions that are essential for monitoring, interpretability, and controllability in reasoning models (see Lines 49–64 in Section 1 and examples in Figure 1, Section 3).

Specifically, Intra-Draft Faithfulness measures both local dependencies (e.g., explicit correction) and global dependencies (e.g., consistent following) among intermediate reasoning steps within the thinking draft. Draft-Answer Consistency assesses whether the model’s final answer aligns with the logical conclusion of the draft, while Draft-Reliant evaluates whether the answer stage depends entirely on the thinking draft.

To further clarify these distinctions, we will expand our discussion of the motivations behind each dimension in the later revision.

Answer 3 (Weakness 3 & Question 1): Incorporating Internal State Analysis

Thank you for this insightful suggestion. To deepen our analysis, we now include experiments correlating internal model calculations (perplexities) with the observed selective integration phenomena of reasoning steps in Intra-Draft Faithfulness.

Specifically, we compute the conditioned perplexity difference given a question $x$, an original thinking draft without counterfactual insertion $(T_{-}, T_{+})$, and $T'\_{-}$, a variant of $T_{-}$ with counterfactual insertion:

$\Delta\text{PPL}(x, T\_{-}, T'\_{-}, T\_{+}) = \text{PPL}(T\_{+} | x, T'\_{-}) - \text{PPL}(T\_{+}|x, T\_{-})$

Where $\text{PPL}$ represents the conditioned perplexity calculation over question and thinking draft. All results are computed based on the conditioned perplexity over the first 100 tokens of $T_{+}$ to ensure a fair comparison. We interpret a higher $\Delta\text{PPL}$ as indicating greater integration into the reasoning process, since the original continuation becomes less likely under the perturbed context introduced by the intervention.

We run experiments on all test traces on MMLU and GPQA, and report the average $\Delta\text{PPL}$ for both continue and backtrack insertion types, comparing first/middle versus end insertion locations, using DS7B, OR1-7B, and Qwen3-14B.

These preliminary results show a significantly higher $\Delta\text{PPL}$ for end-step insertions compared to first/middle-step insertions, supporting our finding of location-dependent faithfulness in reasoning steps. Furthermore, the consistently higher $\Delta\text{PPL}$ for backtrack insertions corroborates our conclusion that backtracking steps are more faithfully integrated. We will incorporate a more detailed analysis of internal states and additional experiments into the revision to strengthen these findings.

Answer 4 (Question 2): Propose Methods for Improving Faithfulness

Thank you for appreciating our analysis! We agree that exploring concrete methods for improving faithfulness is critical. Based on our findings, promising future directions include:

• Carefully design RLVR rewards directly tied to our defined faithfulness metrics.

• Extending thinking interventions [4] to encourage sustained attention to reasoning steps, thereby enhancing intra-draft faithfulness.

However, as demonstrated by prior research [3], effectively enhancing model faithfulness is inherently challenging and involves substantial additional complexity beyond our current scope. Nonetheless, our work offers foundational analyses and measurement tools that we believe are essential for guiding future advancements in this area.

Answer 5 (Question 3): Strengthening the Evaluation Benchmark

Thank you for this valuable suggestion! To enhance our evaluation benchmark, we incorporated additional evaluations using the MMLU College Math subset (100 college math reasoning questions) and included new models (Qwen3-14B, OR1-7B, DS-1.5B).

MMLU College Math Results (%):

Additionally, we report the following results for GPQA and MMLU global facts subset for newly added tested models:

GPQA Results (%):

MMLU Global Facts Results (%):

These expanded evaluations consistently confirm our original observations. For instance, larger model in same model family general presents higher Intra-Draft Faithfulness, RLVR tuning incentive stronger internal preferences with low Draft-Answer Consistency, and math reasoning, which is more complex than factual recall over MMLU-global facts, result less faithful rate in Intra-Draft faithfulness and Draft-Answer Consistency. These observations of strengthened benchmarks reinforcing the reliability and generalizability of our conclusions. We will include more analysis over these experimental results within later revision.

Thank you again for your constructive comments, which have helped us further improve our manuscript.

[1] Chen et al. Reasoning Models Don't Always Say What They Think

[2] Lanham et al. Measuring Faithfulness in Chain-of-Thought Reasoning

[3] Tanneru et al. On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models

[4] Wu et al. Effectively Controlling Reasoning Models through Thinking Intervention

Thank you for your insightful and detailed feedback. To address your questions, we have added the following evaluations:

• An expanded benchmark that includes a mathematical reasoning dataset (MMLU College Mathematics) (Answer 2).
• An additional examination of LLM judge reliability involving human and cross-model annotation (Answer 3).
• A new evaluation comparing our approach with natural, unperturbed evaluation (Answer 6).

We address each of your concerns below:

Answer 1 (Weakness 1): Evaluation relies heavily on multiple-choice questions.

We acknowledge that multiple-choice questions may not fully capture the breadth of real-world reasoning tasks. However, this choice results from the intrinsic difficulty of faithfulness evaluation. Open-ended or free-form reasoning tasks introduce a high output space of final answers, making faithfulness analysis (especially via counterfactual interventions) infeasible at scale. Prior works similarly restrict evaluations to multiple-choice settings due to this constraint [1,2]. Furthermore, the complexity introduced within the thinking drafts of LRMs exacerbates this challenge, making multiple-choice settings a practical starting point. This design choice is also shared by recent concurrent work [1].

Answer 2 (Weakness 1): Evaluation is limited to GPQA (198 questions) and MMLU (88 questions).

We appreciate your concern about dataset diversity. While our evaluation indeed leverages two datasets, the overall scale of our analysis is broader. Specifically, our experiments cover 24 intervention variants for Intra-Draft Faithfulness and 8 variants for Draft-to-Answer Faithfulness, totaling over 9,000 tested traces.

To further broaden task coverage, we have now included evaluations on the MMLU College Math subset (100 college-level math questions). The results across existing models and additional models, including Qwen3-14B, Skywork-OR1-7B (OR1-7B), DeepSeek-R1-Distilled-1.5B (DS1.5B), are presented in Answer 5 of our rebuttal to reviewer hiiz.

Answer 3 (Weakness 2): Reliability of using LLMs for modification and judgment.

Thank you for this important concern. We conducted a detailed reliability analysis, which is discussed in Answer 1 of our rebuttal to reviewer hiiz. We include human annotations and Human-LLM agreement analysis to demonstrate the consistency and trustworthiness of our automated evaluation pipeline.

Answer 4 (Weakness 3): Prioritizing Draft-Answer consistency over factual correctness is problematic. The paper fails to analyze critical trade-offs between faithfulness and performance.

Thank you for raising this point. We respectfully disagree that our framing is problematic. As noted in Lines 164–165 of the manuscript, our primary goal is to assess whether model decisions are causally grounded in their thinking drafts. This property is critical for reliable monitoring and safety, aligning with the emphasis in recent works [1, 4, 5].

For testing purposes, we deliberately require consistency with incorrect traces, as this reveals whether models faithfully follow textual reasoning drafts under extreme conditions. However, in practice, faithfulness should be orthogonal to performance. An unfaithful model may select incorrect internal reasoning traces while ignoring correct final draft answers, making actual failures less tractable and hindering oversight capabilities. For instance, Skywork-OR1-32B-Preview yields higher performance on their reported reasoning benchmarks (including AIME 24, 25, and Livecodebench) compared with QwQ-32B, while still presenting better Draft-Answer consistency. Thus, we did not explore whether unfaithful behavior can improve accuracy, as we observed no such tendency, and it is outside the scope of this paper.

Answer 5 (Weakness 4): Concern about artificial perturbations disrupting reasoning flow.

We agree that counterfactual interventions may not fully reflect natural reasoning processes. However, since our goal is to analyze causal dependencies within the thinking draft, such interventions are necessary and inevitable, similar to prior work on CoT faithfulness [2] that also introduces artificial counterfactual changes.

Nevertheless, we have designed interventions carefully to minimize unnatural disruptions. For Intra-Draft Faithfulness, we:

• Identify natural reasoning shift points within the reasoning flow (Appendix B.1), ensuring insertions do not significantly interrupt the continuity of forward reasoning.
• Restrict perturbations primarily to the option level, preserving the reasoning about the question itself

For Draft-to-Answer Faithfulness, particularly in our "Plausible Alternation" condition, we generate fluent and coherent alternative conclusions using separate LLMs, thus realistically mimicking natural reasoning uncertainty.

While we understand this construction may not be perfect given the complexity of LRM reasoning processes, we adopt this design to make initial steps toward measuring these properties. Future work can indeed improve the naturalness of interventions.

Answer 6 (Question 3): Comparison with faithfulness assessments under natural, unperturbed conditions.

We appreciate this suggestion. To our knowledge, existing faithfulness evaluations of LRMs under natural, unperturbed conditions primarily focus on simulatability [1]. We discuss the differences between our work and these concurrent works in Lines 39–46 (Section 1). While both aim to assess LRM faithfulness, our goals differ. Their approach measures the likelihood of shifting decisions toward hint answers in prompts and verbalize the hint within the trace, but this does not assess whether reasoning steps are faithful to the draft answer or whether the final answer faithfully depends on the draft.

To clarify the difference between the simulatability-based evaluations and our proposed method, we provide additional results using the setup in [1]. As their evaluation data are not publicly available, we reconstruct their setup using four hint types. The final faithfulness scores for nine models are as follows:

Faithfulness Score(%) [1]:

We observe that simulatability-based faithfulness scores are less correlated with model size, and RLVR-tuned models achieve higher scores, consistent with findings in [1].

However, this contrasts with our Draft-Answer Consistency results, where RLVR tuning leads to stronger internal preferences and reduced reliance on the draft logic. An example of such unfaithful reasoning is shown in Figure 1 (Section 3).

We hypothesize that this discrepancy arises because simulatability primarily captures the verbalization aspect of reasoning, whereas our Draft-Answer Consistency metric offers a more fine-grained view of the draft’s logical dependency with the final answer.

Together, our findings complement simulatability evaluations and provide deeper insight into LRM reasoning.

Answer 7 (Question 1): The current definition of Draft Reliance $\mathbb{1}[M(x, T',G) = M(x, T', \emptyset)]$ seems counterintuitive.

We apologize for any confusion caused. Our definition tests whether the final answer $y$ is determined solely by the thinking draft $T'$. Here, $G$ refers to the additional explanation generated during the answer stage. If the answer remains unchanged regardless of $G$'s presence or absence, this indicates that the final decision relies exclusively on the draft itself, implying draft reliance. An illustrative example comparing conditions with and without $G$ is presented clearly in Table 2 (Section 4.3.2) of the manuscript.

Answer 8 (Question 2): Why evaluate small models on drafts from larger models? Why not evaluate Qwen3-32B and DeepSeek-R1 directly?

As clarified in Lines 65–70 (Section 1), we evaluate each model using both self-generated and externally generated drafts from Qwen3-32B and DeepSeek-R1. This cross-model setup ensures controlled and consistent comparisons, as relying solely on self-generated drafts can introduce biases—particularly when evaluating counterfactual interventions. Indeed, we observed that Draft-Answer consistency varies between self- and externally generated drafts, reinforcing the need for controlled baselines. This is one reason we excluded Qwen3-32B and DeepSeek-R1 from direct evaluation.

Another reason is the challenge of ensuring fair experimental control. We deliberately design controlled experiments to isolate different factors, such as differences in base model family, sizes, and post-training datasets and methods. While these larger models are powerful, the presence of multiple confounding variables would hinder fair and accurate comparisons, making it difficult to draw consistent insights. Therefore, we prioritized models with well-defined training conditions to enable rigorous and interpretable analyses.

We thank you again for your insightful comments and hope that our responses and additional results address your concerns.

[1] Chen et al. Reasoning Models Don't Always Say What They Think

[2] Lanham et al. Measuring Faithfulness in Chain-of-Thought Reasoning

[3] Turpin et al. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

[4] Korbak et al. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

[5] Baker et al. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation

[6] Wu et al. Effectively Controlling Reasoning Models through Thinking Intervention

Thank you for your insightful and valuable comments, as well as your appreciation of our work. In response, we have added an additional analysis of LLM judge reliability involving human and cross-model annotations (see Answer 1).

We address your specific concerns below:

Answer 1 (Weakness 1&2 & Question 1&3): The use of LLM-as-a-judge introduces a source of bias that is not quantified.

We appreciate your concern regarding the reliability of using LLMs as evaluators. To rigorously verify the reliability, we conducted additional experiments involving three human annotators and multiple judging models and measure their agreement rates (i.e., the percentage of matching annotations between two annotators) and Cohen’s Kappa coefficient:

• Step Labeling: 3 annotators evaluated 200 randomly sampled reasoning steps (100 self-reflect, 100 continue reasoning) from MMLU Deepseek-R1 traces.
• Behavior Classification and Trace Answer Extraction: 3 annotators evaluated 200 reasoning traces from Qwen3-14B (100 explicitly corrected, 100 consistently followed) across all intervention types on MMLU for Intra-Draft Faithfulness.

Human-Human Agreement Rates:

Human-LLM Agreement (using Majority Vote from human annotations):

We also calculate Cohen's Kappa coefficient between the majority vote of human labels and the LLM for both step labeling and behavior classification. Step labeling yields a coefficient of 0.83, indicating almost perfect agreement. Behavior classification, which is more challenging for humans due to the extended content in the reasoning trace, still yields a coefficient of 0.62, reflecting substantial agreement.

Furthermore, we assessed reliability across multiple LLM judges (Qwen2.5-32B-Instruct, Llama-3.2-70B-Instruct, Qwen3-32B) for behavior classification on three additional models: Deepseek-R1-Distill-1.5B (DS-1.5B), Skywork-OR1-7B (OR1-7B), and Qwen3-14B (faithfulness rate reported in Answer 5 to Reviewer hiiz) over all generated traces on both MMLU and GPQA:

Cross Model Agreement Rates:

These high agreement rates, comparable to human-level consistency, strongly support the reliability of our LLM-based evaluation method.

Answer 2 (Weakness 1 & Question 2): The designed interventions may not represent reasoning failures that occur in practice. Have you considered exploring perturbations that could arise during LRM reasoning?

Thank you for raising this insightful point! We acknowledge that our counterfactual interventions may not fully capture realistic reasoning failures occurring naturally in practice.

However, as discussed in Lines 138–142 (Section 3), exploring counterfactual effects within thinking drafts poses intrinsic challenges due to LRMs' complex multi-path exploratory nature. Our chosen perturbations—corrupting or shifting mappings of multiple-choice options—were specifically designed to enable global dependencies throughout the whole reasoning process, helping us to consistently evaluate faithfulness.

We acknowledge that modifying factual premises or logical connections would better reflect natural reasoning errors. However, generating such interventions at scale and precisely tracking their subtle counterfactual effects poses significant methodological hurdles. Furthermore, this approach is unlikely to establish global dependencies throughout the reasoning process. These considerations guided the current design of our intervention as an initial exploration of the thinking draft faithfulness problem. Nevertheless, we agree that this is a crucial direction for future research, and we will explicitly address these limitations and potential extensions in our revised manuscript.

Answer 3 (Weakness 1 & Question 4): The binary faithfulness metrics may miss nuances in LRM behavior. Have you considered measures of faithfulness with more gradations that could capture nuances in how models integrate or deviate from a line of reasoning?

Thank you for raising this thoughtful concern. Indeed, the inherent complexity of LRM behavior can present diverse forms of integration or deviation in response to our interventions. However, our primary research goal is to rigorously evaluate the faithfulness property, defined as whether the final reasoning or conclusion integrates explicit reasoning steps within the thinking draft, or deviates from the reasoning steps with clear correction.

We intentionally chose binary metrics to clearly delineate such core property. For example:

• If the model explicitly rejects the intervention (at any point), we classify this as CORRECTION, implying the reasoning draft’s global trajectory should remain unchanged.
• Conversely, if a model integrates the new mapping or corrupted option without explicitly rejecting it, we classify this as FOLLOW behavior. Under this scenario, faithfulness implies the final conclusion should change accordingly.

The reasons above lead the classified behaviors to exhibit signs of faithfulness in a straightforward and easily detectable manner. The detailed classification prompts are also provided in Appendix B.1.

Introducing additional nuance, such as degrees of integration or rejection, could potentially introduce ambiguity, making LLM or even human classification more subjective and less reliable, and may not affect too much understanding of faithfulness property alone.

While we agree that more fine-grained behavioral analyses could be insightful for broader behavioral understanding of models, they are less directly relevant to our current, focused investigation of faithfulness. Nonetheless, we acknowledge that this remains an interesting avenue for further exploration and will highlight this point as future work.

Answer 4 (Weakness 2 & Question 3): The function $\phi$ and ANS are underspecified

We apologize for the confusion. Since the definition of faithful FOLLOW (where $\phi$ is used) differs across the two intervention settings—Shift Mapping (with target projection) and Corrupt Option (with untargeted projection)—we opted to present a simplified and unified view in Section 3 to maintain readability. The detailed specification of how faithful FOLLOW is defined under each intervention is provided in Lines 207–209 of Section 4.2.1 in text.

For clarity, we also provide a formal definition of faithful FOLLOW for both settings, using a shared option set $\mathcal{Y} = \\{y_0 = A, y_1 = B, y_2 = C, y_3 = D\\}$:

• Shift Mapping. Let $\phi(y\_i) = y\_{(i+1)\ \bmod 4}, \\text{for } 0 \\le i \\le 3.$ In this case, the faithful following metric is $\\mathbb{1}\\bigl[\\text{ANS}(T'\_{>j+1}) = \\phi(\\text{ANS}(T))\\bigr].$
• Corrupt Option. Suppose the intervention corrupts option $y_i, 0 \le i \le 3$. The model is expected to not return the corrupted option, and the faithfulness metric is measured as $\mathbb{1}\bigl[\text{ANS}(T'_{>j+1}) \ne y_i\bigr].$

Additionally, ANS refers to a function that extracts the final answer from a reasoning trace: $\text{ANS}: \mathcal{V}^* \to \mathcal{Y},$ where $\mathcal{V}$ is the vocabulary space. This function is defined at Line 117 in Section 3. We will make these definitions more explicit and accessible in the revised version.

Answer 5 (Question 5): How do you handle cases where "correct" faithful behavior is undefined?

Thank you for this important question. By design, we specifically aim to minimize scenarios where faithful behavior is undefined or ambiguous. Our methodology achieves this by:

• Selecting interventions targeting multiple-choice options, thus ensuring that model responses to interventions are readily detectable and interpretable within thinking drafts.
• Implementing clear classification rules (as detailed in Answer 3).

In rare instances such as repetitive gibberish output, we explicitly exclude these from evaluation to maintain assessment reliability and interpretability. This careful filtering further ensures we only evaluate clearly defined scenarios, thus strengthening the validity of our measured faithfulness metrics.

Thank you again for your detailed and constructive feedback, which has substantially helped us clarify and improve our manuscript.

Thank you for your responses. I’ve kept my rating as accept and have no further questions.