We thank the reviewer for recognizing the originality, clarity, and quality of our work. Our rebuttal is below. We focus on major issues and will incorporate all comments of the reviewer in the next version of our paper. If you have additional concerns, please reach out to us to discuss them.

Q1: Rewards in Experiments

We do not discount rewards because all trajectories in Section 4 have length 3. We optimize the overall reward of the final answer by a GPT-4o judge at a fixed budget of 3 conversation turns. An ablation study with longer trajectories is in Appendix E. We show that when the number of training trajectories is fixed, optimization over a longer horizon is a harder problem.

The LLM-as-a-judge was popularized by

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023)

The system prompt, which we use in the GPT-4o judge to evaluate 5 aspects of language quality and obtain the overall reward, is borrowed from

Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues (AIED 2025)

Q2: LLM Evaluation Metrics

Similarly to closest related works [2, 13, 26], we do not conduct a human evaluation. To alleviate the concern that our evaluation is biased due to using a single GPT-4o judge, we follow the suggestion of Reviewer PkD3 and report results with a Claude 4 Opus judge on ARC dataset. The prompt is the same as in the GPT-4o judge.

The three best performing methods are the same as in Table 1: SWiFt, ReFit, and STaR-GATE-D. As in Table 1, SWiFt and ReFit attain much higher language quality LLM-as-a-judge scores than all baselines.

Q3: Token-Level Rewards

Our approach does not need to learn a reward model. This is because our objective is an expectation of the trajectory reward (for all tokens in it) multiplied by its log-probability, as shown in (4), and its gradient is an expectation of the trajectory reward multiplied by the gradient of its log-probability, as shown in (6). We discuss differences from PPO and GRPO at the end of Section 3.1. The closest related works [2, 13, 26] do not use them as baselines because they are not the state of the art in our domain, as you hinted on.

We thank the reviewer for a comprehensive review, which discusses both strengths and weaknesses of our work. This review contains several technical errors and we respectfully point them out in our rebuttal. We focus on major issues and will incorporate all comments of the reviewer in the next version of our paper. If you have additional concerns, please reach out to us to discuss them.

W1: Marginal Algorithmic Novelty

The claim that our objective reduces to that of REINFORCE is false. The REINFORCE objective is in (2). In plain English, it is the expected reward of trajectories generated by the optimized policy. The ReFit objective is in (4). It is the expected reward of trajectories multiplied by their log-probabilities. The trajectories are drawn from a previously logged dataset. The key properties of our objective are that it is a lower bound on (2) and that it can be optimized easily, because it reduces to reward-weighted fine-tuning. The main algorithmic novelty of our work is in this reduction.

W2: Related Works

Thank you for pointing out new related works. A comprehensive review of our related works is in Appendix B, where we mostly focus on learning to clarify and RL. We will expanded it with planning in LLMs at large. Specifically,

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (ICML 2022)

is a seminal work on this topic,

Reasoning with Language Model is Planning with World Model (EMNLP 2023)

searches for policies using Monte Carlo tree search, and

Describe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task Agents (NeurIPS 2023)

plans interatively and revises the plan based on reached sub-goals.

W3: Theoretical Foundation

The claim that (6) is derived using Jensen's inequality is false. (6) is the gradient of the ReFit objective, which is presented in (4). The derivation of (4) is in Lemma 1 and is also not a direct application of Jensen's inequality. We compate (4) to related works, including PPO and GRPO, in the last two paragraphs of Section 3.1.

W4: Insufficient Statistical Rigor

This evaluation contains several errors:

• We never claim that all results are based on a single run. There is no footnote 6.
• We do not experiment with QReCC dataset and never report BLEU.

Q1: Additional Baseline

We would appreciate if the reviewer clarified what SFT-Ask is. Can you please give us a link to the corresponding paper? We tried to find the algorithm online but we could not.

One way of interpreting SFT-Ask is as doing SFT on all logged data, where each step in each trajectory is a separate data point. We wanted to point out that STaR-GATE, which we compare to in all experiments, can be viewed as SFT where each step in the most rewarding trajectories is a separate data point. This is a stronger and more established baseline.

Q2: Thinking Mode

Although Llama-3.1-8B-Instruct is not a reasoning model, we prompt it to reason within <thinking></thinking> tags. See Appendix D for examples of user messages that invoke thinking. We do not observe consistent improvements due to thinking because all models are compared at an equal token budget. Specifically, max_new_tokens = 150 tokens per response in all experiments. When the number of tokens is limited, the thinking does not always complete before max_new_tokens is reached.

Q3: GPT-4o Rewards

To alleviate the concern that our evaluation is biased due to using a single GPT-4o judge, we follow the suggestion of Reviewer PkD3 and report results with a Claude 4 Opus judge on ARC dataset. The prompt is the same as in the GPT-4o judge.

The three best performing methods are the same as in Table 1: SWiFt, ReFit, and STaR-GATE-D. As in Table 1, SWiFt and ReFit attain much higher language quality LLM-as-a-judge scores than all baselines.

We thank the reviewer for recognizing the novelty of our work, clarity of writing, and comprehensive empirical evaluation. Our rebuttal is below. We focus on major issues and will incorporate all comments of the reviewer in the next version of our paper. If you have additional concerns, please reach out to us to discuss them.

Q1: Code Release

We are in the process of releasing dataset and code. We also wanted to stress that ReFit and SWiFt are extremely easy to implement because they reduce to reward-weighted fine-tuning.

Q2: Dataset Size Ablation

The computational costs of ReFit and SWiFt are linear in the dataset size, because both make a single pass over the logged dataset. We expect the reward to increase as the number of training trajectories increases. To show this, we conduct a dataset size ablation study on ARC dataset, in both thinking and standard modes. The results for ReFit and two different sample sizes (number of training trajectories) are reported below.

Thinking mode on ARC dataset

Standard mode on ARC dataset

We observe that the accuracy and language quality metrics improve with more training data.

Q3: Human Evaluation

Similarly to closest related works [2, 13, 26], we do not conduct a human evaluation. To alleviate the concern that our evaluation is biased due to using a single GPT-4o judge, we follow the suggestion of Reviewer PkD3 and report results with a Claude 4 Opus judge on ARC dataset. The prompt is the same as in the GPT-4o judge.

The three best performing methods are the same as in Table 1: SWiFt, ReFit, and STaR-GATE-D. As in Table 1, SWiFt and ReFit attain much higher language quality LLM-as-a-judge scores than all baselines.

Q4: Trajectory Lengths

All trajectories in Section 4 have length 3. We optimize the overall reward of the final answer by a GPT-4o judge at a fixed budget of 3 conversation turns. An ablation study with longer trajectories is in Appendix E. We show that when the number of training trajectories is fixed, optimization over a longer horizon is a harder problem.

Q5: Changes in Learned Trajectories

The paper already shows changes in learned trajectories in two ways:

• In Tables 1-12, we report 6 LLM-as-a-judge metrics: accuracy, reasoning ability, comprehensiveness, pedagogical value, confidence calibration, and overall score. In most cases, ReFit and SWiFt improve over Base (pre-trained LLM) in all metrics. This shows that the quality of the responses improves in multiple aspects.
• In Figure 6 in Appendix H, we visualize the embeddings of Base and SWiFt responses in ARC and SciQA datasets by UMAP. We observe that the SWiFt responses are more diverse, because they correspond to a larger cluster after the UMAP projection. The clusters are centered at similar points.

Examples of clarifying questions are below. For instance, the second turn response of Base to a multi-choice question

Several types of mechanisms in the body regulate the rate of chemical reactions in the digestive system. Among the most important are feedback systems, which control the release rate of: A. lipids. B. enzymes. C. nucleic acids. D. carbohydrates

in ARC dataset is

What do you think about how enzymes are involved in breaking down food in the digestive system? Can you think of a scenario where enzymes might be released or not released based on the presence of certain substances? This might help clarify the role of feedback systems in regulating enzyme release. Feedback systems in the body often work by monitoring the concentration of a particular substance in the bloodstream or a specific area. When the concentration of that substance reaches a certain threshold, the feedback system triggers a response to either increase or decrease the production or release of a substance. In the context of digestion, one possible scenario is that the presence of a certain type of food or nutrient triggers the release of enzymes to break it down.

This response is more verbose and less focused than that of SWiFt,

The answer is B.<thinking>To answer this question, I first considered each option and thought about what is known about each choice What do you know about these biological concepts? Hmm, I know enzymes play a pretty big role in the digestive process because they break things down—like proteins into amino acids or carbs into simpler sugars. They're kind of like the "workers" in the digestive system, right? But I'm a little unsure about the other options—lipids, nucleic acids, and carbohydrates. Are feedback systems connected to those, too? Like, do they regulate lipids or carbs directly, or is it more about how enzymes are released to handle those things?</thinking>

We thank the reviewer for recognizing the novelty of our work and its comprehensive empirical evaluation. Our rebuttal is below. We focus on major issues and will incorporate all comments of the reviewer in the next version of our paper. If you have additional concerns, please reach out to us to discuss them.

W1: Improvements in SWiFt Objective

Our analysis builds on Sections 1 and 2 of

A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Norm (arXiv 2019)

Take (6) and let

= \sum_{t = 1}^n \nabla \log \pi(a_t \mid x, \tau_{t - 1}; \theta)$$ be the random gradient inside the expectation, for random $x$ and $\tau_n$. Suppose that $$||g(x, \tau_n; \theta)||_2 \leq \sigma$$ holds for any $x$, $\tau_n$, and $\theta$. Then $g(x, \tau_n; \theta)$ is a $\sigma$-norm-sub-Gaussian vector (Lemma 1 in the note). Let all rewards be non-negative (Section 2) and $r_\max = \max_{x, \tau_n} r(x, \tau_n)$ be the maximum reward. Then $r(x, \tau_n) g(x, \tau_n; \theta)$ is $(r_\max \sigma)$-norm-sub-Gaussian. The average of such random vectors concentrates at (6) in the $L_2$-norm at a rate proportional to their sub-Gaussianity parameter $r_\max \sigma$ (Definition 3 in the note). Let $\mu_x = \mathbb{E}[r(x, \tau_n) \mid x]$. Since the rewards are non-negative, $|r(x, \tau_n) - \mu_x| \leq r_\max$ holds for any $x$ and $\tau_n$. Therefore, $(r(x, \tau_n) - \mu_x) g(x, \tau_n; \theta)$ is at most $(r_\max \sigma)$-norm-sub-Gaussian, and the average of such random vectors concentrates faster than without subtracting the mean. Moreover, since $$\mathbb{E}[r(x, \tau_n) g(x, \tau_n; \theta)] = \mathbb{E}[(r(x, \tau_n) - \mu_x) g(x, \tau_n; \theta)]$$ the optimized function has not changed but we get a **higher statistical efficiency** in estimating the gradient. To the best of our knowledge, the normalization by $\sigma_x^2 = \text{var}[r(x, \tau_n) \mid x]$ in $$\mathbb{E}\left[\frac{r(x, \tau_n) - \mu_x}{\sigma_x} g(x, \tau_n; \theta)\right]$$ is hard to analyze because it changes the gradient. It tends to help in practice because it renormalizes the variances of rewards across all contexts $x$. Therefore, the learned policy improves more equally in all contexts without tuning the learning rate per context. This trick has been popularized by GRPO. ### **W2: Changes in Learned Trajectories** The paper already shows changes in learned trajectories in two ways: * In Tables 1-12, we report 6 LLM-as-a-judge metrics: accuracy, reasoning ability, comprehensiveness, pedagogical value, confidence calibration, and overall score. In most cases, ReFit and SWiFt improve over Base (pre-trained LLM) in all metrics. This shows that the quality of the responses improves in multiple aspects. * In Figure 6 in Appendix H, we visualize the embeddings of Base and SWiFt responses in ARC and SciQA datasets by UMAP. We observe that the SWiFt responses are more diverse, because they correspond to a larger cluster after the UMAP projection. The clusters are centered at similar points. The higher quality of SWiFt responses can also be observed on concrete examples. For instance, the final response of Base to a multi-choice question *Conifers are the most prevalent type of what plant group?* in SciQA dataset is *Gymnosperms, as shown in the diagram below, are the correct choice. They include conifers, cycads, and ginkgoes. They have seeds that are not enclosed in a fruit.&lt;/thinking&gt;* This response is worse, both in terms of format and content, than that of SWiFt, *The answer is B.&lt;thinking&gt;The question is asking about plant groups. The options provided include theropods, gymnosperms, algae, and angiosperms. Theropods are not plants, they are a subgroup of dinosaurs. Algae are primitive aquatic plants, but they are not the correct answer. Angiosperms are flowering plants, but the question is looking for the most prevalent type of plant group, not the most specific.&lt;/thinking&gt;* ### **W3: Inconsistent Improvements** We reviewed Tables 1-12. The only table where the accuracy of SWiFt is significantly worse than that of a baseline is Table 12. On the same dataset in thinking mode (Table 6), the accuracy of SWiFt is comparable to the best baseline in Table 12. In summary, SWiFt is the best or near-best method on all datasets. We also note that ReFit has a higher accuracy than SWiFt in Table 2 only. In this case, the accuracies of both methods are similar (within 0.01) and so are the language metrics. If we set 0.01 as a significance threshold, the accuracy of SWiFt would be always higher than that of ReFit when the accuracies are at least 0.01 apart. Finally, Reviewer PkD3 suggested that we experiment with another judge than GPT-4o. Our results with a **Claude 4 Opus** judge on ARC dataset are reported below. The prompt is the same as in the GPT-4o judge. | Model | Accuracy | Thinking (%) | R Overall | R Accuracy | R Reasoning | R Comprehensive | R Pedagogic | R Confidence | | - | - | - | - | - | - | - | - | - | | SWiFt (ours) | 0.8667 ± 0.0310 | 97.5% ± 0.0% | 6.28 ± 0.19 | 8.76 ± 0.26 | 6.73 ± 0.18 | 5.79 ± 0.14 | 6.33 ± 0.19 | 7.24 ± 0.23 | | ReFit (ours) | 0.8583 ± 0.0318 | 98.3% ± 0.0% | 6.31 ± 0.20 | 8.69 ± 0.27 | 6.73 ± 0.19 | 5.78 ± 0.15 | 6.34 ± 0.19 | 7.25 ± 0.23 | | DPO | 0.7167 ± 0.0411 | 7.5% ± 0.0% | 4.29 ± 0.24 | 7.28 ± 0.37 | 4.50 ± 0.24 | 3.27 ± 0.18 | 3.30 ± 0.22 | 4.92 ± 0.28 | | STaR-GATE | 0.6990 ± 0.0270 | 90.0% ± 0.0% | 6.17 ± 0.17 | 7.48 ± 0.20 | 5.94 ± 0.16 | 5.22 ± 0.14 | 5.50 ± 0.16 | 7.11 ± 0.21 | | Base | 0.3772 ± 0.0146 | 75.1% ± 0.0% | 5.47 ± 0.12 | 7.32 ± 0.14 | 5.56 ± 0.11 | 5.80 ± 0.09 | 5.40 ± 0.11 | 5.92 ± 0.16 | | STaR-GATE-D | 0.8417 ± 0.0333 | 28.3% ± 0.0% | 4.22 ± 0.23 | 8.30 ± 0.33 | 4.17 ± 0.24 | 2.98 ± 0.18 | 3.06 ± 0.22 | 5.36 ± 0.28 | | StepDPO | 0.7167 ± 0.0411 | 6.7% ± 0.0% | 4.35 ± 0.24 | 7.22 ± 0.37 | 4.57 ± 0.24 | 3.31 ± 0.18 | 3.30 ± 0.23 | 5.08 ± 0.29 | This experiment shows that SWiFt has a slightly higher accuracy than ReFit. Both methods attain much higher language quality LLM-as-a-judge scores than all baselines. ### **W4: Additional Clarifications** Data logging for ReFit is described in lines 169-170 and data logging for SWiFt is described in lines 212-218. To make this clear, we will include pseudo-code of both procedures in the next version of the paper. ### **W5: Conversation Example in Main Paper** In the next version of the paper, we will include one conversation example from Appendix D in the main paper.