Thanks for reviewing our paper!

Weaknesses

Q: DPSDP does not resemble an RL algorithm

We adapt our algorithm from PSDP, a classic reinforcement learning method, and formulate iterative refinement as a standard MDP (Section 2). As shown in Algorithm 1, we optimize each step in reverse via policy rollout and update, following the PSDP framework.

A key insight is our estimation of ground-truth Q-values, which removes inter-iteration dependencies and simplifies implementation. Experiments validate this approach, and we compare the theoretical and practical versions in Appendix E.5.

Q: Other SOTA models baselines

While many works aim to improve LLM responses, few share our settings. For instance, SCoRe targets one-shot self-improvement without external feedback. The closest is RISE with an oracle, which assumes access to ground-truth correctness at test-time.

We include this variant as a baseline, re-implementing it by first training an SFTed Ministral-8B-It model (Section 3.4), followed by reinforcement learning on the same problem set used for our main models.

Our results show that our models achieve maj@t5 accuracies comparable to RISE on challenging benchmarks such as MATH and Olympiad. Our models consistently outperform RISE on pass@t5, indicating that the actor—guided by critic feedback—explores the solution space more actively rather than sticking to initial responses.

Q: Unfair comparison with SFT

It is worthnoting that the rows labeled +SFT in Table 1 are not SFT baselines but intermediate results from preliminary training (Section 3.4), which our DPSDP models build upon. For a fair comparison with standard SFT, we evaluated against STaR baselines (Section 4.2), which used the same SFTed base models, problem set, number of trajectories, and training epochs. Results show that STaR fails to enable self-improvement, underscoring the effectiveness of our approach.

Q: Comparison with self-critique

To highlight the benefits of the multi-agent setup, we replicated the full training—preliminary training and DPSDP—using a single model as both actor and critic. As shown in Table 1 (Single-Agent row) and discussed in Section 4.3, this setup consistently underperforms the multi-agent system, especially on harder benchmarks. We confirmed this across LLaMA-based models (see reply to Reviewer jmss, Table 1), and our conclusion holds.

Q: Concerns with Q-value estimation with Monte Carlo

Q-values reflect expected cumulative rewards rather than the immediate correctness of individual actions like feedback $a_1$. The correctness of $a_2 \sim \pi(\cdot \mid s_2)$ provides an unbiased estimate of the Q-value. A related example is DeepSeek-R1-Zero, which shows reduced readability during reasoning but achieves strong final performance.

Q: The soundness of using DPO as optimization method

Our core contribution is introducing PSDP for multi-agent LLM refinement. While we use DPO for optimization, it’s a modular component that can be replaced (e.g., with rejection sampling or KTO) without altering the overall approach.

Questions For Authors

Q: Fair comparison with SFT baseline

Please see the response above.

Q: Single-Agent experiment with Llama-based models

We present single-agent results using Llama-based in reply to Reviewer jmss, Table 1, and our findings consistently demonstrate the general superiority of the multi-agent system over the single-agent setup.

Q: Necessity of Preliminary Training (SFT)

See the reply to Reviewer Hay1, Table 2.

Q: Additional analytical experiments to explore impact of multi-round critiques

1. We visualize the accuracy dynamics in Figure 3 and provide results for more refinement steps in reply to Reviewer jmss, Table 3. Further detailed metrics are also presented in reply to Reviewer 2FHw, Table 1.

2. In reply to Reviewer 2FHw, we identify several failure patterns and illustrate how iterative refinement helps mitigate issues related to over-refinement.

3. In Appendix E.1, we compare maj1@t5 and maj5@t1, showing that the performance gains stem from the refinement process itself rather than from a simple increase in test-time computation.

Q: Data processing

Our data filtering is intentionally simple and transparent to attribute performance gains to our algorithm rather than data curation. We remove duplicates and randomly sample problems for unbiased coverage. For the SFT dataset, we use an oracle model to generate refined answers and filter feedback (Section 3.4). To ensure diverse reasoning styles, we include trajectories from different model famalies as explained in Appendix D.1.

Thanks for reviewing our paper!

Claims And Evidence

Q: Analysis is not highly related to the practical algorithm

We provide further analysis on how approximation in the practical algorithm affects the theoretical results in reply to Reviewer Hay1.

Q: More detailed metrics and failure pattern analysis

We adopt the metrics p1@t1, m1@t5, and p1@t5 in line with prior work [1], and provide additional evaluation details for a more comprehensive analysis.

First, we scale up the number of test-time refinement iterations, as shown in reply to Reviewer jmss, Table 3.

Next, we analyze the dynamics of accuracy over the course of refinement. Using models based on Ministral-8B-Instruct as a representative example, we plot the changes in accuracy across iterations in Figure 3. For each refinement step, we define $\Delta^{c \rightarrow i}$ as the proportion of problems that change from correct to incorrect after refinement, and $\Delta^{i \rightarrow c}$ as the proportion that transition from incorrect to correct.

The table illustrates two key observations:

1. $\Delta^{i \rightarrow c}$ consistently exceeds $\Delta^{c \rightarrow i}$, indicating that the refinement process is generally beneficial.

2. Both $\Delta^{i \rightarrow c}$ and $\Delta^{c \rightarrow i}$ decrease as the number of iterations increases, suggesting an initial exploratory phase followed by stabilization in later refinement steps.

We conducted the same analysis on models based on Llama and Qwen and observed similar patterns. Due to space constraints, we omit those results here.

In addition to the qualitative analysis presented in Appendix E.6, we identified several notable failure patterns:

• Answer enumeration: The critic repeatedly provides negative feedback, prompting the actor to cycle through different answers at each turn—effectively enumerating possible solutions.

• Answer degradation: The critic incorrectly assigns negative feedback, leading the actor to progressively degrade a previously correct answer. However, this over-refinement issue is relatively rare (as evidenced by small $\Delta^{c \rightarrow i}$) and can be mitigated by later refinement steps.

• Incorrect feedback tolerance: Occasionally, a correct answer is incorrectly revised due to faulty feedback. Yet, subsequent iterations can recover the correct answer and lead to correct final answer by majority voting across all turns, helping to mitigate the effects of over-refinement.

Experimental Designs Or Analyses

Q: diverse base models

We conducted DPSDP on Qwen2.5-3B and the results are presented in reply to Reviewer Hay1, Table 1.

Q: Investigate whether models can self-improve

While earlier work suggested that models are unable to self-improve, recent studies—such as SCoRe and RISE—have demonstrated that large language models (LLMs) can develop self-improvement capabilities when properly trained. To further explore this, we conducted an ablation study in which a single model served as both the actor and critic. The results are shown in Table 1 under the row labeled Single-Agent. A detailed comparison between the single-agent and multi-agent setups is provided in Section 4.3, under the paragraph titled Single-Agent vs. Multi-Agent.

Our findings are consistent with those of SCoRe and RISE: a single model can indeed self-improve. However, its performance is generally weaker than that of the multi-agent system, particularly on more challenging benchmarks. We replicated the single-agent setup using Llama-based models and presented the results in reply to Reviewer jmss, Table 1. Our conclusion remains the same.

Q: SFT baseline

One of our baselines, STaR, serves as the SFT counterpart to our algorithm. To ensure a fair comparison, STaR was implemented using the same SFTed models and trained on the identical prompt set. However, as discussed in Section 4.2, STaR fails to enable effective self-improvement in models.

Questions For Authors

Q: Difference between Markovian and non-Markovian

In Section 3.3, under the paragraph titled Optimizing Iterative Refinement with Reduced Complexity, we define the transition function $\delta(s_h, a_h)$, which reflects a Markovian setting. This design includes only the most recent answer and feedback in the prompt, removing all prior conversational history. The motivation behind this choice is the heuristic that recent context is more informative and relevant than earlier interactions.

In contrast, an alternative approach includes the entire conversation history—i.e., all previous responses and feedback—in the prompt. This setting diverges from the Markov Decision Process (MDP) we defined and is therefore referred to as non-Markovian.

[1] Recursive Introspection: Teaching Language Model Agents How to Self-Improve

Thanks for the efforts in reviewing our paper! We will take your suggestions, fix the typo and revise the presentations accordingly in the next revision!

Methods And Evaluation Criteria

Q: unclear how $a_1$ and $a_2$ are labeled as chosen and rejected actions

Algorithm 1 presents the theoretical version of our method and does not explicitly label $a_1$ and $a_2$ as chosen or rejected. Instead, it assumes access to the Q-value function, and the Q-values of $a_1$ and $a_2$ are directly used in the cross-entropy loss (see line 176, left column).

In the practical implementation (Algorithm 2), we estimate the Q-values as described in Section 3.3, under Estimation of Q-values. Specifically, we approximate the Q-values based on the correctness of responses. For each action pair $(a_1, a_2)$, the action with the higher estimated Q-value is labeled as "chosen," and the other as "rejected." We justify the reliability of this estimation both intuitively (Section 3.3) and empirically (Section 4 and Appendix E.5).

Q: Notation issues -- definition of $d_h^{\pi}$

We adopt standard notation from the reinforcement learning literature, where $d_h^{\pi}$ denotes the distribution over the state space at step $h$ when following policy $\pi$ (see lines 97–98, right column). Specifically, when $h = 0$, $d_0^{\pi}$ represents the initial state distribution—i.e., the distribution over prompts drawn from the prompt set or provided by users. When $\pi = \pi_\mathsf{ref}$, $d_h^{\pi_\mathsf{ref}}$ refers to the state distribution at step $h$ under the reference policy $\pi_\mathsf{ref}$.

Experimental Designs Or Analyses

Q: Baseline implementations with Llama-based models

To further validate the effectiveness of our algorithm, we benchmarked it against STaR and STaR-DPO using two additional base models: Llama 3.1-8B-Instruct and Qwen 2.5-3B. We also replicated the single-agent setup using Llama-based models as discussed in Section 4.3. Across both settings, our algorithm consistently outperforms the baselines, demonstrating robust and superior performance.

• Llama

  Task	Model	pass@t1	maj@t5	pass@t5
  MATH	DPSDP(Llama)	55.8	58.4	62.0
  	STaR	50.8	52.2	56.8
  	STaR-DPO	54.2	55.6	59.2
  	Single-Agent	53.4	54.8	58.0
  GSM8K	DPSDP(Llama)	87.5	88.4	91.2
  	STaR	83.6	81.3	87.5
  	STaR-DPO	87.5	87.4	90.3
  	Single-Agent	87.9	87.6	90.4
  MMLU-ProMath	DPSDP(Llama)	56.6	58.0	62.1
  	STaR	53.8	54.6	58.5
  	STaR-DPO	54.8	55.0	60.3
  	Single-Agent	56.1	57.3	62.0
  OlympiadBench	DPSDP(Llama)	22.4	23.0	25.1
  	STaR	20.5	20.3	22.4
  	STaR-DPO	20.9	21.5	24.3
  	Single-Agent	23.0	21.5	25.1

• Qwen

  Task	Model	pass@t1	maj@t5	pass@t5
  MATH	DPSDP(Qwen)	60.4	62.0	65.2
  	STaR	59.0	59.6	64.8
  	STaR-DPO	60.4	60.2	64.8
  GSM8K	DPSDP(Qwen)	79.9	79.9	84.2
  	STaR	80.3	79.5	83.7
  	STaR-DPO	79.4	78.9	82.6
  MMLU-Pro Math	DPSDP(Qwen)	52.6	53.2	57.1
  	STaR	51.9	51.8	56.8
  	STaR-DPO	51.2	52.3	55.9
  OlympiadBench	DPSDP(Qwen)	24.0	24.0	26.0
  	STaR	23.3	22.6	24.8
  	STaR-DPO	23.1	22.8	28.9

Questions For Authors

Q: Stopping Criterion and Overrefinement

We did not implement a specific stopping criterion for the refinement process, aside from a fixed limit on the number of refinement iterations, set to 5 in our experiments. To study the potential issue of over-refinement, we extended the number of iterations to 11. Our results show that accuracy generally improves with more refinement steps—reflecting the benefit of increased test-time computation—until it plateaus around 5 to 7 iterations. Beyond that, performance remains stable and shows minimal degradation, suggesting over-refinement is not a significant concern in practice.

We further conducted analysis on failure patterns in reply to Reviewer 2FHw, Table 1. In qualitative analysis beyond what is reported in the paper, we observed that iterative refinement can help correct over-refined answers. For instance, a correct answer may be incorrectly altered due to faulty feedback, but later iterations may recover the correct answer, leading to correct majority voting answer.

While iterative refinement helps mitigate over-refinement, it does not entirely eliminate the risk. As a potential solution, we propose monitoring performance on a validation set at each refinement step and stopping early if accuracy begins to decline.

Thanks for reviewing our paper!

Theoretical Claims

Q: Analysis of practical algorithm

We analyze the Q-value approximation in practical DPSDP, where only one feedback and refinement step is used during training (Algorithm 2), assuming $H=3$. Let $\hat{\pi}$ be the resulting policy, and let $\widetilde{Q}_h^{\hat{\pi}}$ denote the estimated Q-values, replacing $Q_h^{\hat{\pi}}$ in Assumption 2.

We define advantage function as $A_h^\pi(s_h,a_h)=Q_h^{\pi}(s_h,a_h)-V_h^{\pi}(s_h)$, and $\widetilde{A}_h^\pi(s_h,a_h)=\widetilde{Q}_h^{\pi}(s_h,a_h)-\mathbb{E}\_{a_h\sim \pi(\cdot\mid s_h)}[\widetilde{Q}_h^{\pi}(s_h,a_h)]$.

As detailed in Section 3.3 (Estimation of Q-values), we define the estimated Q-values as follows:

1. At $h=2$, the estimated Q is exact.

2. At $h=1$, the estimated Q is: $\mathbb{E}\_{a_2 \sim \pi_\mathsf{ref}(\cdot \mid s_2)}[r(s_3)] = Q_1^{\pi_\mathsf{ref}}(s_1, a_1)$.

   We define the approximation error:

   $$\Delta = \mathbb{E}_{s_h \sim d_h^{\pi^\star},a_h \sim \pi^\star(\cdot \mid s_h)}[A_h^{\hat{\pi}}(s_h,a_h) - \widetilde{A}_h^{\hat{\pi}}(s_h,a_h)]$$

3. At $h=0$, we have $\widetilde{Q}_0^{\hat{\pi}_1}(s_0, a_0) = r(s_1) + \frac{H-1}{2} = Q_0^{\pi^\star}(s_0, a_0)$. Therefore,

   $$\mathbb{E}\_{a_h \sim \pi^\star(\cdot \mid s_h)}[A_h^{\hat{\pi}}(s_h, a_h)] \approx \mathbb{E}\_{a_h \sim \pi^\star(\cdot \mid s_h)}[A_h^{\pi^\star}(s_h, a_h)] = 0,$$

   where the last equality follows from the definition of $A_h^{\pi}$.

Following steps in Appendix C.1, we obtain the approximate upper bound by adding $|\Delta|$ to the theoretical bound.

To assess the impact of $|\Delta|$, we performed an ablation using the step-by-step DPSDP variant (Appendix E.5), which uses $Q^{\pi_2}$ in the DPO-style loss. The results showed no significant performance gain, indicating that $|\Delta|$ has minimal effect. For simplicity and efficiency, we use the original version in the main paper.

Weaknesses

Q: Other model sizes and families

We further tested our algorithm on Qwen2.5-3B to demonstrate its effectiveness over different model sizes and model families.

Our results show that the proposed algorithm generalizes effectively on smaller models such as Qwen2.5-3B. Furthermore, DPSDP-trained models demonstrate strong generalization capabilities on out-of-distribution benchmarks, such as MMLU Pro Math.

Q: Other reasoning tasks

While our focus has been on mathematical reasoning to showcase the effectiveness of our approach, we anticipate that similar performance gains would extend to other complex reasoning tasks. Exploring these tasks presents an exciting direction for future research.

Q: SCoRe and RISE baselines

See reply to Reviewer X1o306, Table 1.

Questions For Authors

Q: Define the optimal number of refinement iterations

We scaled the number of refinement iterations up to 10. Our results show that accuracy begins to plateau after approximately 5 to 7 iterations as presented in reply to Reviewer jmss, Table 3.

Q: Comparison on training and inference cost with baselines

Both DPSDP and baselines have similar time complexity, dominated by quadratic-scaling attention mechanism during forward and backward passes. On 4× H100 80GB GPUs, DPSDP and STaR-DPO each took ~6 hours to train (5h 45m and 5h 55m, respectively), while STaR completed in 3h 10m. Inference costs are comparable, with responses all refined through 4 iterations.

Q: How essential is the preliminary training (SFT) stage for DPSDP's performance?

Prior work [1,2] has shown that SFT is essential before reinforcement learning RL, especially when the base model struggles to follow instructions.

Our results support this: we trained DPSDP on Mistral-8B-instruct both with and without SFT. Without SFT, performance degrades notably on MATH and OlympiadBench, and iterative refinement offers little benefit—as seen in the small gap between accuracy@t1 and majority@t5 or pass@t5.

These findings highlight the importance of SFT in enabling models to give and use feedback effectively.

[1] Recursive Introspection: Teaching Language Model Agents How to Self-Improve

[2] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training