ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification

Dear reviewer nE5C,

We sincerely appreciate your efforts and comments to improve the manuscript. We respond to your comment in what follows.

---

[Q1] Would the proposed method benefit from multi-round self-correction?

We have already investigated multi-round self-correction in Appendix B.1 of our paper. Specifically, after generating the first refined response ($y_2$), we generate a second refined response ($y_3$) by appending ($y_2$) to the original prompt ($x$). Formally, $y_2 \sim p(\cdot \mid x, y_1), y_3 \sim p(\cdot \mid x, y_2)$.

As shown in Figure 8, ReVISE improves with each refinement round in MATH-500, e.g., 1st: 33.0 %, 2nd: 33.6 %, 3rd round: 34.2 %, demonstrating the effectiveness of multi-round self-correction. We will move the multi-round results to the main part of the final version.

---

[Q2] Compare with SCoRe method.

We first note that our primary focus was to build an efficient way to implement self-correction, where SCoRe requires a heavy online RL; it needs to generate the reasoning path for every training mini-batch, generating 1.5M paths in total (512 batch size × 3,000 steps). In contrast, ReVISE only requires a single reasoning path generation per individual sample to construct a preference pair, resulting in 50k generations. Hence, we expect that our training cost is 30 (=1.5M / 50k) times smaller than SCoRe.

Nevertheless, we agree that SCoRe is a worthwhile method to compare, and compared with ReVISE on Gemma2 2B. Here, we have reported SCoRe’s results from the paper (as it is not open-sourced and difficult to reproduce with our computing resources). As shown in the table below, ReVISE outperforms SCoRe on MATH using the same Gemma 2 2B model. We thank the reviewer for the suggestion and will include the comparison in the final version.

Table: https://bit.ly/4cdMnqN

---

[Q3] Could the authors provide additional evidence to support the claim that the proposed confidence serves as a reliable metric for calibrating the sampling score?

We remark that we have already reported the quantified ReVISE’s calibration performance using AUROC in Section 4.3 and Figure 3 (b). To further support this claim, we compared ReVISE with V-Star’s verifier [1]. Notably, while ReVISE relies on an intrinsic verifier, it still outperforms V-Star’s separately trained verifier, as shown in the table below, with a higher AUROC reflecting superior confidence-correctness alignment.

Table: https://bit.ly/4hW5HtE

[1] Hosseini, Arian, et al. "V-star: Training verifiers for self-taught reasoners." arXiv preprint arXiv:2402.06457 (2024).

---

Whether the results in Figure 3a demonstrate that the self-correction task is particularly challenging and may lead to catastrophic forgetting.

We agree that our original phrasing may have been misleading. We intended to convey that self-correction introduces additional difficulty, as the model must both verify and refine its output—potentially making it harder to maintain verification performance during training. We will revise the sentence to avoid strong terms like “catastrophic forgetting” and clarify the trade-off between verification and correction in joint training.

---

Benchmark on coding domain.

Due to space limitations, we kindly refer the reviewer to our response to cTTG’s comment W2, which addresses this point in detail.

---

ReVISE to instruction-tuned models.

We remark that Table 2 already reports results on instruction-tuned models, where ReVISE consistently outperforms other baselines. This shows that ReVISE complements instruction tuning and can further boost strong models.

---

Sampling for N=64

We extended our sampling setup on N=2 to 64. As shown in the table, ReVISE consistently outperforms baselines across all sample sizes. We will include these results in the final version.

Table: https://bit.ly/4hSZ23t

---

Running experiments multiple times and reporting the mean and variance.

As shown in the table above, we report the mean, standard deviation over 5 random seeds for all sample sizes N=2 to 64. ReVISE not only consistently outperforms baseline methods in terms of mean accuracy, but also maintains stable variance. We will report the mean variance across multiple runs in the final version.

---

Evaluate other kinds of model

To address your concern, we conducted experiments on the Gemma2 2B. As shown in the table below, ReVISE still consistently outperforms other baselines, including SFT, RFT, and SCoRe, showing its effectiveness beyond the Llama family. We will include this result in the final version of the paper.

Table: https://bit.ly/4cdMnqN

Dear reviewer Aquu,

We sincerely appreciate your efforts and comments to improve the manuscript. We respond to your comment in what follows.

---

[W1] The datasets employed and experimental designs are not sufficient; the paper proposes to make demonstrations in mathematics and coding domains in lines 81-82, but the actual experiments are only conducted in mathematics.

In response to the concern about limited coverage of task domains, we conducted additional experiments in the coding domain using the MBPP benchmark, as initially proposed in lines 81–82. ReVISE, trained on the Llama 3.2-1B model, achieves 33.1%, outperforming the second-best baseline at 30.7%. Since MBPP lacks reasoning annotations, we generated ground truth reasoning using GPT-4o. These results complement our mathematics experiments and demonstrate that ReVISE generalizes effectively beyond mathematics tasks.

\begin{array}{lc} \hline Method & Pass@1 \newline \hline \text{3 Shots} & 24.5 \newline \text{SFT} & 30.0 \newline \text{RFT} & 29.6 \newline \text{STaR}^+ & 30.7 \newline \text{ReVISE (Ours) } & 33.1 \newline \hline \end{array}

---

[W2] The lack of OOD evaluations.

To address your concern and demonstrate robustness to domain shifts, we conducted an OOD evaluation by training ReVISE on the MATH dataset and evaluating it on GSM8K. As shown in the table below (Maj@1), ReVISE consistently outperforms other baseline methods (the gap decreases a bit though), including SFT, RFT, and StaR+, under the OOD setting. We will add the OOD results to the final draft.

\begin{array}{lcc} \hline Model & Method & GSM8K Acc. \newline \hline \text{} & \text{SFT} & 7.3 \newline \text{Llama 3.2-1B} & \text{RFT} & 8.2 \newline \text{} & \text{STaR}^+ & 8.0 \newline \text{} & \text{ReVISE (Ours) } & 8.8 \newline \hline \text{} & \text{SFT} & 60.3 \newline \text{Llama 3.1-8B} & \text{RFT} & 60.3 \newline \text{} & \text{STaR}^+ & 58.7 \newline \text{} & \text{ReVISE (Ours) } & 61.5 \newline \hline \end{array}

---

[W3] The concerns of the paper on open source.

In the final draft, we will release the training and evaluation code for ReVISE, including all hyperparameter configurations, to ensure full reproducibility of the results presented in the paper.

---

[W4] Concern about temperature consistency in the experiment.

We clarify that we have used a consistent temperature value (t=0.7) throughout the entire experiment. We will include the information about temperature value in the final version of the paper.

---

[W5] The lack of evaluation of Llama 3.1-8B on GSM8K.

Following your suggestion, we have additionally trained and evaluated ReVISE using the Llama 3.1-8B on GSM8K. As shown in the table below (Maj@1), ReVISE consistently outperforms baseline methods under this larger model setting, demonstrating that our approach scales well with model size.

\begin{array}{lc} \hline Method & Accuracy \newline \hline \text{SFT} & 58.2 \newline \text{RFT} & 58.9 \newline \text{STaR}^+ & 59.2 \newline \text{ReVISE (Ours) } & 61.6 \newline \hline \end{array}

Dear reviewer cTTG,

We sincerely appreciate your efforts and comments to improve the manuscript. We respond to your comment in what follows.

---

[W1] The diversity of the benchmarks is limited. It would be better to increase the range of task types here to fully verify the effectiveness of the proposed method.

To address the concern about benchmark diversity, we added experiments on the MBPP coding benchmark. ReVISE, trained on the Llama 3.2-1B model, achieves 33.1%, outperforming the second-best baseline at 30.7%. Since MBPP lacks reasoning annotations, we generated ground truth using GPT-4o, which is used to train the baselines and ReVISE. These results further support ReVISE general applicability across domains.

\begin{array}{lc} \hline Method & Pass@1 \newline \hline \text{3 Shots} & 24.5 \newline \text{SFT} & 30.0 \newline \text{RFT} & 29.6 \newline \text{STaR+} & 30.7 \newline \text{ReVISE (Ours)} & 33.1 \newline \hline \end{array}

---

[W2] As the baseline, how much performance gain can be achieved by RL finetuning only (specifically DPO)?

Following your suggestion, we have additionally considered an RL-based baseline, specifically a DPO-trained model. To be more specific, we trained the DPO model from an SFT model and constructed a preference pair dataset that paired the ground truth as chosen and the wrong answer as rejected. As shown in the table below (Maj@1), ReVISE even outperforms the RL-based baseline, indicating the 22.7% in gsm8k and 10.8% in MATH-500.

\begin{array}{lcc} \hline Model & GSM8K & MATH \newline \hline \text{DPO} & 22.7 & 10.8 \newline \text{ReVISE (Ours)} & 28.1 & 13.4 \newline \hline \end{array}

---

[Q1] Can ReVISE be extended to perform, refinement within the reasoning trajectory?

Thank you for your great question. It is true that the current version explicitly verifies after generating all reasoning traces and answers. We believe it will be a great future work if the model can detect errors during the reasoning trace generation, e.g., train the model to emit a [refine] token midway when there is an error in the reasoning trace where the error can be detected by using an LLM judge. We thank the reviewer for the question and will discuss this future direction in the final draft.