Thank you for your positive feedback.

Data size, prompt details, and filtering strategies

We have ensured that the core content regarding data size, prompt details, and filtering strategies is included in the main text, so readers can access the necessary information directly.

• In the Experiments section under Dataset Sources, we have specified the size of the dataset as 40k QA pairs. Additionally, we provide further details about the dataset size in the Appendix.
• In the Methods section under Thought Mask Mechanism, we have briefly described the prompt details. For example, we provide refinement instructions such as, "Please continue thinking and refine your answer." Further specifics about the prompt instructions and inference results are included in the Appendix.
• For filtering strategies , are discussed in 3.1.1 QUERY PREPARATION , where we detail the steps:
  1. Removing noisy and irrelevant queries, such as URLs or image links.
  2. Ensuring that test set questions are not mixed into the training set to prevent data leakage.
  3. Employing Thoughts-Answer Consistency Filtering to further verify logical consistency and improve the quality of the training dataset. Details of this consistency verification method are also provided in the Appendix.

Given the limited space in the main text, we have placed more detailed and technical information at the beginning of the Appendix, so readers can quickly find supplementary material. We hope this structure effectively helps readers access relevant content efficiently. If you have any further suggestions, we are happy to optimize this further.

Clear guidelines on the optimal number of iterations for masking and fine-tuning.

We appreciate your feedback regarding the need for clearer guidelines on the optimal number of iterations.

Based on the content of the article, we have already addressed both the number of fine-tuning steps and the number of reasoning iterations required for optimal performance:

1. Fine-tuning Steps:

2. In section 4.4, the article demonstrates that significant improvements in performance emerge after approximately 24,000 training steps (equivalent to 93 million tokens) for our base model. At this point, the model begins to show refined capabilities. The performance continues to improve as training progresses, with \model{} achieving an overall improvement from 40.1% to 55.6%. This shows that, while there is a clear upward trend in performance during training, a substantial number of training steps (around 24,000) are needed for more complex reasoning tasks to emerge.

3. Reasoning Iterations:

4. In section 4.5, Regarding iterative thinking steps, the article reveals that iterative refinement plays a crucial role in achieving optimal performance. In particular, the first three iterations show significant performance improvements across various tasks, such as GSM8K (from 75.0% to 79.9%) and ARC (from 58.6% to 65.2%).

more robust demonstration of PTR’s reasoning capabilities.

Thank you for your suggestion! We appreciate your recommendation to test reasoning and generalization capabilities using datasets like MMLU-Pro+ (arXiv:2409.02257). In fact, we have already conducted experiments on 10 different datasets, which we believe cover a wide range of capabilities, including general knowledge, code comprehension, summarization, math reasoning, and knowledge. These datasets include MMLU, Humaneval, DROP, XSum, GSM8K, Math, ARC, GPQA, Winogrande, and CommonsenseQA. We believe this range of tests is sufficient to demonstrate the robustness and generalization capabilities of our model.

That being said, we have also added results from MMLU-Pro+ as follows:

We observe similar improvement results on MMLU-Pro+, where the base model fails to improve for multiple iterations and shows a significant drop in performance after the second iteration. However, our model demonstrates self-improvement, increasing accuracy by 2.5%. These results further validate the effectiveness of our approach, demonstrating its ability to enhance reasoning capabilities. We are pleased that our findings align with the robustness requirements highlighted in your comment.

We will include this experiment in the revision version of our paper and cite the MMLU-Pro+ (arXiv:2409.02257). We sincerely appreciate the reviewer’s constructive suggestions. The additional experiments, analyses, and explanations have significantly enhanced the quality of our submission. We hope this provides a strong basis for a score increase.

clear guidelines on the optimal number of iterations for masking and fine-tuning.

Fine-tuning Steps:

1. In section 4.4, the article demonstrates that significant performance improvements emerge after approximately 24,000 training steps (equivalent to 93 million tokens) for our base model. At this point, the model begins to show refined capabilities. The performance continues to improve as training progresses, with our methods achieving an overall improvement from 40.1% to 55.6%. This shows that, while there is a clear upward trend in performance during training, a substantial number of training steps (around 24,000) are needed for more complex reasoning tasks to emerge.

To be more specific

fine-tuning setting

Our fine-tuning costs are significantly lower than other methods. Only around 24000 steps of iteration with 40K data could achieve self-refinement capabilities across various domains. It is unnecessary to fine-tune and construct data for every specific task.

data size

• In the Experiments section under Dataset Sources, we have specified the size of the dataset as 40k QA pairs. In line 286 Our model has trained on our PTR (Progressive Thought Refinement) dataset, and derived queries from the WizardLM dataset. After thorough cleaning, we reduced the original dataset from approximately 50k QA pairs to 40k high-quality QA pairs.

prompt details

• In the Methods section under Thought Mask Mechanism, we have briefly described the prompt details. For example, we provide refinement instructions such as, "Please continue thinking and refine your answer." Further specifics about the prompt instructions and inference results are included in the Appendix.

For training stages:

we simply use "Please continue thinking and refine your answer." to let model learn how to improve their answers.

For inference stages in line324:

For main experiements' results, we also use "Please continue thinking and refine your answer." to ask model to improve their answers.

For inference stages in line397:

we also tests robustness of our methods on different prompts. As in real world tasks, users may ask model to refine their answer in multiple ways.

Results of our methods with different prompts to test the robostic of our trianing methods:

• Assume that this thought could be either correct or incorrect. Carefully review the thought and provide a better answer.

• Review your previous thought and assess whether it's correct. Then, provide a better response based on your answer.

• Regardless of whether your previous thought is correct or not, provide a better answer.

Notibly, the model is not trained with these prompts.

The results of our methods shows that our methods can improve models performance on different prompts.

filtering strategies

Our filtering strategies include 3 parts:

In 3.1.1 Query Preparation

To enhance the model’s generalization, we avoid creating domain-specific datasets. Instead, we use queries from open-domain general datasets ensuring the model develops general refinement abilities rather than specializing in specific areas. Our data preprocessing involves three key steps. First, we perform data cleaning to remove noise and irrelevant content, such as images or URLs. Second, to prevent data leakage, we exclude domain-specific testing queries during training. Finally, we incorporate traditional SFT data (queries and answers) into our dataset to mitigate the risk of catastrophic forgetting.

And in Thoughts-Answer Consistency Filtering in line 245

To further ensure that the thought process exhibits logical coherence, we apply consistency filtering to remove inconsistent outputs. If the consistency score is below a certain threshold, the pair is considered inconsistent and removed, ensuring that only coherent thought sequences are used for the final output.

​

Thank you for your valuable efforts and insightful comments. We have carefully addressed your concerns in detail below and hope our responses meet your expectations. As the discussion phase approaches its end, we look forward to any additional feedback or clarification requests and hope for a constructive dialogue to refine our work further.

Claiming that LLMs have intrinsic/inherence progressive refinement ability is sort of a big claim that needs more detailed investigations.

To address the reviewer’s concern regarding the claim that Large Language Models (LLMs) have an intrinsic ability for progressive refinement, we offer the following clarifications and elaborations:

1. Distinguishing Between Model Capabilities and Self-Improvement Ability

It is important to clarify that the inherent ability of a model to respond to tasks (its base performance) is different from its ability to self-improve or refine answers over time. While the base model's capabilities are largely determined by the knowledge it has accumulated during pre-training, we recognize that enabling progressive refinement is a separate, significant challenge. Our experiments show that while the base model does not exhibit any self-correction or refinement ability over multiple iterations, this ability can be triggered with our method. In our work, we demonstrate that, with just 50,000 data pairs and 24,000 training steps (equivalent to 93 million tokens), a 7B parameter model can exhibit significant self-improvement in 10 different unseen tasks. This process of fine-tuning activates the model’s ability to progressively refine its outputs, which is different from its original pre-trained capabilities. Thus, the self-improvement ability is not shown in the base model but rather emerges through additional fine-tuning, and this does not contradict the model's original base solid performance.

2. Comparison with Related Work

We also compare our method with three other approaches in the "Costs Comparison Table for each methods" responses, analyzing the cost and effectiveness of each. Our findings show that, while other methods can achieve similar improvements (2-3%), they require more expensive training and inference processes. Specifically, methods like Verifier rely on extensive retraining, which involves using clear positive and negative samples to construct datasets. This process demands additional training data every time a new task or domain is encountered. In contrast, our approach allows for the model to demonstrate self-improvement across a variety of tasks without the need for task-specific datasets. Furthermore, our method requires significantly fewer inference steps during evaluation: while other methods like Verifier require 100 inference steps per iteration, we find that just two or three iterations with our method provide comparable or better performance, making our approach more practical and cost-effective.

3. Key Contribution: Enabling Self-Improvement Across Unseen Tasks

The primary contribution of our work is not in improving scores in specific domains, but in enabling the model to learn self-improvement through this training approach, allowing it to demonstrate progressive refinement across different tasks.Since our method is not task-oriented, the model trained with our approach can still refine itself when encountering new scenarios or tasks it has never seen before, unlike the base model, which either does not improve or makes incorrect corrections.This significantly enhances the practical applicability of our method in real-world scenarios and also reduces the cost of training. We believe that, in real-world scenarios, even the most advanced models cannot achieve 100% accuracy, and there will always be instances where the model's responses are not ideal. Therefore, users may want the model to improve upon its previous answers. Existing methods mainly focus on specific domains, constructing task-oriented datasets to train models for improving self-refinement within those domains. However, in real life, tasks are not limited to just mathematics. Achieving self-improvement across a wider range of tasks is more challenging but also more valuable. Our method enables the model to demonstrate self-improvement across multiple tasks it has not been specifically trained on, which is the primary contribution of our work.

Continued in https://openreview.net/forum?id=pUbbLHjCPM&noteId=aBIAkvxvtw

When comparing this baseline with your method you should consider the complexity of your approach and the time analysis.

We also compared with other methods:

We present a detailed comparison of our method with existing approaches, highlighting the key differences in cost, applicability, and performance across tasks such as GSM8K and general downstream tasks.

Comparison Table

[1] Self-refine：Self-refine: Iterative refinement with self-feedback https://arxiv.org/abs/2303.17651

[2] Pair Self Correction： Generating Sequences by Learning to Self-Correct https://arxiv.org/abs/2211.00053

[3] Reward-model Verifier: Training Verifiers to Solve Math Word Problems https://arxiv.org/abs/2110.14168

Figure 3 has 4 plots with lots of information, small numbers w/o proper take-home messages in the caption. more details showed in paper

For Figure 3:

Plot A: A multi-line plot illustrating performance trends across 10 tasks, with the average performance depicted in black and variance shown as shaded regions. The overall trend is upward, with a red line marking the point where the average performance reaches 46%. Plot B: A bar plot comparing each task's initial, baseline, and final performance. The initial performance is lower than the baseline, but the final performance surpasses it, indicating that the model effectively learns and improves through successive iterations. Plot C: A box plot showcasing the performance distribution across tasks. The varying lengths of the bars represent performance improvements, with longer bars indicating greater improvements across different tasks. Plot D: A heat map visualizing task performance over training steps. The x-axis represents training steps, and the y-axis represents tasks. The color intensity reflects performance levels, with darker colors indicating better performance.

For Figure 4:

The performance of our methods is evaluated over ten iterations across various tasks. Left Plots: These show accuracy improvements across categories such as mathematical reasoning (GSM8K, MATH), reasoning tasks (ARC, GPQA, Winogrande, CommonsenseQA), comprehension tasks (MMLU, DROP, XSum), and coding tasks (HumanEval). The dashed line represents the model's baseline performance. Most tasks exceed the baseline after the second or third iteration, demonstrating significant learning and adaptation. Right Plots: Radar charts illustrate performance over six iterations. The blue-shaded areas highlight remarkable performance improvements during the first two iterations, followed by saturation in subsequent iterations. This pattern demonstrates rapid early gains with diminishing returns as the iterations progress.

The description for PTR vs. Knowledge Distillation (IFT) in line 320 is not clear and hand-wavy.

Description of PTR is not Knowledge Distillation

We also compare our method with a straightforward approach that uses a strong model's answers to queries via IFT. Our findings show that our method is not equivalent to knowledge distillation.

In the first iteration, we observe that when models are trained on general datasets rather than domain-specific tasks, their initial performance tends to decline. This performance drop primarily arises from supervised fine-tuning amplifying the base model’s inherent biases. When trained on general datasets, base models often accumulate biases that may not align with specific domains, resulting in poorer performance on domain-specific tasks.

However, the IFT approach fails to activate the model’s self-refinement ability and does not significantly improve performance after the initial attempts. For example, on CommonsenseQA, the IFT approach yields a lower performance at the second iteration (40.3%) compared to the first (46.1%).

In contrast, our approach demonstrates consistent improvement through iterative attempts without relying on domain-specific knowledge. This highlights that our method is not merely distilling knowledge but rather activating the model’s ability to refine its outputs and enhance performance through self-driven iterative improvement.

What is the "PRD" dataset mentioned in the paragraph of line 303?

We clarify the PTR dataset in line286 Our model has trained on our PTR (Progressive Thought Refinement) dataset, and derived queries from the WizardLM dataset[1]. After thorough cleaning in Section 3.1.1, we reduced the original dataset from approximately 50k QA pairs to 40k high-quality QA pairs.

More detail in Section 3.1.1 Query Preparation

Our filtering strategies include 3 parts:

To enhance the model’s generalization, we avoid creating domain-specific datasets. Instead, we use queries from open-domain general datasets ensuring the model develops general refinement abilities rather than specializing in specific areas. Our data preprocessing involves three key steps. First, we perform data cleaning to remove noise and irrelevant content, such as images or URLs. Second, to prevent data leakage, we exclude domain-specific testing queries during training. Finally, we incorporate traditional SFT data (queries and answers generated by a strong model) into our dataset to mitigate the risk of catastrophic forgetting.

In line 203 Thought-Answer responses in section 3.1.2

We also construct Thought-Answer responses, to construct thoughts via the weak model and final answer via the strong model. The objective is to ensure that the final answer is progressively improved through multiple iterations rather than relying on a single-step response.

Through this approach, we aim to enable the model to learn the ability of self-refinement, gradually enhancing its answers from initial, imperfect responses to more refined ones. Our experiments demonstrate that, by using a self-improvement prompt, our model can significantly improve the quality of its answers. However, the base model lacks this self-improvement capability and even performs worse in multiple iterations. This is one of the key contributions of our paper.

In line 239 section 3.2 Progressive Weighted Thought-Mask Fine-tuning

and finally, our dataset is constructed below:

$D = { ( q_i, S_{i, thought},\hat{ y_{i,s,icl}} ) }_{i=1}^N$

Consisting of the input query $q_i$, the initial thought sequence $S_{\text{i, thought}}$, and the final answer $\hat{ y_{i,s,icl}}$.

[1] WizardLM: Empowering Large Language Models to Follow Complex Instructions

Have the authors investigated the effect of different losses of equation 3.4 in an ablation study?

1. We conduct an ablation analysis to evaluate the impact of each hyperparameter.
2. Additionally, we also explore suggestions for setting these parameters, which could simplify the tuning process and make our method more accessible for broader applications.

In our experiments, we conducted sensitivity studies to evaluate the impact of these hyperparameters on model performance. The results indicate that different configurations of λ1,λ2, and λ3 significantly affect the balance between accuracy, reasoning consistency, and confidence distribution.

We observed that setting the weights to λ1=0.8, λ2=0.1, and λ3=0.1 resulted in the highest average accuracy of 64.3%, effectively balancing final answer accuracy, reasoning consistency, and confidence distribution.

From a broader application perspective, however, using λ1=1.0,λ2=0.0,λ3=0.0 (focusing solely on final answer accuracy) also achieved an average accuracy of 63.0%. To simplify training and reduce complexity for real-world scenarios, we recommend focusing solely on λ1, as it can still achieve competitive performance.

We also identified the following findings:

1. Impact of low λ1: Experiments show that setting λ1 too low negatively impacts the accuracy of the final answer. For example, when λ1=0.6, the model’s training deviates, significantly reducing the accuracy of the initial response and diminishing its ability to self-refine. This indicates that λ1 should not be set too low in practice.
2. Roles of λ2 and λ3: Adding small weights for λ2 (reasoning consistency) and λ3 (confidence progression) supports the model’s ability to self-refine during reasoning, thereby improving overall performance. Logically, these parameters help the model refine its reasoning over time. However, their contribution to final accuracy is relatively limited compared to the cost of hyperparameter tuning and computational overhead. Therefore, unless the task explicitly requires higher interpretability or improved confidence in subsequent reasoning steps, excessive tuning of these parameters may not be worthwhile.

For tasks requiring high reasoning consistency and confidence distribution optimization, we recommend λ1=0.8,λ2=0.1,λ3=0.1. However, for most real-world applications, the simpler configuration of λ1=1.0,λ2=0.0,λ3=0.0 is also sufficient.

4. Avoiding Explicit Labeling for Dataset Construction

Another significant innovation in our work is that we do not require explicitly labeled data (such as correct/incorrect labels) to train the model. Many existing methods rely on external feedback to construct labeled datasets, a process that can be costly and labor-intensive. In contrast, We use a strong-weak model collaborative filtering approach, which eliminates the need for external feedback to construct datasets, addressing the high costs of using external feedback for dataset construction. We believe that, in general, a stronger model performs better than a weaker one, and we have demonstrated this through the Wilcoxon Signed-Rank Test analysis in Appendix B.5. Even though our dataset does not provide explicit feedback or labeling for correctness, the model can still learn to improve itself through our datasets and training approach. The experiments confirm that the model can indeed refine its previous answers.

5. Practical Value of Self-Improvement in Real-World Scenarios

Furthermore, since fine-tuning for each task is not a feasible solution, our method offers greater practical value when encountering new or difficult-to-annotate tasks. Our method offers a practical solution by enabling the model to self-improve across diverse tasks after fine-tuning. This reduces the need for task-specific fine-tuning, making it more feasible to apply our approach in real-world settings. Our method demonstrates that models can self-refine even when encountering tasks they have never seen before, enhancing their utility and applicability in dynamic environments.

In summary, we acknowledge that claiming intrinsic self-improvement ability is a significant assertion; however, we believe our detailed investigations and experiments provide strong evidence that such an ability can be activated and leveraged through fine-tuning. Our work does not merely improve model performance in specific tasks but empowers the model to refine its answers across a wide range of tasks, even those it has not encountered during training. This is a key contribution to the field, offering both practical and cost-effective benefits for real-world applications.

Below are the experimental results comparing performance on mathematical tasks.

We compare our method to prior and baseline approaches, evaluating performance on GSM8K and MATH.

We sincerely thank the reviewer once again for their valuable comments and constructive feedback. We have worked diligently to address all concerns and improve the clarity, quality, and depth of our work. We hope that our revisions meet your expectations and address the issues raised. We are open to further discussion and would greatly appreciate any additional feedback to enhance the manuscript. Your insights have been instrumental in refining our work, and we look forward to the possibility of an improved evaluation. Thank you for your time and thoughtful review.

Q1 clearly defined on 3.4 equation in main paper

We acknowledge that the components of Equation 3.4 could benefit from more explicit definitions in main pages. We clarify each element of the equation to ensure it is comprehensible. We clarify each element of the equation to ensure it is comprehensible. Due to space constraints, more details can be included in the appendix.

Q2 broader application advice for hyperparameter complexity, ablation study

We appreciate your observation regarding the hyperparameter complexity. Indeed, balancing the model's loss function with multiple hyperparameters can introduce challenges.

1. We conduct a sensitivity analysis to evaluate the impact of each hyperparameter.
2. Additionally, we also explore suggestions for setting these parameters, which could simplify the tuning process and make our method more accessible for broader applications.

Hyperparameter Complexity: While hyperparameters λ1,λ2,λ3 help balance various aspects of the model's loss function, extensive tuning hyperparameters could limit PTR's practical adoption.

In our experiments, we conducted ablation studies to evaluate the impact of these hyperparameters on model performance. The results indicate that different configurations of λ1,λ2,λ3 significantly affect the balance between accuracy, reasoning consistency, and confidence distribution.

We observed that setting the weights to λ1=0.8,λ2=0.1,λ3=0.1 resulted in the highest average accuracy of 64.3%, effectively balancing final answer accuracy, reasoning consistency, and confidence distribution.

From a broader application perspective, however, using λ1=1.0,λ2=0.0,λ3=0.0 (focusing solely on final answer accuracy) also achieved an average accuracy of 63.0%. To simplify training and reduce complexity for real-world scenarios, we recommend focusing solely on λ1, as it can still achieve competitive performance.

We also identified the following findings:

1. Impact of low λ1: Experiments show that setting λ1 too low negatively impacts the accuracy of the final answer. For example, when λ1=0.6, the model’s training deviates, significantly reducing the accuracy of the initial response and diminishing its ability to self-refine. This indicates that λ1 should not be set too low in practice.
2. Roles of λ2 and λ3: Adding small weights for λ2 (reasoning consistency) and λ3 (confidence progression) supports the model’s ability to self-refine during reasoning, thereby improving overall performance. Logically, these parameters help the model refine its reasoning over time. However, their contribution to final accuracy is relatively limited compared to the cost of hyperparameter tuning and computational overhead. Therefore, unless the task explicitly requires higher interpretability or improved confidence in subsequent reasoning steps, excessive tuning of these parameters may not be worthwhile.

For tasks requiring high reasoning consistency and confidence distribution optimization, we recommend λ1=0.8,λ2=0.1,λ3=0.1. However, for most real-world applications, the simpler configuration of λ1=1.0,λ2=0.0,λ3=0.0 is also sufficient.

We will include the ablation study results and further recommendations in the appendix of the next version of our paper.

Q4 modest improvements

1. Key Contribution: The primary contribution of our work lies in enabling models to develop self-refinement capabilities through training. Our experiments show that without training, open-source models tend to worsen their initial answers when attempting to refine them. However, our training method equips the model with the ability to refine its responses effectively. The activation of self-refinement capabilities in models is an exciting development, and we plan to explore methods to further enhance the magnitude of improvement in future work.

   Generalization Across Domains: Even in related self-refinement approaches, the improvements achieved through fine-tuning in mathematical domains are relatively modest. In contrast, our method demonstrates improvements across a broader range of domains. For instance, the overall performance of our model increases from 49.6% to 53.5%. This improvement in the model's self-refinement ability, particularly its generalization across diverse fields, is a major contribution to our work.

Q3 cost of this method and our main contribution

Thanks for pointing out the problem of training costs, here I will further elaborate on the reasons and compare other approaches. Other methods have higher computational costs, but they don't perform as well as ours.

We did a comparison to previous work in the field of mathematics in Reviewer 9DXQ Q2 responses, our model also has several strengths below:

Constructing data Costs lower

PTR does not require separate dataset construction or fine-tuning for each task, significantly reducing costs.

Constructing data for other approaches is relatively more challenging, and the data construction process takes significantly longer. The Reward Model Verifier approach incurs high costs during data construction, as it requires best-of-n sampling and ground-truth answers to filter positive and negative examples. Additionally, its inference process is n times slower.

Additionally, while our method shows slightly lower absolute performance compared to some other methods, it proves effective across all tasks without relying on external verifiers or additional task-specific fine-tuning, which those methods require.

Inference Costs lower

• As outlined in our response to reviewer 9DXQ:

  • The Reward Model Verifier approach incurs extra costs during inference, as it requires verifier and ground-truth answers to check the responses.
  • Our method requires only 2–3 iterations for inference and eliminates the need for external feedback, significantly reducing inference complexity.

Training Costs lower

• our data construction process requires only a single inference pass on a general dataset, allowing for performance improvements across all domains.

• In contrast, methods such as self-refine and Reward Model Verifiers are task-oriented, requiring separate training datasets for different tasks and relying on domains like mathematics, where correctness can be clearly defined.

• These approaches face limitations in more open-ended domains. However, our method achieves improvements even in non-mathematical domains, highlighting its broader applicability and efficiency.

We sincerely thank the reviewer for their thoughtful feedback and encouraging comments on our work. We have carefully addressed each concern and provided detailed clarifications and additional insights in our responses. We hope these revisions meet your expectations and effectively highlight the contributions of our work. Your insights have been invaluable in refining the manuscript, and we welcome any further feedback or suggestions to improve it further. We look forward to the possibility of an improved evaluation and continued discussion. Thank you again for your time and constructive review!

If these clarifications or revisions pdf are not visible in the current stages, I apologize for the oversight. It’s possible that the revised edition is not yet accessible to reviewers at this stage. To address this, I will provide the relevant details here for your reference.

Main paper line:259

$L_{PTR}(\theta) = \sum\limits_{(q_i, S_{i, \text{thought}}, y_n) \in \tilde{D}} = \Bigg[ -\lambda_1 \log \Pr(y_n \mid q_i, S_{i,\text{thought}}; \theta) \nonumber$ $+ \lambda_2 \sum\limits_{t=2}^{n} F_{\text{cons}}(y_t, y_{t-1}) + \lambda_3 \sum\limits_{t=1}^{n} \beta_t \left(1 - \Pr(y_t \mid q_i, S_{i,\text{thought}}; \theta)\right) \Bigg].$

Unlike standard supervised fine-tuning, which trains the model to produce a single response $\hat{y}$ given $x$,

$-\lambda_1 \log \Pr(y_n \mid q_i, S_{i,\text{thought}}; \theta)$ focuses exclusively on the accuracy of the final response generated after the thought refinement process.

$F_{\text{cons}}(y_t, y_{t-1})$ ensures that the current response remains logically consistent with the previous thought sequence by computing the Cosine Similarity with the Sentence-BERT [1] (See Appendix A.2)

$\left(1 - \Pr(y_t \mid q_i, S_{i,\text{thought}}; \theta)\right)$ represents the model’s uncertainty or error probability at each reasoning step, which measures the confidence of the model’s predictions. The term $\beta_t$ represents the confidence score at each reasoning step, which increases as reasoning progresses to encourage higher certainty in later steps (see Appendix A.3).

Here, $\lambda_1$, $\lambda_2$, and $\lambda_3$ are dynamically adjusted according to the model's needs. We set λ1=0.8, λ2=0.1, and λ3=0.1, with their sum weighted to 1.

Appendix A.2 line 773

A.2 Consistency Compute

To evaluate the consistency between the thought sequences and the final answer, we vectorize the thoughts and the answer using sentence-BERT embeddings. Sentence-BERT provides an effective way to embed both the thought sequences and the final answer into a shared vector space, capturing semantic similarities between them.

The similarity between each thought $y_t$ and the prior thought $y_{t−1}$ is computed using the cosine similarity between their Sentence-BERT embeddings. The consistency score is designed to capture how well the current thought $y_t$ is consistent with the prior thought $y_{t−1}$ and how closely it relates to the final answer.

We define a Consistency Function $F_{\text{cons}}(y_t, y_{t-1})$ as the cosine similarity between the current thought $y_t$ and the prior thought $y_{t−1}$. The function is computed as:

Where $\text{CosineSimilarity}(a, b)$ is defined as:

Here, a and b are the embeddings of $y_t$ and $y_{t−1}$ respectively, generated by the Sentence-BERT model. The cosine similarity measures the angle between the two vectors in the embedding space, with values closer to 1 indicating high similarity and values closer to 0 indicating low similarity.

A.3 Dynamic Confidence Adjustment

By introducing a dynamic confidence decay strategy, the model can gradually increase its confidence during the reasoning process. For example, the confidence $\beta_t$ can increase as the reasoning step tt progresses, instead of staying constant throughout the process. This will allow the model to gain more confidence as it refines its thoughts and reasoning.

By adjusting the weights of each reasoning step, $\beta_t$ can control how the model’s confidence evolves. At the initial steps, the model can have a lower confidence since the thought process is still being refined. As the reasoning progresses, the model should gradually increase its confidence and have higher confidence at the final step. This can be achieved by adjusting $\beta_t$ dynamically like so:

where $\beta_0$ is the initial confidence and n is the total number of reasoning steps. This approach ensures that the confidence $\beta_t$ increases as reasoning progresses.

Q2 compare with prior works on refinement methods or approaches with verifiers

We appreciate your valuable suggestion. As noted by the reviewer, existing methods that rely on additional feedback for comparison are not applicable to all datasets and require domain-specific data construction, such as in mathematics. However, our approach does not involve fine-tuning on mathematical datasets, which introduces a potential unfairness in comparison. To address this, we have supplemented our experiments and found that even when additional feedback is leveraged, the improvements are similar. In contrast, for non-mathematical and non-reasoning tasks, such methods struggle to generalize, whereas our approach continues to achieve improvements.

This highlights the primary contribution of our work: it does not require task-specific fine-tuning, which distinguishes it from previous self-refinement methods.

Below are the experimental results comparing performance on mathematical tasks.

We compare our method to prior and baseline approaches, evaluating performance on GSM8K and MATH. Reward-model Verifier https://arxiv.org/abs/2110.14168 Training Verifiers to Solve Math Word Problems Self-refine：arXiv preprint arXiv:2303.17651 Self-refine: Iterative refinement with self-feedback Pair Self Correction：https://openreview.net/forum?id=hH36JeQZDaO Generating Sequences by Learning to Self-Correct

Performance similar to other refine-work on math

Experimental results show that performance improvements across methods are modest, typically within 1-2%. The self-refine approach relies on ground-truth feedback for slight gains; without it, performance often deteriorates. Similarly, Pair Self-Correction and Reward-Model Verifier achieve comparable improvements but remain limited.

General Task Improvement

In contrast, my method achieves similar improvements and demonstrates effectiveness in other non-mathematical and non-reasoning tasks where their approach fails to test and hard to construct training datasets. This highlights the limitation of the self-refine structure in effectively improving model performance without external guidance.

Without external verifier or task-oriented fine-tuning

• Our method does not require separate dataset construction or fine-tuning for each task, significantly reducing training costs.
• The Reward Model approach requires constructing positive and negative samples for each task and training a reward model to supervise the generation process.
• Similarly, the Pair Self-Correction method involves generating and labeling a large number of correct and incorrect sample pairs to achieve error correction capabilities. These processes are both costly and heavily dependent on specific domains.

Inference Costs lower

• The Reward Model Verifier approach incurs high costs during data construction, as it requires best-of-n sampling and ground-truth answers to filter positive and negative examples. Additionally, its inference process is n times slower.

• Our method requires only 2–3 iterations for inference and eliminates the need for external feedback, significantly reducing inference complexity.

• Furthermore, our data construction process requires only a single training pass on a general dataset, allowing for performance improvements across all domains.

Coding task

The data construction for coding tasks is significantly more complex, requiring the creation of a large number of positive and negative samples to train the verifiers. We plan to further evaluate and validate the effectiveness of our approach compared to previous methods in future work. This also highlights the higher applicability and feasibility of our method, as it does not rely on fine-tuning for specific domains but instead enhances self-improvement capabilities across a wide range of tasks.

Q4 Effectiveness of the Thought-Mask Strategy

Effectiveness of the Thought-Mask Strategy We appreciate your interest in the role of the thought-mask strategy.

We also conducted experiments without the masking strategy, and the results were worse compared to when the mask was applied. This is because our mask covers the intermediate thought process, and we aim to guide the model's attention during training on how to refine its answer based on previous thoughts to arrive at a better response. Without the mask, the model also computes immature thoughts that arise after the question, but these are often incorrect, leading the model to lean toward providing a worse answer initially, which is unnecessary.

In our case, the IFT method involves removing the thought process and directly fine-tuning using the input and the strong model's answer. This method helps demonstrate that our data construction format is not just a simple distillation of the large model's answering ability into a smaller model. We observe that basic distillation using general data does not lead to significant improvements on specific tasks and cannot enable the model to continuously self-improve. On the contrary, it validates that our method successfully triggers the model's ability to self-refine.

Q5 fine-tuning setting

Our fine-tuning costs are significantly lower than other methods. Only around 24000 steps iteration with 40K data could achieve self-refinement capabilities across various domains. It is unnecessary to fine-tune and construct data for every specific task.

Inference requires only two to three iterations, without needing feedback for supervision.

However, other method use best-of-n to construct datasets and use ground-truth to confirm the correctness, which is much higher than our methods, typically 10 times larger when using best-of-10 to construct data.

In contrast, other methods involve generating multiple answers during data construction and using external feedback mechanisms to distinguish correct and incorrect responses. This reliance on external feedback has two major drawbacks:

1. External feedback mechanisms are task-specific, making it difficult to ensure accurate feedback for more open-ended tasks.
2. Training a separate feedback model for supervision needs higher costs.

PTR avoids these issues while maintaining lower training and inference costs with comparable or superior results.

We sincerely thank the reviewer for their thorough evaluation and thoughtful feedback on our work. We have addressed each of the points raised, clarifying our contributions, methodologies, and experimental results in detail. Your comments have been instrumental in helping us refine our manuscript, and we greatly appreciate the opportunity to further improve our work. We look forward to any additional feedback or suggestions you may have and remain hopeful for a positive evaluation. Thank you again for your time and constructive insights!

Q1 main contributions and novel points of this paper over some prior refinement works

Prior works typically rely on external feedback to assess the quality of model responses and to construct labeled datasets for training. This approach is costly, especially for open-ended tasks where evaluating generated responses is challenging. As a result, most benchmarks in previous work have been confined only to mathematical or coding tasks. In contrast, our goal is to empower the model's self-improvement capabilities, allowing it to continually refine itself without the need for rewards or explicit correctness indicators. This method enables the low-cost and large-scale construction of datasets, ultimately improving the model's performance in all fields.

Additionally, earlier self-improvement approaches, such as designing prompts or fine-tuning with specific preference datasets, allow the model to improve in certain domains. However, this comes with significant challenges when facing unseen tasks. Limited generalization beyond their training domains.

In contrast, our method does not rely on domain-specific datasets and no labeling for correctness. Our experimental results confirm that this approach can enhance the model's capability without being limited by the task-specific fine-tuning required by previous methods.

Q3 our baseline showed worse performance

Our contribution lies in demonstrating that our reasoning mask strategy and the optimization of the dataset using strong and weak model pairs can effectively unlock the model's self-improvement capabilities. This self-improvement ability is then exhibited consistently across all tasks.

The baseline models used for comparison include an initial model without fine-tuning, which serves to demonstrate that models lack inherent self-improvement capabilities without additional fine-tuning. Additionally, we compared against an SFT (supervised fine-tuning) model, which is explained in detail in the experimental section. The SFT model was fine-tuned using answers directly taken from a strong model, aimed at proving that our dataset does not suffer from data leakage.

Moreover, our training approach goal is not about memorizing the correct answers to specific questions but about enabling the model to learn how to iteratively refine its responses.

Q6 misunderstanding of extra feedback

There seems to be some misunderstanding here, particularly regarding the role of external verifiers providing feedback. These verifiers include compiler tools and fine-tuned models.

Let me elaborate further on this feedback. External feedback can be categorized into several types:

1. Before training (during dataset construction): External feedback is used to label correct and incorrect answers to construct the dataset. This requires external tools or models to annotate data accurately.
2. After training: External feedback supervises the model's responses to decide whether a response is correct and whether the model should generate further responses.
3. During training (online training): Feedback in this stage often involves a reward model that scores the model's outputs and dynamically updates the weights. However, training a reward model requires a pre-labeled dataset, and these reward models are typically designed for specific domains with specialized discriminative capabilities.

Thus, external feedback, whether used before, during, or after training, has significant limitations. Firstly, constructing a discriminative model or dataset is costly and often domain-specific, requiring specialized feedback models. Secondly, it adds extra inference overhead, requiring more complex pipelines to evaluate and refine model responses.

In contrast, our approach eliminates the need for external feedback at any stage, whether before or after training. This is because we recognize that in many scenarios, the boundaries of correctness are ambiguous, making external feedback both costly and less practical. Instead, we rely on self-improving datasets to enable the model to continually enhance its performance across all domains. This universality and simplicity represent our key contribution.

Sorry for the late responses, and thanks for the authors' detailed responses!

Most of things get clear for me, especially the novelty points of this paper compared to prior works.

To be clear, as I understood, this paper introduce PTR framework that enables the models to self-refine with the proposed PTR dataset. This dataset is constructed through two weak and strong models, thus; the cost is relatively small to make this and the task is agnostic as it doesn't need the correctness label compared to previous verifier-based methods.

Related to specific questions I asked:

Q1. Thanks to clarifying the novel points.

Q2. Thanks, and I think this experiment will make the contribution of this paper much clear. Though the model is not fine-tuned to specific datasets, which makes the authors' method suffer from lower initial performance.

Q3. There is one followup question: other three methods show worse performance after refinement, because this model is not fine-tuned on downstream tasks? If they are fine-tuned on each tasks, will they possibly show good capability in refinement, as you mentioned in your answer to Q1?

Q4. Thanks.

Q5. Thanks.

Q6. One followup question: can we say PTR does not use external feedback, given they used external strong model while constructing dataset? It's not feedback and very cheap, but could be quite vague?---Though I like the contributions of this method.

I raised my score to 6. Sorry for misunderstanding the novel points for the paper at the first time. But I think the authors could enhance the writing and provide much clear explanation to compare to prior related works (also please add results and discussions related to Q2 and Q3).

Thanks for elaborating more the contributions and comparison table with prior works. I believe that these explanations would enhance the paper much stronger. I maintain my updated score at this point, but I'm quite positive for the acceptance of this paper.

As 2 reviewers asked about the costs and effeciency of our method compared with other prior works, we clairfy the main contribution and advantages here to show our novel points and strength.

Advantages of Our Approach

1. Lower Cost:(https://openreview.net/forum?id=pUbbLHjCPM&noteId=XL2MjH2S3C)

• Both Pair Self-Correction and Reward-Model Verifier rely on large-scale models (e.g., 175B parameters) to generate correct and incorrect answers for dataset construction. Reward-Model Verifier further requires 100 inference samples per query to improve the probability of a correct answer. (https://openreview.net/forum?id=pUbbLHjCPM&noteId=nRyyXFHi6D)
• In contrast, our method avoids stringent requirements for model accuracy during data generation. We use a strong model (Qwen-70B) and a weaker model (Qwen-2.7B) to generate data with only two inferences, allowing the model to learn self-improvement from the responses of both.

2. Broader Applicability:

• Existing approaches are limited to tasks with clear, objective correctness criteria, such as mathematical problems. They require task-specific datasets and fine-tuning, which are costly or infeasible for subjective or open-ended tasks. (https://openreview.net/forum?id=pUbbLHjCPM&noteId=nRyyXFHi6D)
• Our method, however, maintains strong adaptability across diverse tasks without additional fine-tuning. This enables the model to generalize effectively to unseen tasks.

Key Contributions to Our Work

The primary contribution of our work is not in improving scores in specific domains, but in enabling the model to learn self-improvement through this training approach, allowing it to demonstrate progressive refinement across different tasks. Since our method is not task-oriented, the model trained with our approach can still refine itself when encountering new scenarios or tasks it has never seen before, unlike the base model, which either does not improve or makes incorrect corrections.

We believe that, in real-world scenarios, even the most advanced models cannot achieve 100% accuracy, and there will always be instances where the model's responses are not ideal. Therefore, users may want the model to improve upon its previous answers. Existing methods mainly focus on specific domains, constructing task-oriented datasets to train models for improving self-refinement within those domains. However, in real life, tasks are not limited to just mathematics. Achieving self-improvement across a wider range of tasks is more challenging but also more valuable. Our method enables the model to demonstrate self-improvement across multiple tasks it has not been specifically trained on, which is the primary contribution of our work.

1. Eliminating the Need for External Feedback:
   1. Our method addresses the high cost of dataset construction by removing the reliance on external feedback for labeling correctness. Instead, the model learns self-improvement from general domain queries and the interactions between strong and weak models.
   2. Even without explicit correctness labels, our datasets still can enable the model to improve previous answers.
2. Practical Value Across Tasks:
   1. Unlike task-oriented methods, which require fine-tuning for each specific task, our approach is highly practical for new or difficult-to-annotate tasks. The model undergoes a single fine-tuning step, after which it demonstrates self-improvement across all unseen tasks.
3. Real-World Adaptability:
   1. Our method reduces the reliance on clear correctness criteria, making it more accessible and effective in real-world scenarios where explicit answers are not always available. By fostering self-improvement, our approach ensures performance gains across diverse, unseen domains — a contribution that other methods cannot achieve.