Self-Consistency Preference Optimization

We thank you for your detailed review and comments. We are also glad to see you found our paper to have “intuitive motivation” as well as “extensive experimental results”. Please find our in-depth response to your comments below.

Additional relevant baselines:

Thanks for pointing these papers out to us, we contrast ScPO to these works as:

• Chen et al. 2024: We argue that they are focused on general instruction following, and their method is not proven to work in reasoning tasks (as we do in ScPO) like math etc. Also, note that their main technical contribution is to add a length control factor to DPO’s implicit reward term. While “length bias” has been an issue with general instruction following tasks (Dubois et al 2024) as models spuriously prefer longer generations, preference evaluation for reasoning domains is more straightforward – preferring correct answers and rejecting the rest, so we expect length-controlled DPO to perform similarly to the standard DPO algorithm used in ScPO.
• Kim et al. 2025: At its core, this paper uses the model's confidence on two generations corresponding to the same prompt, to construct preference pairs with additional preference label smoothening for the bottom 10% annotations that are expected to be noisy. This stands in contrast to ScPO which uses self-consistency of predictions to generate preference annotations, and we compare the quality of the two metrics (confidence vs self-consistency) to curate preference data below to demonstrate the superiority of ScPO on reasoning tasks:
  • Setup: We incorporate Kim et al.’s strategy for preference data creation by using the model’s confidence. Specifically, we compute the confidence for each generation (from the same sample set used to compute SC) and create preference pairs by choosing the most confident response and rejecting the least confident response. Further, following Kim et al. (2025), we filter out the last 10% of training instances with the lowest difference in confidence of chosen and rejected responses (indicative of noise in annotation), and compare its Somer’s D correlation with accuracy with that of self-consistency (Table 6).

• Results: On ZebraLogic, we find that the accuracy of the most confident answer is in fact 0 rendering Kim et al.’s method completely ineffective, whereas SC on the other hand correlates strongly with correctness. Similarly, on GSM8K we find that confidence score of generation shows little correlation with correctness, and is most likely not effective to generate preference training data. The results indicate that across datasets self-consistency offers a better estimate of correctness of an answer, and therefore yields higher quality preference for training. These findings are consistent with prior work that uses self-consistency to estimate model confidence for reasoning tasks (Xiong et al. 2023, Kabra et al. 2023).

Ablation with number of samples (K)

We measure the impact of the number of samples used to measure self-consistency on its Somer’s D correlation with correctness (as done in Table 6) for K=2,4,8,16.

The results indicate that (i) lower values of K (e.g. K=2/4) have lower correlation with correctness or accuracy which we find is because of fewer instances where any answer gets multiple votes; (ii) while larger values of K=16 yield slightly higher correlations, we prioritize computational efficiency in the data generation phase (L100-122), and use a sufficiently large value of K=8 in addition to filtering (L118-121) and a weighted loss (L141-149) for ScPO training.

related works like $1, 2, 3, 4, 5$ where selecting high-quality data or weighting techniques are already common in DPO-based methods

Thank you for pointing out these works. We haven’t discussed them in our paper because they are about weighting prompts based on their quality, removing noisy or unhelpful questions. In contrast, our work is about weighing solutions, giving less importance to solutions that are likely to be wrong. However, it could be helpful to discuss those works and their differences, which we will do in the final revision.

We hope these additional results and explanations address your questions and will allow you to revisit your score.

We thank you for your review and for appreciating the “intuitive and simple” design of our method as well as our evaluation setup.

The choices are to certain extent reasonable, one concern here is that the self-improvement baseline, LSMI, is not quite up-to-date. Considering self-alignment as a quite active area, there should be more follow-ups suitable for serving as a baseline.

We are not aware of a more up-to-date follow-up of the LMSI baseline that is applicable to our unsupervised setting but please let us know if we missed something you had in mind. Furthermore, we note that two variants of IRPO as well as the 8B RM are fairly up to date since IRPO was published in Neurips 2024, and the Armo RM (also released in mid 2024) was among the best performing 8B reward model as per the RewardBench leaderboard at the time of development.

Another concern is that this work is very similar to $2$ . See their Figure 2 for their method with self-consistency feedback. Since $2$ is made public last Nov, I consider this as a concurrent work to this paper.

While the “pseudo feedback from self-consistency” idea in $2$ is concurrent and related to ScPO, we would like to identify the following key differences:

• As shown in Fig 1 (left) and Sec 2 (L 81-99), we show that ScPO can be used to augment the initial seed training data with additional questions sampled from the same model. Furthermore, we show that self-consistency plays a crucial role in filtering for well-formed and answerable questions, which $2$ lacks. The effectiveness of generating additional data can be seen from not only the unsupervised results, but also our results from the semi-supervised setting where ScPO outperforms the supervised IRPO (gold) baseline by up to 2.35% on GSM8K.
• Next, we also incorporate self-consistency in our weighted loss that adjusts the weight of a training instance based on the relative confidence, i.e. difference in vote share of chosen and rejected responses (L141-149). We demonstrate the importance of this weighted training loss in Table 4 of Sec 5. Thus, we argue that $2$ is equivalent to the w(x)=1 baseline (only trained on seed data) which ScPO outperforms.

That being said, we thank you for pointing out this concurrent work and we will cite it in future versions of our paper.

We thank you for your extensive review and comments and are glad to see you appreciate the “creative application of self-consistency”, “impressive results”, and “thoughtful ablations”. Please find our detailed response to your comments below and let us know if you have any follow up questions.

Impact of number of samples (K)

We measure the impact of the number of samples used to measure self-consistency on its Somer’s D correlation with correctness (as done in Table 6) for K=2,4,8,16.

The results indicate that (i) lower values of K (e.g. K=2/4) have lower correlation with correctness or accuracy which we find is because of fewer instances where any answer gets multiple votes; (ii) while larger values of K=16 yield slightly higher correlations, we prioritize computational efficiency in the data generation phase (L100-122), and use a sufficiently large value of K=8 in addition to filtering (L118-121) and a weighted loss (L141-149) for ScPO training.

Computational Overhead of ScPO:

We reiterate that all our baselines including IRPO, LMSI, as well as ScPO have the same computational overhead by design as we use similar size training datasets, same number of samples (K), and the same training hyperparameters. Specifically, to your point on generating training data, we note that this process is done once at the start of each iteration when using the LLM at inference-time, and not during training. Therefore, we can make use of popular strategies for speeding up LLM inference such as the vLLM library, increasing batch-size, and utilizing multiple GPUs in parallel, making the data-generation process far less computationally demanding than the training itself. We will include this discussion in the paper.

Impact of ScPO on model diversity:

In Fig 2 of Sec 5, we visualize the vote share of the model's responses across iterations, and find that it increases across iterations for all datasets, indicating a decrease in the number of unique answers (diversity). We suspect this is a consequence of RLHF training that has been well-documented in prior work (Kirk et al. 2024, Murthy et al. 2024, Murthy et al. 2024) and is outside the scope of our study. At the same time, in Sec 4 (Tables 1-3 and 8-9), we report test accuracy after 8-way self-consistency and find that models trained with ScPO continue to benefit from diversity in generations via SC at test-time.

Number of Iterations:

We refer you to Table 7 in Appendix B where we find that performance on math reasoning largely plateaus after the second iteration, with <1 point gain with a third iteration on training. However, in the same table we find that sampling questions from a different distribution, i.e., the distributions of questions (without using the answers) from the test set and using it to sample additional related problems (L81-98) yields additional improvements in the third iteration on top of the M_2 models (L270-274). Therefore, we believe that increasing the diversity of the seed problems, or after each iteration can be an effective way to delay performance saturation. Also, developing a RLHF method that does not diminish diversity can be a solution.

Open Ended Reasoning Tasks:

We reiterate that ScPO is designed to improve model’s reasoning performance with unsupervised or semi-supervised training. Different from general instruction following tasks such as creative writing with subtle human preferences, on reasoning tasks we desire to prefer correct solutions and disprefer incorrect ones. Nevertheless, in such scenarios, we believe ScPO can be combined with techniques to measure consistency in more generative settings such as universal self-consistency (L432-436) or using executable programs to measure correctness (Lambert et al. 2024). Note that we use a similar programmatic or symbolic approach to measure consistency for the ZebraLogic benchmark where the answer is in a complex “json” format that cannot be directly compared via exact string match and requires a symbolic function to measure equivalent answers and multiple votes. Measuring consistency in more open-ended tasks (e.g. creative writing) is indeed under explored, and likely to be first worked out in inference time uses, which we leave for future work. We will expand on the final paper.

We thank you for your detailed review and questions. Please find our response below your comments:

Significance of Results

Our test sets include >= 1K samples for GSM8K and ZebraLogic, and 5K problems for MATH. In our primary unsupervised paradigm with greedy decoding, ScPO consistently outperforms the IRPO (RM) and LMSI baselines across three datasets and two base models (Tables 1-3, 8-9). The unsupervised baselines exhibit high variance across different datasets and models. For example, while LMSI is the second-best unsupervised method behind ScPO (by 7.2%) with Llama-3 8B in Table 1, it performs significantly worse with Llama-3.1 8B on the same GSM8K dataset in Table 8, trailing ScPO by 11.83% and IRPO RM by 7.6%. Additionally, IRPO RM is the least effective on GSM8K and ZebraLogic with Llama-3 8B. Given that all methods have similar computational and data budgets (see Sec 4), this strongly supports the effectiveness of ScPO for reasoning tasks.

Configuration of Baselines

We conduct zero-shot evaluation to assess the improvement in the model's reasoning abilities from zero-shot instruction training. As shown in Tables 1-3, 8-9, using 8-way self-consistency (SC) on the test set, SC improves the performance of all baselines. However, models trained with ScPO still achieve the highest performance (after SC) in both supervised and semi-supervised settings across two datasets and two model families. Therefore, we expect similar results for other inference-time variants, such as weighted SC or Best-of-N sampling.

Vote-margin and Correctness

We reiterate that the intuition behind self-consistency is that model errors are generally random, making it unlikely to repeat the same incorrect answer across different samples (L 28-36). This concept, effective in various domains and predating its use in LLMs (e.g., RANSAC by Fischler & Bolles, 1981), is supported by popular LLMs showing improved accuracy with majority voting in reasoning tasks (e.g., e.g. DeepSeekMath, Gemini1.5). Empirically, our results (Appendix A, Table 6) demonstrate a strong correlation between consistency and correctness across datasets, indicating LLMs are not widely over-confident in math and logical reasoning tasks. This aligns with prior findings that LLMs are well-calibrated (Kadavath et al. 2022) and that self-consistency reliably estimates model confidence (Xiong et al. 2023, Kabra et al. 2023). In domains with prevalent overconfidence, self-consistency could be combined with inference-time calibration techniques (Zhao et al. 2021).

Number of Iterations

Refer to Table 7 in Appendix B, where we observe that math reasoning performance largely plateaus after the second iteration, with less than a 1-point gain in the third iteration. However, the same table shows that sampling questions from a different distribution—using the distributions of questions (without answers) from the test set to sample additional related problems (L81-98)—yields further improvements in the third iteration on top of the M2 models (L270-274).

Sensitivity to Threshold τ

As noted in L352-378 and Table 5, the initial threshold is based on the training data quality (Margin: Acc(preferred) - Acc(rejected)) and the number of instances meeting the cutoff, i.e., Vote(preferred) >= τ. Even at lower thresholds, Table 5 shows we can improve the base model's performance. While the training data quality and sample size depend on the LLM's inherent consistency for a specific domain/dataset, we applied the same method to train the Llama-3.1 8B model in Appendix C and achieved significant gains without tuning hyperparameters for the new base model.

Dealing with Ambiguous Answers

We reiterate that ScPO is designed to improve model’s reasoning performance with unsupervised or semi-supervised training. Different from general instruction following tasks such as creative writing with subtle human preferences, on reasoning tasks we desire to prefer correct solutions and disprefer incorrect ones, making the domain relatively less ambiguous (please let us know if you have any specific dataset in mind). Nevertheless, ScPO can be combined with techniques to measure consistency in generative settings, such as universal self-consistency (L432-436) or executable programs for correctness (Lambert et al. 2024). For example, in the ZebraLogic benchmark, we use a programmatic approach to measure consistency, where answers in complex "json" format require a symbolic function for comparison and multiple votes.

We hope our response has addressed all of your questions and will allow you to revisit your score. We are happy to answer any followup questions and requests you may have.