Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision

For the ILR framework to be effective, it seems that the SFT model needs to generate responses that are often better than those in the SFT dataset, and the annotator must be able to identify these better responses. While the latter assumption is discussed in the paper, the former depends on the SFT model being sufficiently capable and not overfitted to the SFT data, so that it can generate responses than the SFT data at least some of the time. This depends on factors such as the size of the SFT dataset, the proportion of unreliable data, and the size of the SFT model, raising questions about how generally applicable ILR is.

Thank you for this important point about ILR's prerequisites. However, we note that RLHF methods also rely on a similar assumption—that models can generate some responses better what humans can provide [1]. This is fundamental to the entire paradigm of improving models through feedback rather than direct demonstration.

The phenomenon of models sometimes outperforming their training data, while not yet theoretically guaranteed, is an empirical fact observed across all tasks and model sizes considered in our work (Section 4) as well as recent work [2, 3]. Moreover, Figure 7 provides a concrete example where the SFT model generates better responses than time-constrained human annotators.

[1] Stiennon et al. “Learning to summarize from human feedback.” (2020)

[2] Burns et al. “Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision.” (2023)

[3] Guo et al. “Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Models.” (2024)

How does ILR compare to having annotators review each sample in the SFT data to remove low-quality ones? This approach seems less expensive, as it avoids generating new responses with the SFT model and lets the annotators avoid pairwise comparisons.

We agree that data filtering in general helps with learning under weak supervision. Asking annotators to think twice, review their answers, and rewrite or remove should improve data quality, but it seems orthogonal to our assumption that the final demonstration data we obtain is unreliable. As an evidence, some of ChatGPT’s training data initially rated as “flawless” has only been found in recent work [1] to contain flaws.

In addition, pairwise comparisons are often cognitively easier than absolute quality judgments. For example, while an annotator might struggle to identify a mistake in a mathematical solution simply by proofreading, they may be able to notice the it when a correct solution is provided—in this case the mistake will be revealed by the difference between the two solutions.

[1] McAleese et al. “LLM Critics Help Catch LLM Bugs.” (2024)

How frequently are the model-generated responses considered better than the SFT data depending on the size of the model?

Here's how often model-generated proposals were accepted across rounds in two settings of GSM8K:

Higher acceptance rates in the 2B → 7B setting suggest that there could be more room for improvement when initial demonstrations are more unreliable (as those generated by the 2B model are, compared to those generated by the 7B model).

Additionally, declining acceptance rates across rounds in both settings indicate the increasing difficulty of obtaining further improvements as data quality increases. This could be due to either 1) the evaluator struggling to identify better responses, since those that can be identified are already used or 2) the model is generating less responses that are better than the original demonstrations.

Nonetheless, we acknowledge that a more systematic study controlling for either supervisor or student model size while varying the other (e.g., compare 2B → 7B and 2B → 70B) would be valuable future work.

Typos: “Ground Truth” in Figure 5.

Thank you for catching this typo! We will fix it in our revision.

We thank the reviewer for their detailed and thoughtful comments. We appreciate their recognition that our work addresses a “highly practical problem”. We have addressed their specific concerns below.

Limited evaluation of alternatives. Several approaches exist for handling noisy feedback data, including conservative DPO [1], which combines regular DPO with label smoothing, IPO [2], MMPO [3], and filtering noisy labels, among others. However, empirical evaluations of some of these methods in the unreliable supervision setting considered are missing.

Thank you for highlighting these important baselines. We have conducted additional experiments with conservative DPO (cDPO) [1], IPO [2], and robust DPO (rDPO) [3] on GSM8K (2B → 7B): (we will include these results in the final paper)

These results demonstrate that while these methods may help with random preference noise, they are less effective for handling unreliable supervision in our setting. This could be because 1) in our setting, both task demonstrations used in SFT and comparison feedback used in DPO are unreliable, and 2) noise in demonstrations or comparison feedback are systematic (Section 2) instead of random, unlike the random label flip assumed in some theoretical results [1, 3]. Such systematic errors are particularly challenging because they may not be easily modeled using standard robust learning techniques.

We haven’t included MMPO [4] in our experiments since it is not explicitly designed to address noise and it is not implemented in the HuggingFace framework that our experiment pipeline is based on. We will further explore this method in future work.

[1] Mitchell. "A note on DPO with noisy preferences & relationship to IPO." (2023).

[2] Azar et al. "A general theoretical paradigm to understand learning from human preferences." (2024).

[3] Chowdhury et al. "Provably robust dpo: Aligning language models with noisy feedback." (2024).

[4] Kim et al. “Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback.” (2024).

The concept of unreliability is not quantitatively well-defined. For example, it is unclear whether a small LM trained as a classification model actually provides unreliable comparison feedback, and if so, to what extent the labels are considered inaccurate. Relatedly, it is unclear how unreliable the “time-constrained” human supervision really is.

Thank you for raising this important concern about quantifying unreliability.

Firstly, we can measure unreliability using task accuracy in our math (GSM8K) and coding (BIRD). For example, in GSM8K, unreliability of task demonstrations can be measured by comparing them with ground truth. Figure 5a shows that LM-generated demonstrations for GSM8K (2B → 7B) only has an accuracy of 0.34, which can be considered unreliable.

For comparison feedback, we measure how often the evaluator selects the objectively better answer (GSM8K 2B → 8B):

This declining accuracy over rounds demonstrates the evaluator's increasing difficulty in identifying better responses. Moreover, it is shown in Figure 5 that an unreliable evaluator can not improve the SFT data to a level that can be achieved with an oracle.

For our human studies, we also have multiple indicators of unreliability: 1) Response length: Figure 6 shows time-constrained human responses are significantly shorter than GPT-4 responses; 2) Qualitative analysis: Figure 7 demonstrates clear misconceptions in human responses; 3) No qualification requirements: We recruited participants without domain expertise requirements through CloudResearch Connect.

While measuring unreliability in more subjective tasks like SaferPaca is challenging, our primary goal is to demonstrate that when supervision quality falls below a certain threshold, existing methods break down and alternatives like ILR become necessary.

Nonetheless, we believe that more nuanced understanding of how different level of unreliability affects ILR and RLHF would be an interesting direction that could be explored in future work.

Dear reviewer J4qi,

We have added new experiments and explanations to address your concerns in our rebuttal. As the discussion period is concluding soon, please let us know if there are any additional questions we can address. Thank you!

Best, Authors

Task Diversity: While the paper tests ILR on math, coding, and safe instruction-following, it would be beneficial to see it applied to a broader range of tasks, particularly more complex or open-ended ones where human feedback is highly subjective or unreliable (e.g., creative writing or subjective question-answering or toxicity).

Thank you for this suggestion. We note that the SaferPaca dataset we use already includes subjective tasks about following instructions (which includes question-answering) and rejecting unsafe or toxic prompts. That said, we agree that exploring additional complex domains could further demonstrate the robustness of our results.

Theoretical Depth: The theoretical tracking of ILR is solid, but the paper could elaborate more on the theoretical limits of ILR. In particular, the long-term stability of ILR, and whether it risks introducing new forms of bias through the iterative refinement process, could be explored.

We appreciate the request for deeper theoretical analysis. Our main goal in this work is to provide empirical evidence that current methods may break down under unreliable supervision, and that an alternative method could work. We agree that understanding ILR's theoretical properties is crucial, and we aim to explore it in our future work.

Does ILR introduce any risks of bias through its iterative process, particularly as models continue to generate and refine their own data? How would you address or mitigate such risks?

In practice, ILR would be deployed with human oversight—similar to our time-constrained human study—rather than automatically replacing human labeled data with synthetic data. Our experiments with naive ILR (Section 5.2) show that simply replacing demonstrations with model outputs without supervision leads to poor results, indicating that incorporating human oversight can reduce risks during the iterative process. We believe that these risks could be further mitigated when combining ILR with other scalable oversight techniques, such as decomposing LM-generated responses [1] or using other LM to discover flaws in the synthetic data [2].

[1] Wen et al. “Learning Task Decomposition to Assist Humans in Competitive Programming.” (2024)

[2] McAleese et al. “LLM Critics Help Catch LLM Bugs.” (2024)

Thank you authors for your detailed response + additional experiments. I have increased my score and advocate for acceptance!

We thank the reviewer for their insightful review. We appreciate that they found our approach is “fresh”, “thoroughly tested and clearly explained”, and “encourages researchers to rethink more about how to utilize human feedback in post-training.” We have responded to their individual points below.

Potential ensembling effect that may explain performance gain: 5.1 describes training one model on each half of the SFT data, then using these models to cross-label during ILR. There does not seem to be an ablation either. I think a fair comparison would be against a two-model ensemble, as opposed to a single model baseline.

Thank you for raising this important concern. We did test a version of DPO that uses two models trained on disjoint data subsets to generate proposals in our preliminary ablation studies. Results on GSM8K (2B → 7B) show this method (denoted as sDPO) provides minimal benefit:

We conjecture that even with potentially better proposals from two models, sDPO still struggles due to similar reasons to standard DPO. It suffers from 1) difficulty correcting errors learned during initial SFT during preference optimization and 2) overoptimization issues caused by unreliable feedback, as discussed in Section 4.1.

Limited Comparison with other RLHF Methods: While the paper focuses on comparing ILR to DPO, it does not address how ILR might perform compared to other RLHF algorithms, like PPO. Including more comparisons could make the paper’s conclusions more robust and introduce a broader impact.

Thank you for highlighting the need for broader comparisons. We've evaluated several recent algorithms designed to handle noisy preferences on GSM8K (2B → 7B) and will include these comparisons in the final paper. As shown below, they all perform similar to DPO and underperform ILR:

Our work is focused on DPO-based methods due to their stability and computational efficiency, which enabled our thorough experiments using limited compute. Also, the above results suggest that preference optimization methods face similar challenges under unreliable supervision. Nonetheless, we agree that extending these findings to PPO would be valuable future work.

[1] Mitchell. "A note on DPO with noisy preferences & relationship to IPO." (2023).

[2] Azar et al. "A general theoretical paradigm to understand learning from human preferences." (2024).

[3] Chowdhury et al. "Provably robust dpo: Aligning language models with noisy feedback." (2024).

Computational Cost and Scalability: The ILR approach, which requires iterative refinement and cross-evaluation of datasets, which means both SFT model and DPO needs to be retrained for every iteration. This might introduce higher computational complexity compared to DPO or traditional RLHF methods. The paper could benefit from a more detailed discussion of the trade-offs between performance improvement and computational cost.

Thank you for raising this practical consideration. We’d like to clarify a few points:

1. ILR does not involve DPO training—it only requires SFT in every iteration.
2. ILR trains one SFT model plus two additional half-data models, requiring about 2N forward passes where N is the number of training samples.
3. DPO only trains one model in each round, but it uses both a chosen and a rejected response for each prompt; moreover, it requires computing logits of the reference model. Therefore, it requires about 3N forward passes, which is actually more inefficient than ILR.
4. Nonetheless, compared to cost of pretraining, the difference in the costs of ILR and DPO should be almost negligible.

We thank the reviewer for their thoughtful and review and for recognizing that “the introduced method is simple and performs clearly better.” We have addressed their points below.

Doesn't compare against PPO

Thank you for raising this important point about comparisons to PPO. While we focused on DPO since its stability and computational efficiency allowed us to conduct more thorough experiments, we have conducted additional experiments with several recent robust preference optimization methods on GSM8K (2B → 7B), which we will also include in the final paper:

All of these preference optimization methods show similar patterns under unreliable supervision. We agree that validating this with PPO experiments would strengthen our conclusions and plan to include this in future work.

[1] Mitchell. "A note on DPO with noisy preferences & relationship to IPO." (2023).

[2] Azar et al. "A general theoretical paradigm to understand learning from human preferences." (2024).

[3] Chowdhury et al. "Provably robust dpo: Aligning language models with noisy feedback." (2024).

Is it specifically necessary to do 1:1 replacement of training data points with new synthetic ones? Does it make sense to decouple the number of data points removed due to low quality, and the number of new synthetic data points accepted? How much of the effectiveness is due to the filtering vs the addition of synthetic data points?

Thank you for this insightful question about ILR's refinement process. We conducted two ablation studies to understand the relative importance of removing low-quality examples versus adding synthetic ones. In the first ablation, we simply remove original labels rejected by the evaluator; in the second, we add chosen new proposals without removing old labels.

Surprisingly, our preliminary experiments on GSM8K (2B → 7B) show that:

1. Removing low quality labels identified during ILR provides most of ILR's benefits, achieving similar performance to full ILR.
2. Simply adding new data without removing old labels cannot recover full ILR’s performance.

We hypothesize that proposals generated by the 7B model may not be high-quality enough to provide benefits beyond what we get from removing bad demonstrations. Also, pure removal strategies could be risky for tasks requiring coverage of diverse inputs such as pluralistic alignment [1]; the balance of removing and adding would help maintain both quality and coverage of the training data.

To investigate these hypotheses, future work can conduct the ablations on larger models that might generate higher-quality synthetic data, and analyze ILR’s performance on tasks that require broad coverage of diverse inputs.

[1] Sorensen et al. "A roadmap to pluralistic alignment." (2024).

We thank the reviewer for their insightful review and suggestions. We are glad they found that “the experimental setup is good” and our human experiments and analysis of DPO are “interesting”. We have responded to their individual points below.

Missing Baseline: it would be good to compare against just doing SFT on the chosen sample or unreliable DPO pairs.

Thank you for suggesting this important baseline. We tested this in our preliminary ablation study, but we found that it led to immediate performance degradation in the fist round. On GSM8K (2B → 7B), after just 1 round of SFT on chosen samples generated by the initial SFT model, accuracy decreases from 0.3556 to 0.3497 (averaged over 3 seeds). This highlights a key insight behind ILR's design: it's crucial to maintain the full dataset to include originally high-quality demonstrations while replacing low-quality ones with new ”chosen” proposals, because simply training on “chosen” samples may lose valuable information present in the original dataset.

I wonder if the comparison between DPO and SFT/ILR methods are entirely fair since the DPO method doesn’t use the original ground truth answer, just two model generations. Could the DPO results perform better if the original ground truth label was also provided as an option when picking chosen and rejected pairs (as it is for the ILR approach)?

Thank you for this important question. To clarify: neither ILR nor DPO use ground truth labels in our experiments—both methods rely solely on unreliable supervision. We did explore a variant of DPO (which we call wsDPO) that constructs preference pairs by comparing model proposals against original demonstrations rather than only comparing between model proposals. While this seemed to slightly improve over standard DPO, it still underperformed ILR. For example, in GSM8K (2B → 7B):

This underperformance could be due to errors learned during the initial SFT stage being hard to correct through preference optimization, similar to DPO. Also, wsDPO still suffers from the overoptimization issues discussed in Section 4.1 under unreliable feedback.

Would it be possible to run the same ILR algorithm outlined in section 5.1, but rather than replace labels, just use the accepted/rejected proposals to construct chosen/rejected pairs instead?

Thank you for this suggestion. We intentionally designed ILR to avoid using chosen/rejected pairs for preference optimization, as our analysis in Section 4.1 showed that preference optimization methods tend to suffer from overoptimization under unreliable supervision. However, we agree that finding ways to penalize poor responses could be valuable in addition to making use of good model-generated proposals. In future work, we plan to explore algorithms that can leverage both positive and negative feedback while remaining robust to unreliable supervision.

Figure 6b is quite hard to parse, as the label box overlaps with a large chunk of the figure, and the differences between round 1 and 2 are somewhat small.

Thank you for pointing out this problem! We will adjust the figure to make it clearer in our final paper.

Did the authors try longer DPO training with a high beta? They clearly show that using a larger beta with unreliable feedback can improve a bit, I wonder if training for longer with higher beta results in further improvements with DPO or if it then starts to over-optimize as well.

As shown in Figure 3a, even with 4 rounds of training, high-beta DPO shows minimal improvement when using partially unreliable feedback (50% unreliable) and even less with fully unreliable feedback. While longer training with high beta might yield marginal gains, this comes at significant computational cost. Furthermore, new preference comparisons need to be collected for each round of DPO, so it would be expensive in terms of annotator time. More importantly, the fundamental limitation that strong regularization constrains model updates still persist, making it difficult to correct errors learned during the initial SFT stage. ILR avoids this trade-off by improving the training data itself rather than attempting to optimize unreliable preferences directly, and it does not require longer training time.

I was a bit unclear on how the unreliable DPO dataset is constructed: so, you generate responses from a model checkpoint, and train a classifier to pick the original ground truth label instead of the model response, and then use this classifier to pick between two samples from the same model checkpoint? Or is the classifier the same across rounds?

Yes, the classifier is the same across rounds and it is the same as the classifier in ILR. This ensures both methods operate under identical unreliable supervision conditions. More training details of this classifier can be found in Appendix B.