Self Iterative Label Refinement via Robust Unlabeled Learning

Thanks for the valuable feedback. We appreciate your suggestions and will use them to improve the manuscript. Our responses follow.

---

Weakness 1: Restriction to binary classification

Thank you for this critical point.

Because our primary goal was to study LLM self‑refinement, we first concentrated on a binary‑task to anchor the iterative framework.

Could the method be extended beyond binary classification to improve its applicability to more practical, real-world tasks?

Regarding your question, the answer is yes.

• Our framework is not inherently limited to the binary case.
• It can be generalized to multi-class problems via methods like complementary-label learning [1], whose assumptions closely match those of UU-learning.
• We will revise our conclusion to discuss the extension to multi-class classification as a promising future direction.

[1] https://dl.acm.org/doi/10.5555/3295222.3295315

---

Weakness 2: Performance on the Fake News Dataset

Thank you for highlighting the one setting in which our method lags behind CCP.

In this experiment Gemma‑2.2B starts from exceptionally noisy pseudo‑labels: Fig. 2 reports just 59.1 % test accuracy, also 58.7 % for train dataset. When label noise is this extreme, Robust‑UU may yield only limited gains.

Noise alone, however, does not preclude improvement. For example, starting at 57.3 % accuracy on Protein‑Structure(GPT‑4o) and 57.6 % on Sarcos (Llama‑3.2‑1B), our pipeline still climbs to ≥ 80 % and > 90 %, respectively. These results illustrate that performance can rise from a low initial accuracy, confirming that the Fake‑News/Gemma scenario is an isolated anomaly rather than a systematic weakness.

Indeed, this run is the sole outlier among 21 dataset–model combinations; in the remaining 20 settings our approach outperforms every baseline, underscoring its broad applicability.

To probe this anomaly, we will analyze embeddings of correctly and incorrectly classified samples to pinpoint issues specific to Gemma/Fake‑News. Furthermore, upon acceptance, we will release the initial pseudo‑labels to facilitate replication and deeper investigation by future researchers.

---

Weakness 3: Comparison with State-of-the-Art SSL Methods

Thank you for this valuable suggestion.

• Could you suggest any specific SSL papers you consider essential? Your recommendations will help ensure that the camera-ready version cites the most relevant work.
• In the revision, we will broaden the related-work section to include these papers and other modern SSL approaches, thereby better contextualising our contribution.

---

Question: Fully Supervised Baseline

I would also expect the inclusion of a fully supervised baseline using true labels as a strong point of reference.

A fully supervised baseline (train-on-true-labels) already appears in Appendix B.2, Table 2. We will surface it in the main text and reference it prominently to make comparisons clearer, alongside the figure revisions.

---

Figure Clarity:

Figure 2 and 3 will be revised for readability, and we will add tables in the appendix that present the precise numerical data plotted in each figure.

---

Once again, we thank you for your time and constructive comments. We are confident that by incorporating these changes, we can significantly improve the clarity and impact of our paper.

We appreciate your careful assessment and constructive feedback. Below we respond to each of your concerns and clarify the novelty, scope, and robustness of our work.

---

Weakness 1: Limited novelty

As other reviewers (e.g., Lj7L, e1xr) have remarked, our approach is innovative and novel in this context, and NeurIPS reviewing guidelines explicitly acknowledge that a novel combination of existing techniques is a valuable contribution.

Our novel contributions are:

• First application to LLM self‑refinement. We are the first to apply theoretically grounded UU learning to the critical task of LLM self‑refinement, opening a new avenue for performance gains where existin methods that rely on an LLM’s internal knowledge have struggled. Figures 2 and 3 demonstrate consistent per‑iteration improvements.
• Bridging theory and practice. By embedding principled weak‑supervision theory in a modern alignment pipeline, we provide an effective tool that boosts performance benchmarks (Figs. 2–3) and directly benefits RLHF (Fig. 4).

---

Weakness 2: Restriction to binary classification

You correctly observe that our empirical validation focuses on binary classification.

Because our goal was to study LLM self‑refinement, we first concentrated on a binary‑task to anchor the iterative framework.

However, our framework is not inherently limited to the binary case. It can be generalized to multi‑class problems via a method like complementary‑label learning [1], whose assumptions closely match those of UU‑learning. We will update our manuscript to reflect this promising future direction.

[1] https://globals.ieice.org/en_transactions/information/10.1587/transinf.2021EDP7035/_p

---

Weakness 3: Dependence on initial pseudo‑labels

Robustness to noisy pseudo‑labels is central to our design, and both theoretically and empirically confirm it.

• Theory. UU risk remains unbiased as long as the pseudo‑positive split’s true‑positive rate exceeds that of the pseudo‑negative split, even when the initial accuracy is near 0.5
• Practice. Starting from extremely noisy labels (Protein‑Structure 57.3 %, Sarcos 57.6 %), our method reaches ≥ 0.80 and > 0.90 within five iterations, while GPT‑4o, DeepSeek‑R1, PIE, and CCP plateau or decline (Figs. 2–3).
• Low‑knowledge regimes. Because our pipeline relies on data-centric feature extraction rather than internal model knowledge, it keeps improving precisely even when there is a lack of internal knowledge, addressing the your concern about positive ratios between splits are similar.

---

Weakness 4: Broader applicability

Our work's implications extend to a wide range of LLM applications, including generative tasks.

• Proven effectiveness in a generative task. Section 5.4 and Fig. 4 demonstrate that our UU‑trained reward model steers RLHF toward safer outputs, significantly outperforming a baseline.
• Classification underpins modern LLM alignment. Most training paradigms hinge on binary decisions [2]:
  • RLHF/DPO: Decide which of two responses is preferred. Our pipeline produces these preference labels with minimal human input, even for subjective or domain‑specific traits.
  • GRPO: Relies on feedback labeling answers as correct or incorrect. UU refinement supplies such labels even when the ground truth is unclear.
  • Generative guidance. High-accuracy classifiers, such as those learned via our UU refinement, are essential for guiding generative models toward desired properties (e.g., protein design) [3].

[2] https://dl.acm.org/doi/10.5555/3692070.3694015
[3] https://neurips.cc/virtual/2023/poster/71899

---

We trust our responses address your concerns. Our method for LLM self-refinement is novel and valuable. Thank you for your time and feedback.

Thank you for your thoughtful reassessment and constructive engagement. We appreciate the chance to further clarify the theoretical foundations of our approach.

---

The central idea of Unlabeled-Unlabeled (UU) learning is to construct an unbiased estimator of the supervised classification risk, $R_{pn}(g)$, such that its expected value accurately represents the true classification risk.

In standard supervised learning, a classifier $g$ is trained by minimizing the risk, calculated from a positive dataset $C_p$ and a negative dataset $C_p$: $R_{pn}(g) = \pi_+ E_{x \sim C_p}[\ell(g(x), +1)] + (1-\pi_+) E_{x \sim C_n}[\ell(g(x), -1)]$ where $\pi_+$ is the positive class prior, and $\ell$ is the loss function.

In contrast, UU learning utilizes two unlabeled datasets: a pseudo-positive dataset $\tilde{C}_p$ and a pseudo-negative dataset $\tilde{C}_n$, each containing true-positive proportions $\tilde{\theta}_p$ and $\tilde{\theta}_n$, respectively. Note that standard supervised learning corresponds to the special case $\tilde{\theta}_p = 1$ and $\tilde{\theta}_n = 0$.

The UU risk is then defined as: $R_{uu}(g) = a E_{x \sim \tilde{C_p}}[\ell(g(x), +1)] - b E_{x \sim \tilde{C_p}}[\ell(g(x), -1)] - c E_{x \sim \tilde{C_n}}[\ell(g(x), +1)] + d E_{x \sim \tilde{C_n}}[\ell(g(x), -1)]$ where $a, b, c, d$ are coefficients that are determined by the $\tilde{\theta}_p$ and $\tilde{\theta}_n$ (see details in lines 136-137).

Critically, $R_{uu}$ remains unbiased as an estimator of $R_{pn}$ when $\tilde{\theta}_p > \tilde{\theta}_n$ (i.e., pseudo‑positive split's true‑positive rate exceeds that of the pseudo‑negative split), enabling classifier learning without clean labeled data.

This theoretical foundation has been established and empirically validated in existing theoretical research [1, 2]. The theoretical guarantee is provided in Theorem 4 of the original paper [1].

[1] Lu et al., 2019, "On the Minimal Supervision for Training Any Binary Classifier from Only Unlabeled Data," ICLR
[2] Lu et al., 2020, "Mitigating Overfitting in Supervised Classification from Two Unlabeled Datasets: A Consistent Risk Correction Approach," AISTATS

---

Our novel contribution is the first to conceptualize LLM judgments as weakly supervised signals and to integrate the theoretically grounded UU learning into LLM self-refinement and modern alignment pipelines. This integration represents a substantial advancement compared to previous methods reliant exclusively on internal LLM knowledge.

---

Update for Camera-Ready Version: We will enhance the Preliminaries section with an expanded discussion of UU learning theory to clearly articulate these theoretical foundations and their practical implications for LLM self-refinement and alignment.

We appreciate the thoughtful evaluation. Below, we address every point raised.

---

Weakness: Significance of Technical Contribution

Thank you for raising this point. As other reviewers (e.g., Lj7L, e1xr) have remarked, our approach is innovative and novel in this context, and NeurIPS reviewing guidelines explicitly acknowledge that a novel combination of existing techniques is a valuable contribution.

Our novel contributions are:

• Novel self‑refinement direction: We are the first to apply theoretically grounded UU learning to the critical task of LLM self‑refinement, opening a new avenue for performance gains where existin methods that rely on an LLM’s internal knowledge have struggled. Figures 2 and 3 demonstrate consistent per‑iteration improvements.
• Demonstrated practical utility in LLM post‑training: We further show effectiveness in LLM post‑training (RLHF) for generative safety alignment (Fig. 4), indicating that our method offers practical value beyond classification tasks.

---

Question 1: Performance Stabilizing After One Iteration

Thank you for your clarification.

As you observed, the largest gain occurs in the first iteration because the initial noisy labels allow robust UU to correct the biggest errors. Nevertheless, Figures 2 and 3 reveal continued performance gains in later rounds, particularly when the base model is weaker or the dataset is highly noisy (e.g., Protein‑Structure). These incremental improvements confirm that iterative refinement continues to provide value beyond the initial correction.

To make this trend unmistakable, we will add a per‑iteration accuracy table in the appendix, clearly quantifying the cumulative gains.

---

Question 2: Alternative Approaches to Improve Label Quality

We thank you for this question and apologize if this was not clear in the text.

We did, in fact, compare our method with strong baselines described in Sec 5.2 ("Baselines") and Table 1, covering state-of-the-art label‑refinement techniques from related fields:

• PIE [65]: An iterative, noise-robust method from the field of weakly-supervised learning.
• CCP [25]: A method using contrastive learning for label refinement from the field of semi-supervised learning.
• Self‑Refinement with GPT‑4o / DeepSeek‑R1: Uses the LLM’s internal knowledge to iteratively critique, relabel, and improve its outputs.

Our experiments (Figure 2, 3) show that our proposed pipeline consistently outperforms these state-of-the-art methods, particularly on more challenging datasets.

---

We believe these clarifications resolve the your concerns. Given the method’s novelty, empirical strength, and practical relevance, we kindly ask you to consider raising the recommendation.

Thank you again for your careful review.

Thank you for your constructive feedback and for noting the novelty of our approach. We address your points below.

---

W1:  Requirement for Labeled Data

Our method indeed uses a small labeled set, but this requirement is minimal compared with existing techniques.

• Tiny supervision, large gains. With just 50–100 labels, our model achieve iterative performance gain (Figures 2–3), whereas strong baselines plateau, or even decline, with the same supervision. Conventional fully‑supervised pipelines need thousands of labels; our method therefore slashes annotation cost by 10–1000×.
• Value in expert‑scarce domains. Patent and protein tasks, where annotation is costly, still benefit markedly, demonstrating practicality in low‑resource settings.
• Why other methods fail. Prior self-refinement approaches cannot transform additional labels into meaningful signals. Table A shows accuracy stays flat or even drops even when adding 100 in-context examples with LLM's self-refinement. Robust UU, by contrast, does leverage those labels effectively.

---

W2  Confirmation‑Bias Risk

We share your concern and purposely design our method to mitigate it:

• External, data‑driven correction. The LLM’s potentially biased knowledge is used only once to generate the initial pseudo‑labels. From iteration 1, learning is purely data‑driven: a discriminative classifier trained with robust UU risk gradually removes residual noise in those initial labels.
• Empirical evidence. Accuracy rises monotonically across all datasets (Figures 2–3), even when initial pseudo‑labels are noisy (e.g., ≤ 60 % on Protein).
• Planned mitigations. We believe bias-reduction techniques can be integrated into our method and would welcome relevant literature to guide our camera-ready revision and future work.

---

Questions

Do the baselines also use the same amount of labeled data?

We conducted a fair comparison about the supervision as follows.

• CCP (Semi-Supervised): Employs the same 100 labeled examples as our method used, following the CCP protocol.
• PIE (Weakly-Supervised): We add the same 100 labeled examples to PIE’s training data, alongside original pseudo-labels.
• LLM Self-Refinement Frameworks: These rely solely on in‑context learning. Because longer input often degrade accuracy [1,2], we used 10 few‑shot examples in the main runs. To answer your question, we re-ran with 100 examples using gpt-4o-mini; as Table A shows, accuracy is nearly unchanged from the 10-example case (see Figure 3) and flat accuracy across iterations.

Table A: Self‑Refinement Accuracy (GPT‑4o‑mini, 100 in‑context examples)

This setup provides each baseline with consistent supervision, matching its paradigm and ensuring a fair comparison. Also, Table A indicates that existing self‑refinement struggles to extract useful signals from additional labels, while our UU‑based approach overcomes this limitation.

[1] https://aclanthology.org/2025.acl-long.1396/\ [2] https://aclanthology.org/2023.findings-emnlp.745/

---

Notation and Clarity of Figure

We are grateful for these concrete suggestions for improving the paper's presentation.

• Notation: We will revise the notation for pseudo-positive and pseudo-negative corpora in the camera-ready version to avoid confusion, as suggested.
• Figures: Plots will be revised for the camera-ready version to use more distinguishable, consistent colors and markers across all figures. For clarity, we'll add tables with the exact numerical data from each figure.

---

We appreciate your thoughtful review and valuable feedback. We look forward to addressing your concerns in our camera-ready submission.