Progressively Label Enhancement for Large Language Model Alignment

Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. Regarding your concerns, I would like to provide the following explanations:

If I understand correctly, the paper seems to assume that $\pi^\star$ is $\pi _\theta(y|x,principles)$. If that's the case, then why bother designing such a complicated algorithm, instead of just doing context distillation through SFT (basically just use $\pi _\theta(y|x,principles)$ as target when just providing $x$? If you want to claim that PLE has better sample efficiency or has better performance, then context distillation should be the SFT baseline which makes much more sense comparing to using offline data for SFT.

Thank you for your thoughtful question. We would like to clarify that $\pi^\star$ and $\pi _\theta(y \mid x, \text{principles})$ are fundamentally different in our framework. As defined in Equation (2),

$\pi^\star = \arg\max _\pi \mathbb{E} _{x \sim p(x),\ y \sim \pi(\cdot \mid x)} [R(x, y)],$

$\pi^\star$ represents the optimal policy that maximizes expected reward. In contrast, $\pi _\theta(y \mid x, \text{principles})$ is only a component used within the PLE algorithm to generate a reference response under principle-guided prompting.

Our goal is not to imitate the principle-guided response directly via context distillation, but rather to progressively guide the base model $\pi _\theta(y \mid x)$ toward $\pi^\star$ by comparing its outputs to those generated under the principle prompt, and selectively applying ranking or weighted learning based on reward differences. As we show in Theorem 5.3, this progressive strategy ensures that $\pi _\theta(y \mid x)$ converges toward $\pi^\star$ under certain conditions.

According to the descriptions of the experiment setup, it is not clear what reward model is being used during model training? Do training and evaluation use the same reward models?

For the HH dataset, we used RM-Gemma-2B as the reward model during both training. For the IMDb dataset, we trained a sentiment classifier using the 0/1 labels provided in the dataset. The reward score is defined as the logit of the positive class predicted by this classifier. For the TL;DR dataset, we trained a reward model based on the preference pairs from the tldr-preference-trl-style dataset, which was then used consistently during both training and evaluation. In all cases, the same reward model is used throughout training and evaluation to avoid distribution mismatch.

Using BLEU and PPL as evaluation metrics for model alignment does not make sense to me either. Those two metrics are focusing on the similarity to references on a surface level which is not the goal of alignment. In addition, the references are already quite outdated, SOTA aligned models are expected to exhibit quite large differences to the references.

We agree that BLEU and perplexity (PPL), while widely used, do not fully capture the goals of alignment. Additionally, our evaluation does not rely solely on BLEU and PPL. We additionally report:

​ • Reward model scores to directly reflect alignment with learned human preference signals,

​ • Human evaluation based on qualitative assessments of response helpfulness and harmlessness,

​ • Evaluations using a strong LLM (Claude API) to provide an automated yet high-quality comparative judgment.

These complementary metrics together offer a more comprehensive view of alignment performance. As shown in our results (Tables 1–3 and Figure 2), our method consistently outperforms baselines across these diverse evaluation settings.

Due to space limitations, other responses can be found at the anonymous link https://anonymous.4open.science/r/ICML_rebuttal_PLE-1F6E/responses_2.md

Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. Regarding your concerns, I would like to provide the following explanations:

The proofs lack intuitive explanations for key parameters, leaving their practical impact unclear.

Thank you for your valuable feedback. We would like to clarify that all key parameters and symbols used in the theoretical analysis are formally defined and explained in Sections 4 and 5 of the paper. If there are any specific parameters whose meaning remains unclear, we would be happy to provide further clarification or improve the exposition accordingly.

Figure 1’s current design fails to clearly illustrate the interplay between principle-guided responses, thresholding, and training phases; a redesigned figure with step-wise visuals would improve understanding.

Thank you for your helpful suggestion. We will improve the framework diagram in the next version to provide a clearer representation of the overall process.

Equation (2) defines the optimal policy as $\pi^* = \arg\max _{\pi} \mathbb{E} _{x,y\sim\pi}[R(x,y)]$, omitting the KL divergence regularization term commonly used in RLHF (e.g., $-\beta \mathbb{D} _{\text{KL}}(\pi \| \pi _{\text{SFT}})$) to prevent excessive deviation from the initial policy. Could the authors clarify why KL regularization is absent in their theoretical framework?

Thank you for pointing this out. Equation (2) presents a general formulation of the alignment objective, which captures the goal of maximizing expected reward. Common approaches such as PPO introduce a KL divergence regularization term to prevent the learned policy from deviating too far from the reference policy. Such regularized objectives can still be encompassed within our formulation by considering the reward function $R(x, y)$ to implicitly include regularization terms. This perspective has also been adopted in prior work [1].

[1] FUNDAMENTAL LIMITATIONS OF ALIGNMENT IN LARGE LANGUAGE MODELS, ICML2024

In Table 1 (HH dataset), the PPO-aligned model shows higher perplexity and lower reward scores compared to SFT. What might explain this? A discussion on the failure modes of PPO in specific tasks (e.g., multi-turn dialogue) would strengthen the paper’s critique of existing methods.

To ensure a fair comparison across methods, we adopted an offline training setup for PPO, using the same set of queries from the HH dataset. This constraint limits PPO’s ability to fully explore and optimize responses beyond the fixed dataset, which may explain the relatively modest performance gains compared to SFT. Similar observations have been reported in prior work [1].

[1] KTO: Model Alignment as Prospect Theoretic Optimization, ICML 2024

We hope that our revisions have addressed all of your concerns, but please let us know if there is anything else we can do to improve the manuscript. We would be happy to answer any additional questions or provide any further information you may need.

Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. We would be happy to answer any additional questions or provide any further information you may need.

Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. Regarding your concerns, I would like to provide the following explanations:

The proposed approach PLE shares considerable similarity with prior alignment methods, raising concerns about novelty. In particular, RAFT (Dong et al., 2023) already “expands the SFT dataset by generating additional samples and selecting those with high reward scores”. PLE’s core idea – generate model responses and utilize reward scores to decide how to train on them – is a direct generalization of RAFT. The main difference is that instead of discarding low-scoring samples entirely (as RAFT does), PLE continues to use them with reduced weight or as negative examples. This is a fairly incremental change: it addresses data utilization inefficiency noted in RAFT but does not introduce a fundamentally new alignment paradigm. In essence, PLE combines RAFT’s data generation with a ranking loss akin to RRHF (Yuan et al., 2023), which “encourages the model to generate preferred responses with higher probability and poor responses with lower probability” by integrating a preference-based regularization. PLE’s ranking loss when $s_prompt−s>\tau$ serves the same role as RRHF’s regularizer, and its weighted fine-tuning when the difference is small is reminiscent of label smoothing techniques.

we would like to emphasize that the core motivation and overall framework of PLE are fundamentally different. As stated in our paper, many existing methods treat data generation and model training as separate, static processes, which leads to suboptimal use of generated data. In contrast, PLE explicitly couples these two stages through a dynamic and adaptive training strategy, where the model’s behavior and reward feedback guide the evolving use of generated responses. This synergy between data generation and adaptive training is central to PLE’s novelty. Therefore, rather than being an incremental extension of RAFT or a hybrid with RRHF, PLE proposes a new unified framework that systematically addresses the inefficiency in prior alignment approaches.

The idea of using a set of principles to guide a model’s own generated responses for further fine-tuning has been explored in recent “self-alignment” or “constitutional AI” approaches (e.g., Sun et al., 2023). The authors do cite Sun et al. (2023) as motivation for designing the principle prompt, but PLE does not substantially go beyond those ideas – it uses principles in a straightforward way (simply prepending a fixed set of rules to the query during generation). Other works (like Constitutional AI by Bai et al., 2022b) used principles to generate feedback or to iteratively refine outputs, which is arguably a more novel use of model-generated data. PLE’s use of a principle-guided prompt is comparatively basic and could be seen as a minor variation on these existing alignment techniques.

As we mentioned above, the core contribution of our work lies not in the specific design of the principle-guided prompt itself, but in the overall framework that couples data generation and model training in a dynamic and synergistic manner.

The principle-guided prompt serves as one component within this framework—its role is to produce alternative responses that reflect desirable alignment properties, which are then evaluated and integrated into the training pipeline using our adaptive strategy. While our current implementation adopts a simple prompting mechanism, we do not claim novelty in the prompt design itself.

In fact, one of the strengths of our method is that it is modular and compatible with more sophisticated principle-guided strategies, such as feedback-based refinement or multi-turn rewriting used in works like Constitutional AI. We chose a straightforward setup to demonstrate the effectiveness of our approach even under minimal settings. We appreciate the suggestion and will explore integrating more advanced principle-guided generation techniques in future work to further enhance the overall performance.

Due to space limitations, other responses can be found at the anonymous link https://anonymous.4open.science/r/ICML_rebuttal_PLE-1F6E/responses_1.md