Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only

Q1: Improvement in math and reasoning is less evident.

We appreciate the reviewer’s observation. In our experiments, the SFT policy is trained on the TULU-v2-mix dataset, which contains limited coverage of math and multi-step reasoning tasks. Consequently, the SFT policy exhibits weaker performance in these domains, and since SRPPO derives its reward signal directly from the SFT model, this limitation naturally affects the effectiveness of downstream performance.

That said, SRPPO still demonstrates consistent gains over the SFT baseline even in math and reasoning tasks—despite no access to preference data or task-specific supervision. We believe this highlights the robustness of our method. Moreover, we expect that incorporating more math-focused demonstrations during the SFT stage would further improve the coherent reward and lead to stronger gains on math and reasoning benchmarks, which we plan to explore in future.

Q2: Experiments are limited to small models.

PPO-type methods require hosting multiple models, including actor, reference, and critic models, which consumes a lot of memory and computation. Therefore, Llama-8B is the largest model we can afford given our limited computational resources. Nevertheless, we expect our result to hold for larger models, which can be examined with more resources.

Q3: The authors do not experiment with a systematic variation in the amount of SFT data to establish the data requirements for SRPPO.

We appreciate the reviewer’s point and agree that understanding how SRPPO responds to different SFT data regimes is important. To systematically investigate this, we define three experimental setups in Section 4.1: (1) Minimum overlap, where SFT demonstrations and PPO prompts are drawn from disjoint sources. (2) Medium overlap, where a small subset of PPO prompts are included in the SFT data. (3) Diminished overlap, where we introduce additional training data to reduce overfitting to overlapping prompts.

These setups allow us to examine how different levels of prompt overlap affect the coherence and generalization of the reward, and ultimately, the effectiveness of SRPPO. As shown in Tables 1–3, SRPPO consistently improves performance even under minimal overlap, demonstrating that the coherent reward generalizes beyond the original demonstration data, enabling scalable alignment without requiring additional annotations.

Q4: Typos in line 77, 170-173, 196, 236, and missing related work https://arxiv.org/abs/2406.00888.

Thank you for catching these typos and mentioning related work. We will fix them in the revision.

Q5: Why is an SFT model mid-aligned?

Recent RLHF practices suggest that SFT is not enough for getting a highly aligned model. While SFT can make the model follow some instructions, the generalization is largely limited by the scarce gold responses. Therefore, a typical framework is to leverage human preference data and apply offline (e.g. DPO) or online policy optimization methods (e.g. PPO) after SFT for better alignment. Thus, we call an SFT model mid-aligned as it only performs the first half of the pipeline.

Q1: Can you rewrite this sentence such that it's clearer what "off-policy" means here?

We will clarify that “SFT, being an off-policy method, learns solely from expert-provided demonstrations rather than adapting to its own generated outputs.”

Q2: I'm a bit doubtful about the baselines, specifically there's no other true baseline than a DPO-based approach. A comparison to "Self-Rewarding Language Models" would be helpful.

We appreciate the reviewer’s suggestion and would like to clarify the scope and baseline selection in our work. As described in Section 1, the standard LLM alignment pipeline includes: (1) Supervised fine-tuning (SFT) with high-quality (prompt, response) demonstrations, and (2) RLHF with human preference data to train a reward model and further fine-tune the policy.

Our work focuses exclusively on stage (1)—alignment from demonstrations without any preference annotations. Accordingly, our primary baseline is SFT trained on the same demonstration data. Additionally, we include SPIN as a representative DPO-style baseline that also aligns from demonstrations and does not require preference data, making it a directly comparable method.

To broaden context, we also include a PPO baseline with a simple reward model (Fsfairx-LLAMA3-RM) in Table 2. While stronger reward models (e.g., UltraRM, DeepSeek-RM) could improve RLHF results, our aim is to show that SRPPO—despite not using preference data—can already match or outperform these preference-based methods under the same annotation budget.

Regarding the “Self-Rewarding Language Models (SRLM)” paper, we thank the reviewer for the suggestion. SRLM proposes a contrastive learning setup built on pairwise comparisons generated from a model’s own outputs. In contrast, our approach derives a reward signal from the log-likelihood ratio between the SFT and pretrained models, which aligns more directly with demonstration-based supervision and is compatible with smaller models.

Moreover, SRLM is demonstrated primarily on 70B-scale models in open-ended generation tasks, and requires substantially more compute and sampling. SRPPO, by contrast, is lightweight, scalable to smaller models, and optimized for efficiency in the low-data regime. We will clarify these distinctions in the revision and add discussion of SRLM in the related work section.

Q1: A stronger SFT baseline would continue the SFT model (initial policy for SRPPO RL finetuning) with demonstrations where the prompt is used by SRPPO stage for all three data overlap settings.

We appreciate the reviewer’s suggestion and agree that aligning the prompt distributions across baselines can offer a more controlled comparison. However, we would like to clarify a core motivation behind our method. In practice, prompts without paired responses are far easier to obtain at scale, while collecting high-quality demonstrations (i.e., expert-written responses) is non-trivial and costly. Methods like SFT and SPIN require human-annotated responses to train effectively on new prompts, which incurs significant labor and annotation overhead. In contrast, a key contribution of SRPPO is that it can leverage additional prompts without requiring any new ground-truth responses. By deriving a coherent reward from the SFT policy itself, SRPPO enables on-policy fine-tuning solely from unlabeled prompts, effectively amplifying the utility of a small, fixed set of high-quality demonstrations.

In our current experimental setup, all methods are trained using the same amount of human-annotated responses to ensure a fair comparison in terms of supervision cost. Introducing additional labeled responses for SFT or SPIN to match the prompt distribution used by SRPPO would result in unequal supervision budgets, making the comparison less meaningful. We agree it is meaningful to assess whether SRPPO’s improvement stems from broader prompt coverage or the algorithm itself. To clarify, it is precisely our algorithmic design that allows SRPPO to effectively utilize broader prompt distributions—without requiring additional annotations—to enhance LLM alignment performance.

Q2: The reward model baseline experiment can be further improved.

We appreciate the reviewer’s suggestion and would like to clarify the scope and intent of our work. As outlined in Section 1, the standard LLM alignment pipeline typically consists of two stages: (1) Supervised fine-tuning (SFT) using demonstration data—i.e., high-quality (prompt, response) pairs annotated by experts; and (2) RLHF via preference learning, which requires separate human-annotated comparison data to train a reward model and fine-tune the policy through reinforcement learning.

In this paper, we focus exclusively on the first stage—alignment from demonstrations without preference annotations. Consequently, we do not use any human-labeled preference data, and RLHF methods that rely on such data fall outside the scope of our paper. Our primary baseline is therefore SFT using the same demonstration data. Nevertheless, in Table 2, we included results from RLHF with a basic preference-based reward model (Fsfairx-LLAMA3-RM) as an additional point of reference. While more advanced reward models (e.g., UltraRM, DeepSeek-RM) could indeed yield higher RLHF performance, our intention was to illustrate that SRPPO, despite using no preference data, can already match or outperform basic RLHF pipelines.

Q3: The proposed coherent reward (log p_SFT / p_PT) seems to be only meaningful when the SFT-ed policy is noticeably better than the base model.

We agree with the reviewer that the effectiveness of the coherent reward depends on the quality of the SFT policy. Intuitively, if the SFT policy fails to meaningfully improve over the pretrained baseline—e.g., due to low-quality or misaligned demonstration data—then the resulting reward signal may be weak and unable to provide meaningful signal. Indeed, the quality of SFT plays a crucial role in establishing a useful alignment direction, which SRPPO builds upon through on-policy fine-tuning. Ensuring that the SFT model captures meaningful improvements over the base model is therefore essential for the coherent reward to provide reliable training signals during the PPO stage.

Q1: Why could the proposed method be better than PPO with a preference reward model?

Thank you for the insightful question. One of the key advantages of our proposed method lies in the coherent reward formulation, which is derived directly from the SFT policy rather than a separately trained reward model. it is inherently sensitive to variations in the model’s own responses $y\sim p_{\theta}$ during on-policy RL fine-tuning. In contrast, PPO with an external preference reward model often depends on human-labeled comparison data, which can be noisy or inconsistent. Moreover, such reward models may fail to generalize well beyond the preference distribution they were trained on—especially when applied to specific tasks or unseen domains. As shown in Table 2, SRPPO can match or outperform PPO with a basic public reward model, despite requiring less human supervision.

Q2: How would the quality of demonstration data influence the performance of SRPPO?

Since the coherent reward in SRPPO is derived from the SFT policy, the quality and coverage of the demonstration data used for SFT are critical to the effectiveness of the reward and downstream fine-tuning. A well-trained SFT policy—grounded in high-quality, diverse demonstrations—provides a more meaningful and stable reward signal for on-policy learning.

Empirically, we observe that greater overlap between SFT training data and PPO prompts leads to more robust reward evaluations and improved performance. To systematically study this, we design three experimental setups (Section 4.1): (1) Minimum overlap, testing generalization to unseen prompts; (2) Medium overlap, simulating partial prompt coverage; (3) Diminished overlap, where overlapping prompts are diluted by unrelated ones.

As shown in Tables 1–3, SRPPO demonstrates strong generalization even under limited overlap, confirming that it can effectively leverage additional unlabeled prompts to improve performance—without requiring human preferences.

In future revisions, we will include additional ablation studies on demonstration quality, such as: (1) Varying the number and diversity of demonstration samples. (2) Testing on different datasets to evaluate how SRPPO generalizes across domains. (3) Comparing SRPPO performance under synthetic, low-quality demonstrations vs. expert-curated ones.

I have read the response from the authors. While I appreciate their efforts, I think the lack of SPIN results in addition to the other factors means that the paper probably should go through another round of revision before final submission. I will hold my rating.

Q1: There is no apparent baseline with human feedback.

We appreciate the reviewer’s question and would like to clarify the scope of our work. As discussed in Section 1, the alignment process for large language models typically consists of two stages: (1) Supervised fine-tuning (SFT) using demonstration data—i.e., high-quality (prompt, response) pairs labeled by experts; and (2) Reinforcement learning from human feedback (RLHF), which uses human-annotated preference data to train a reward model and further fine-tune the policy.

In this paper, we focus exclusively on the first stage—alignment from demonstrations without preference annotations. Consequently, we do not use any human-labeled preference data, and RLHF methods that rely on such data fall outside the scope of our paper. Our primary baseline is therefore SFT using the same demonstration data. Nevertheless, in Table 2, we included results from RLHF with a basic preference-based reward model (Fsfairx-LLAMA3-RM) as an additional point of reference. While stronger reward models (e.g., UltraRM, DeepSeek-RM) could improve RLHF performance, our goal is to demonstrate that SRPPO—despite relying only on demonstration data—can already match or exceed such basic preference-based approaches.

Q2: The differences from SPIN appear to be minimal. The overlap in approach is substantial.

We appreciate the reviewer’s question and would like to clarify the key distinctions between SRPPO and SPIN. As discussed in Section 1, SPIN relies on the assumption that demonstration responses are always preferred over model-sampled alternatives, and optimizes a DPO-style objective using only labeled (prompt, response) pairs. Consequently, SPIN is restricted to training on demonstration data and cannot make use of unlabeled prompts—a limitation in settings where high-quality responses are scarce and expensive to annotate.

In contrast, our method SRPPO derives a coherent reward function explicitly from SFT and pretrained policies. This self-derived reward serves as a training signal for a range of on-policy reinforcement learning algorithms (e.g., PPO, REINFORCE, RLOO). A core advantage of our approach is that it enables on-policy fine-tuning using additional prompts without requiring new human-labeled responses. This allows SRPPO to generalize alignment beyond the limited coverage of the demonstration data.

While both methods align from demonstrations without preference labels, SRPPO’s ability to perform on-policy sampling and incorporate unlabeled prompts is a fundamental difference. This flexibility is particularly important in practice, where high-quality responses are expensive to annotate, but diverse prompts are easy to collect.

Q3: For some decisions (e.g., reward function) and experiments (the three experimental setups on page 7), it was not clear why these were made or what intuitions we should derive from them.

We appreciate the reviewer’s question and would like to clarify the rationale behind both our reward function and the three experimental setups. As discussed in Q2, a key advantage of SRPPO is its ability to leverage additional unlabeled prompts during fine-tuning—thanks to the coherent reward, which is derived directly from the SFT and pretrained policies. This makes SRPPO especially valuable in settings where high-quality responses are limited, but prompts are abundant.

However, since the reward signal is computed from the SFT policy, the choice of SFT training data directly impacts the effectiveness of generalization to new prompts. Empirically, we observe that greater overlap between SFT demonstrations and PPO prompts leads to stronger and more stable coherent rewards.

To systematically investigate this, we define three experimental setups in Section 4.1: (1) Minimum overlap, where SFT demonstrations and PPO prompts are drawn from disjoint sources. (2) Medium overlap, where a small subset of PPO prompts are included in the SFT data. (3) Diminished overlap, where we introduce additional training data to reduce overfitting to overlapping prompts.

These setups allow us to examine how different levels of prompt overlap affect the coherence and generalization of the reward, and ultimately, the effectiveness of SRPPO. As shown in Tables 1–3, SRPPO consistently improves performance even under minimal overlap, demonstrating that the coherent reward generalizes beyond the original demonstration data, enabling scalable alignment without requiring additional annotations.

Q4: SPIN (a reasonable baseline) is not evaluated across all scenarios.

In the revised version, we will add SPIN results for the medium and diminished overlap settings, using the subset of prompts for which expert responses are available. This will allow for a more complete baseline comparison while respecting the supervision requirements of SPIN.

Q5: Is it possible to release the code.

Our code will be publicly available when we release the paper.