Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis

We thank the reviewer for the constructive and helpful feedback, and for recognizing the value of our work. Below we address your concerns in detail.

W1: Clarification of the source of improvement (MLP or high-quality synthesized data?)

We thank the reviewer for raising this important point. We have included the details of "textual synthesis" in Section 5.1 and Appendix A.1, and are happy to elaborate on this. For each prompt in the seed set, we sample multiple responses. A reward model then ranks these responses, and we select the top (preferred) and bottom (non‑preferred) to form a pair.

To disentangle whether our performance improvements stem from the quality of synthesized preference data or from the MLP reward mapping phase, we have conducted explicit ablations in Table 1. In particular, the baseline "Embedding MLP (textual)" uses preference pairs generated in the textual space, while fixing the feature layer and only retraining the MLP with the new samples. The only difference between this baseline and our method lies in the source of synthesized data. As shown in Table 1, our approach significantly outperforms this baseline (e.g., 1.94 vs. 1.62 reward score under 4× augmentation), which demonstrates that the gains primarily come from the higher-quality synthesized samples rather than from the MLP.

We further examined this question by ablating simpler latent synthesis methods in Table 1: (1) Direct Noise Perturbation, where Gaussian noise is directly added to embeddings, and (2) Gaussian Sampling, where embeddings are sampled from a multivariate Gaussian fit. Both variants showed almost no improvement compared to the baseline, indicating that our VAE-based synthesis is crucial for producing semantically meaningful and diverse samples. Together, these ablations confirm that the performance improvements in Table 1 are indeed driven by the high quality of latent-synthesized preference data.

W2/Q2: Additional baselines that generate textual augmentations in the original text space

We agree that it is important to compare against strong textual synthesis baselines. In fact, our experiments in Table 1 already include three representative text-space augmentation methods. These methods differ in how the preference data is generated, rather than focusing on the reward modeling approach itself.

• Self-rewarding [10]: Prompt the model itself as a judge to rank pairwise responses generated in the text space.
• Self-evolved [11]: Relies on reward-based ranking of diverse responses, also performed directly in the text space.
• IPO [12]: Determines preference rankings by comparing the likelihoods of generated responses, another text-based augmentation strategy.

These baselines involve generating multiple textual responses for each prompt and applying ranking or selection strategies to construct preference pairs. As shown in Table 1, our latent-space synthesis method consistently outperforms these text-space augmentation baselines, demonstrating that our approach is more effective despite operating entirely in the embedding space, which is far more computationally efficient.

W3: comparison on the RewardBench.

Thank you for your suggestion! We have evaluated the learned reward model on RewardBench. Our reward model trained with latent-space synthesis achieves competitive accuracy on the RewardBench safety subset (the goal of our alignment training) compared to models trained on textual augmentations, demonstrating that the performance gains extend to out-of-distribution preference tasks. We will include these results in the revised version. The reported results are based on Llama-3-8B on HH-RLHF with 4x augmentation.

W4: online training experiments with reward model.

While PPO would be an ideal downstream evaluation, its training cost is prohibitively high for large models. Instead, we adopt rejection sampling as a cost-efficient alternative to approximate online training. As demonstrated in Table 3, the reward model trained with our latent synthesis notably improves SFT outcomes and achieves a 61% win rate in GPT-4 evaluations, compared to 39% for text-based baselines. Prior work (e.g., [1]) has shown that rejection sampling can approximate the benefits of PPO-style updates, making our setup both practical and representative.

[1] Gao et al., Scaling laws for reward model overoptimization, ICML 2023.

Q1. Does the "original" column in Table 1 represent results using only the subsampled 1,000 data points, while the 2x~8x columns indicate performance with progressively larger augmented datasets generated by the proposed method or simply sampling more responses in the text space directly?

Yes, the "Original" column corresponds to training the reward model using only the 1,000 seed preference pairs subsampled from the HH dataset. The subsequent 2×–8× columns show performance when the dataset is augmented to larger sizes, starting from the same 1,000 seed pairs. Importantly, each row in the table corresponds to a different augmentation method: textual synthesis, simple latent variants (e.g., Gaussian sampling), and our proposed latent-space synthesis.

Q2. The comparison methods appear to be reward modelling approaches rather than textual synthesis techniques. Could the authors explain how these baselines are directly comparable to the proposed data augmentation approach?

We realize that the naming of some methods in Table 1 may have caused confusion. To clarify, all baselines in Table 1 are indeed data augmentation methods for reward modeling, not alternative reward model architectures. Please also see our response in W2 for details.

Q3: Possibility of mapping embeddings to texts

We thank the reviewer for this interesting question. In principle, one could attempt to decode synthesized embeddings back into text and then apply reward modeling in the textual space. However, this would require training or fine-tuning a high-capacity decoder that can faithfully map embeddings to fluent and semantically consistent text. This is a highly non-trivial problem by itself, since LLM embeddings are not guaranteed to be invertible to unique text sequences, and imperfect decoding could easily introduce artifacts or distort the preference signal.

In contrast, our approach avoids these pitfalls by operating directly in embedding space, where the synthesized representations can be seamlessly integrated into reward model training. This design choice is both efficient (no need for text decoding) and theoretically grounded: as shown in Theorem 1, our latent-space synthesis preserves preference ordering with high probability. That said, mapping embeddings back into text is an intriguing direction for interpretability and inspection, and we view this as a promising avenue for future work, complementary to the efficiency-focused contribution of our current paper.

We thank the reviewer for the constructive and helpful feedback, and for recognizing the potential of our work. Below we address your concerns in detail.

W1: How was the textual preference data generated?

We have included the details in Section 5.1 and Appendix A.1, and are happy to elaborate further on this. For each prompt in the seed set, we sample multiple responses. A reward model based on Llama-3.1-8B trained on the training split of HH-RLHF or TL;DR ranks these responses, and we select the top (preferred) and bottom (non‑preferred) to form a pair and random pair multiple times for augmentation. We deliberately avoid the Skywork RM to prevent leakage of its 'gold' preferences into the training loop, thereby preserving the integrity of downstream evaluation.

W2: Generation to not only obtain a single new pair but potentially 3 new ones.

We thank the reviewer for this excellent suggestion! Our current implementation indeed aligns with your idea. To clarify, when generating a new pair of latent-space preference data, we effectively obtain three additional pairs by compositionally combining each synthetic pair with the original preference data. This results in up to a 4× increase in training pairs per original triplet. This strategy is captured in Equation (8):

$\mathcal{E}_{\text{aug}} = \lbrace (\tilde{e}^+, \tilde{e}^-) \mid \tilde{e}^+ \in \mathcal{E}^+ \cup \mathcal{E_synth^{+}} \; \tilde{e}^- \in \mathcal{E}^{-} \cup \mathcal{E_synth^-} \rbrace.$

This compositional pairing not only increases the diversity of training signals but also amplifies the impact of each synthesized sample. We appreciate the reviewer’s idea and will include a clarifying remark in the final version to make this more explicit.

W3: Ablations on temperature for the textual preference data.

The temperature we used for the textual response generation is 1.0. As suggested, we conducted additional ablation for your reference. We report the results based on Llama-3-8B on HH-RLHF with 4x augmentation. The experiments are based on HH-RLHF with default augmentation scale 4×. Below we report the experimental results for different temperatures {0.6,1.0,1.2} for the textual synthesis compared to ours.

We find that temperature 1.0 can yield relatively strong performance for textual synthesis, and our latent space synthesis consistently outperforms all tested temperatures.

W4: Standard errors on the baseline method.

We absolutely agree that reporting standard errors improves evaluation rigor. We have conducted multiple runs for the strongest baseline (full fine-tune with text-based synthesis) in Table 1 and include the results below, demonstrating that the improvement of our method is statistically significant. In the final version, we will report standard errors for baselines with textual space synthesis.

For the Llama-3-8B on HH-RLHF,

For the Llama-3-8B on TL;DR,

W5: Comparison with active learning methods.

We thank the reviewer for bringing up active learning, which is indeed a relevant and important direction in the limited data setting. Our method addresses a complementary challenge: rather than selecting new queries for human annotation (as in active learning), we focus on augmenting an existing set of preference-labeled data without requiring any additional human labeling---as you correctly recognized.

To clarify, we do not claim that our method outperforms baselines trained with human-labeled data. Instead, all compared methods—including ours—start from the same set of existing preferences, and differ in how they augment this base dataset. There is no human involved in the augmentation phase for all baselines. While prior work typically augments in text space (e.g., generating more responses and labeling them with a reward model), our method performs augmentation directly in embedding space, which is more efficient and yields better downstream generalization.

Though direct comparison is not fair or strictly applicable, we do believe that exploring hybrid strategies—e.g., using active learning to select high-impact prompts for latent synthesis—is a promising direction for future work, and we will add this discussion and cite relevant works as you mentioned to the paper.

W6: What stage does the method taper off?

Great question! We provide experimental results training the reward model with varying training set sizes, including sizes larger than 50K, as shown below. Even when using the full training set (~100K samples)—the largest size we were able to test—our method still outperforms textual space synthesis by a significant margin, achieving an absolute reward gain of 0.4. This non-trivial improvement indicates that our method has not yet saturated at this scale. We will amend this result in our manuscript - thanks again for the helpful comment!

We thank the reviewer for the constructive and helpful feedback, and for recognizing the value of our work. Below we address your concerns in detail.

W1/Q1: lack of interpretability and quality control

You are raising a very thoughtful comment here. We fully agree that understanding and evaluating the quality of positive and negative examples is essential, and interpretability provides a valuable lens for this.

That said, we believe interpretability—while important—is not the only route to ensuring quality. Our method takes a more principled and rigorous approach by providing theoretical guarantees on the quality of synthesized preference pairs. Specifically, our main result (Theorem 1) shows that under mild regularity assumptions, latent-space perturbations preserve preference ordering with high probability, and the resulting synthesized pairs maintain semantic consistency with the original labeled data. This provides a provable basis for quality assurance that complements interpretability-driven techniques.

Moreover, we empirically validate the quality of the generated embeddings along multiple axes—e.g., improved reward model performance (Table 1), generalization across different model families and tasks (Table 1 and 4), and latent space quality via t-SNE visualization (Fig. 3). These evaluations offer evidence that our synthesized embeddings are meaningful, even without decoding them into text.

We do acknowledge that interpretability of latent embeddings remains an open challenge, and we are excited about possible extensions. For instance, techniques such as [1] may help project synthesized embeddings back into text space for inspection. We view our method as complementary to such interpretability tools, and incorporating them is a promising direction for future work.

[1] Belrose et al., Eliciting Latent Predictions from Transformers with the Tuned Lens, arXiv preprint 2303.08112

W2: The need for retraining the VAE with other prompts.

We agree with the reviewer that our latent-space synthesis method is based on a model trained on static embeddings, and adapting to substantially different prompts or new preference data may require retraining the VAE. This is a valid limitation, and we have explicitly acknowledged it in our Limitations section (Appendix C) with full transparency.

That said, we would like to emphasize that retraining the VAE is computationally very lightweight and fast—our model has only 0.5M parameters and can be trained efficiently even on modest hardware. In contrast, adapting text-space synthesis methods typically requires rerunning large-scale generation and annotation with LLMs, which is far more resource-intensive.

We will clarify this trade-off more explicitly in the final version and appreciate the reviewer’s thoughtful feedback.

Thanks for your feedback. I agree with your response to the interpretability problem that I mentioned:

That said, we believe interpretability—while important—is not the only route to ensuring quality

I'd like to maintain my score (borderline accept).

We thank the reviewer for the constructive and helpful feedback, and for recognizing the value of our work. Below we address your concerns in detail.

Q1: The choice of embedding layer

This is a great question! We use the last-layer embeddings for synthesis because the reward model predictions are directly based on this layer. To verify this choice, we conducted ablation studies on different embedding layers, and the last-layer embeddings consistently yielded the best generation performance. We report the results based on Llama-3-8B on HH-RLHF with 4x augmentation.

Q2: Additional details on how the top-k synthetic latent codes are selected

The selection of the top-k synthetic latent embedding works as follows:

1. Generate candidates: For each original latent vector (representing a preferred or non-preferred response), several candidate latent codes are created by adding Gaussian noise.
2. Compute likelihood scores: Each candidate latent code is evaluated based on how likely it is under the VAE’s learned Gaussian distribution. This is done by checking how close the candidate is to the mean of the distribution, with smaller deviations resulting in higher likelihood scores.
3. Select top-k: From all the candidates, the $k$ codes with the highest likelihood scores are chosen. This ensures that only the most plausible synthetic latent codes (those that are close to the high-density regions of the learned latent space) are kept and later decoded into synthetic embeddings.

We will revise our manuscript to make this process clearer - thanks for pointing this out!

Q3: At what point does the performance begin to plateau for the augmentation scale increases?

We have conducted ablation studies on varying augmentation scales and observed that performance improvements gradually taper off beyond the tested augmentation rates. We report the results based on Llama-3-8B on HH-RLHF with different augmentation rates. Additional experiments with even higher augmentation scales, as large as 32x, are the point where performance saturates.

Q4: At what model size does your method remain beneficial?

We extended our experiments to larger models, including Qwen/Qwen2.5-14B. Our method continues to provide significant benefits at this scale. Due to computational constraints, we were unable to conduct experiments on models larger than 70B. We report the results based on Llama-3-8B on HH-RLHF with 4x augmentation.