Reward Learning From Preference With Ties

1. Poor Layout and Presentation: The formatting of figures (e.g., Figure 1, Figure 2) and tables (e.g., Table 2, Table 3, Table 4, Table 5) is suboptimal, often appearing overly large and occupying excessive space. This layout issue detracts from the readability and professional appearance of the paper, suggesting a rushed submission. Enhancing the visual presentation and ensuring that figures and tables are appropriately sized would significantly improve the reader's experience.

2. Lack of Baseline Comparisons: While the paper introduces a novel method, it fails to compare its performance against established baselines. Specifically, the proposed method is described as a variant of DPO with an offset (ODPO) (Amini et al., 2024), but no direct comparisons are provided. This omission makes it difficult to assess the effectiveness and reliability of the new approach. Including such comparisons would strengthen the paper's claims and provide more concrete evidence of its contributions.

3. Limited Dataset Diversity: The experiments rely solely on the HH-RLHF dataset, which limits the generalizability of the findings. Using a more diverse set of datasets would help validate the robustness and broader applicability of the proposed method. Expanding the experimental scope to include additional datasets would provide a more comprehensive evaluation.

Importance of the problem:

• While incorporating ties in preference modeling is conceptually important, questions arise about its practical significance. If ties occur infrequently in real annotations, as noted in lines 457-459 ("We observe that Anthropic's HH-RLHF dataset contains over 160k samples, with only a small portion labeled as ties"), the significance of this issue may be limited.
• Moreover, how does the significance of ties change with data scaling? Would the impact of ties diminish as the dataset size increases?

Questions about experiments:

• How are the parameter of the BTT model $\theta$ determined?
• While the experiments were conducted on synthetic preference datasets, would it be possible to do experiments using publicly available real-annotated datasets?

1. Method: This paper based on the problem setting of the existence of ties, but the problem setting needs to relabel the preference data for the debias method, which is not suitable for most existing public preference datasets. For this problem, I think the author should focus on the noisy label of the preference dataset, rather than relabeling the data and make some analysis.
2. Theory: The theoretical analysis is based on the ground truth label of ties, rather than the noisy label modeling with the ties. Thus, the superiority of the theoretical results are not meaningful.
3. Experiments: The experimental evaluation results are not rational. The main simulation dataset construction is not clear enough, the evaluation on the HH dataset is also not convincing due to the lack of the ties samples. Additionally the evaluation metrics should include the commonly used benchmark, such as alpaca eval, mt bench and arena hard etc. Also the authors should investigate some robust preference learning methods as their baselines, such as [1][2].

References

[1] Robust Preference Optimization for Large Language Models

[2] Robust Preference Optimization through Reward Model Distillation

I do not see the necessity to introduce algorithm to deal with ties from Table 1. As stated in Lines 56-58, the reward models are trained with BT assumption. And Table 1 shows that these reward models could not distinguish the ties in the datasets due to 0 preference strength. However, the chosen and rejected responses in Table 1 are both suitable as the preferred response. The reward models of BT assign similar rewards to the chosen and rejected responses, which means the BT reward model could already recognize ties without explicitly introducing BTT. Therefore, the necessity of introducing BTT should be further explained.

The experiment is not sufficient enough. Only one dataset (HH) is used with only one baseline model (DPO). Preference optimization has been explored for more than a year. There are several datasets that could be used to evaluate the performance of preference optimization. Moreover, more recent advances in preference optimization, especially those considering noise in datasets, should be reviewed and be integrated as baselines.