$i$REPO: $i$mplicit Reward Pairwise Difference based Empirical Preference Optimization

My major concern is the contribution of the algorithm and the theoretical analysis listed as below

• In section 4.1, the delivered message, specifically $w_i / (w_i + w_j) = h_{ij} / (h_{ij} + h_{ji})$ is somewhat trivial. The rationale behind highlighting this equivalence is unclear, as it does not seem to directly address any core technical challenges. A stronger connection to the problem statement or contribution would help clarify the authors’ intent here.
• In section 4.2, The proposed method employs the square loss to approximate the binary distribution, defined as L = E[(p - p^)^2]. The authors do not provide a clear reason for selecting this loss function over the more conventional KL divergence, $L = E(\mathrm{KL}(p \parallel p^))$, which is typically a “soft” version of cross-entropy loss. It would be beneficial for the authors to clarify why KL divergence was deemed unsuitable for this application and how the square loss better serves the purpose of approximating the distribution $p$.
• In addition, the algorithm requires repeated evaluations of $h$ for each individual preference pair $(i, j)$. The authors should clarify why this $h$-step data cannot be directly used within DPO but instead must be aggregated into a squared loss function. For fair comparison, given that $h = 5, 9, 15, 21$ were chosen in the appendix, the iterative DPO dataset should ideally leverage these same values ($h = 5, 9, 15, 21$) rather than a single annotator in vanilla iterative DPO. A clarification on this point is necessary.
• In section 4.3, the theoretical justification here requires further detail. For example, Assumption 4.1 cannot hold universally for any $P_h$. The authors may want to define the releasability on the ground truth P^, indicating that P_h \to P^ as $h \to \infty$ (per Assumption 4.2). However, this assumption may be also impractical, as it requires a significantly large $h$. Moreover, the theoretical analysis as presented is insufficient. For example, equation (18) essentially states that if $L = (p - q)^2 = 0$, then $D_{TV}(p, q) = 0$, which is self-evident. Similarly, equation (19) mirrors the DPO analysis. Furthermore, the role of $C_t$ in Theorem 4.5 is not well-explained, especially within the online iterative framework. A more thorough explanation of these theoretical points would strengthen the analysis.
• The authors should explicit show the selection of $h$ in the main paper.

Regarding the proof in the appendix, here're a few issues I'm suggesting to revise.

• When applying the mean-value theorem, the correct expression is $|\sigma(x_1) - \sigma(x_2)| = \Big|\sigma’(x_0)\Big| \times |x_1 - x_2|$ (with absolute value of $|\sigma'(x_0)|$ rather than simply $\sigma’(x_0)|x_1 - x_2|$, though this does not affect the correctness of the proof.
• In line 810, this is a concentration inequality rather than the law of large numbers. The scaling factor of $O(1 / \sqrt{n})$ relies on certain conditions, which should be explicitly verified by the authors (e.g. Hoeffding's inequality with R-sub-Gaussian). The law of large numbers only guarantees convergence to zero but does not provide scale guarantees.

1. The authors do not address the additional computational costs to update the weights $w_i$ in each iteration. The comparison between DPO/IPO and iRepo should take into account this increase in time complexity
2. A fundamental limitation of this method is that it requires multiple annotators for each pair of output. This is not applicable to DPO/IPO.
3. There seems to be a disconnect between the description of DPO and IPO across multiple sections. Until section 3, the authors refer to vanilla DPO/IPO. In the experiments, the authors use iterative DPO/IPO as the baseline without describing the iterative version of the algorithm.
4. It seems like simply increasing the number of annotators does not lead to better alignment. Infact, this seems like a strange hyperparameter to tune. Given the marginal improvement over the baselines, Figure 2(b) needs further explanation/investigation.
5. I would encourage the authors to add stronger baselines apart from DPO/IPO: SPPO, KTO, SimPO etc.

• Correctness.
  • Line 042: online and offline alignment. Online alignment does not necessarily involve RLHF based on PPO. In fact, typical RLHF approaches are instances of offline RL, where a reward model is learned using offline dataset then an RL algorithm learns the policy (LLM) using that learned reward.
• Motivation of Zermelo's Model. Zermelo's algorthm is used to estimate the target regression values of implicit reward (eq16). In my opinion, it's also possible to leverage any reward model (trained with eq2) to provide such regression target. Could you elaborate why Zermelo's model is proposed and what is its advantages?
• Minor issues. Grammar or spelling errors: line 011 (Large), 108 (LLM Alignment), 253, 852.

1. Comparative Analysis: The choice to compare an online method to several offline methods (e.g., DPO, IPO) is questionable and potentially misleading, as offline methods generally face limitations in out-of-distribution scenarios that online approaches can better handle. This mismatch in methodology reduces the validity of the experimental claims.
2. Model Selection: The experiments are conducted only on Phi-2 and Mistral-7B, which are not state-of-the-art models. To demonstrate iREPO’s robustness and generalizability, it would be more insightful to test on current high-performing models like LLaMA 3. Without this, the results are less relevant to ongoing research in language model alignment.
3. Limited Real-World Applicability: Given the reliance on certain assumptions (e.g., no data distribution disparity), the applicability of iREPO in real-world, dynamic settings remains uncertain.

• Currently, the annotation scheme is rather expensive. Collecting a single preference pair requires sampling D completions, making D choose 2 pair and annotating each pair multiple times. After computing the logit corresponding to the strength of the preference, all other completions are thrown away.
• The experimental setup can be improved and made fairer (questions below). First, the main contribution of the paper is a new offline objective for alignment. A more informative experiment would be to compare the method first in a completely offline setting as follows: Take a SFT model, sample D completions per prompt, get preference labels for each pair. Use iREPO to regress onto the strength of preference (with and without filtering down to weakest/strongest pair), but there are many ways to compute ranking here and you could consider putting in all the preferences into DPO/IPO too (as they are implicitly ranking losses).
• More experimental evidence would help, as a lot of these evaluations are fairly noisy.
• In general, the paper needs to justify why expending this additional effort on collecting preference label is justified. The claim can be backed by showing better performance / unit of annotation effort. Or you could show that these preference labels result in higher signal-to-noise ratio for preferences, and can be scaled better.