Selective Preference Optimization via Token-Level Reward Function Estimation

My concerns are listed as follows:

1. My major concern is about the token selection mechanism. The motivation behind using $\hat{r}(s_t,a_t)$ as the proxy of the reward is unclear to me. Theorem 1 only proved that $\sum \hat{r} (s_t, a_t) + V^{*}(s_1) = \sum r(s_t, a_t)$, which only guarantees that the sum of $\hat{r}$ and the sum of $r$ is the same (up to a constant). However, the value distribution of $r$ and $\hat{r}$ might still be drastically different. Therefore, the token selection based on $\hat{r}=\log \pi_{\theta} / \log \pi_{\text{ref}}$ does not make sense given the current illustration.

2. It looks like that that SePO is quite sensitive to the parameter $\gamma$. The search space of $\gamma=\{2.1, ...,2.5\}$ looks weird and it seems that there is a fluctuate of performance when $\gamma$ varies in this set. This makes a issue give that the improvement over baseline is not that significant.

3. (Minor issue): The $\propto$ in equation (6) looks like a typo

• The proof of Theorem 1, which asserts that after training a DPO, the reward function can be expressed as a decoupled reward $\hat{r}$, inherits this property (Line 810) from the assumption that the reward can be written in such a manner (Assumption 1). This raises the question of whether all reward functions can be expressed in a decoupled way. From a naive perspective, a decoupled reward is not normalized, and longer texts might have larger absolute values of reward. In my attempts to learn reward models in online settings using a decoupled approach, I found that without normalization, their accuracy dramatically reduced. Normalizing the sum of small rewards over tokens led to improvements. Therefore, I strongly feel that not all rewards can be expressed in this way. Moreover, the training objective (Equation 11) uses a normalized reward, making it unclear why Theorem 1 was presented.
• Some of the writing is ambiguous. The SePO objective (Equation 11) is hard to parse visually and would benefit from a human-understandable explanation before the equation.
• The experiments lacked exploration of the dependence of performance on the KL divergence with the reference policy. It is evident that training a policy with the SePO objective could cause it to diverge significantly. This is similar to observations in Rafailov et al. [1]. For instance, could selecting a lower $\beta$ value enable DPO or other baselines to perform better than SePO?

[1] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

1. The method is limited by the requirement that oracle and policy models share the same vocabulary and tokenizer, which reduces flexibility across different model architectures.
2. The use of the DPO reward format as an automated credit assignment behaviour has been attempted by other works, and the paper's contribution is weaker as only quantifies the results of this assignment to the weights of the DPO loss.
3. Suppose the confidence given by the Oracle model is used as the gold label for the credit distribution. In that case, we do not need dpo to fit the reward distribution given by the optimal policy (https://arxiv.org/abs/2404.12358, https://arxiv.org/abs/2408.14874). Alternatively, the paper needs to discuss the error problems associated with this approximation in the method to validate the need for SePO further.
4. Performance increases are relatively minor in the experiments, and the comparison model used is GPT-4-0314 (not the current optimal model, but again, not compared to the model untrained itself to provide a more intuitive increase)

The experiments primarily involve relatively moderate-sized models. Testing SePO on stronger models, such as LLaMA2-Chat-70B, would provide further insights into its scalability and potential bottlenecks, especially for the weak-to-strong generalization experiment.

Compared to other methods, the improvement seems to be slight.