We want to thank you for reviewing our paper and your strong endorsement of our work.

Thank you for your detailed review and questions. We hope that our response will satisfy your concerns.

1. FaRMA Efficiency vs DPO

DPO is more expensive to train because it loads and makes calls to an additional reference model, $\pi_{ref}$, along with the model which is trained, $\pi_\theta$. FaRMA does not require access to $\pi_{ref}$. This also increases the memory footprint of DPO.

We compared the training time of PPO, DPO and FaRMA, on the TLDR dataset, on the llama3.2-1B model, on a single A100 GPU (to ensure fair comparison). As shown in the table above, FaRMA trains three times faster compared to DPO.

We also looked at the peak memory usage per training batch for each of these methods. Again FaRMA has a much lower memory footprint.

2. GPT-4 Evaluation

The evaluation shows the win-rate of baselines vs FaRMA, i.e. DPO vs FaRMA, CARDS vs FaRMA, etc. This comparison is standard in literature [1,2,3]. We average this result over a hundred prompts and use the same hundred prompts for all comparisons. These prompts are randomly sampled, initially, and then fixed for all evaluations. All the comparisons on the graph, including win-rate and inference time, are with respect to FaRMa. The blue point for FaRMA is at the $50\%$ mark on the y-axis for reference. We will add a horizontal line on the plot for an easier comparison.

1. Khanov, Maxim, Jirayu Burapacheep, and Yixuan Li. "ARGS: Alignment as Reward-Guided Search." ICLR 2024
2. Li, Bolian, et al. "Cascade Reward Sampling for Efficient Decoding-Time Alignment." ICML 2024 Next Generation of AI Safety Workshop.
3. Rashid, Ahmad, et al. "A Critical Look At Tokenwise Reward-Guided Text Generation." ICML 2024 Workshop on Foundation Models in the Wild.

3. Diversity result for CD and PARGS

HH Dialogue Dataset

TLDR Dataset

We present additional diversity experiments on CD and PARGS and observe that FaRMA still produces the most diverse text. The temperature used for all diversity tests is 1.

4. Additional References

We thank the reviewer for pointing us to these works. We will add them to our paper. The three suggested papers [1,2,3] train Generative Adversarial networks for dialogue generation. They employ either the policy gradient method [1,2] or PPO [3] to train the generator, and train the discriminator to provide rewards. To mitigate the problem of sparse rewards they employ methods of training step-wise Q-functions.

However, we would like to clarify that the discussion in the abstract and introduction of our paper is in the context of RLHF, specifically reward guided text generation, that aligns LLMs to preference data at inference. Whereas the aforementioned works explicitly apply RL techniques to train text generators, RGTG methods avoid the use of off-line RL and instead employ reward guided decoding.

1. Tuan, Yi-Lin, and Hung-Yi Lee. "Improving conditional sequence generative adversarial networks by stepwise evaluation." 2019 IEEE Transactions
2. Li, Jiwei, et al. "Adversarial Learning for Neural Dialogue Generation." EMNLP 2017
3. Tuan, Yi-Lin, et al. "Proximal policy optimization and its dynamic version for sequence generation." arXiv preprint(2018).

5. Optimal Sequence

In FaRMA the search for an optimal sequence is done during training of the value function V. While we could do a search for an optimal sequence during decoding, this is not desirable since this would increase decoding time. Note that the search for an optimal sequence is implicitly achieved by the loss function in Eq. 17. This loss function ensures that by the end of training the value of a partial sequence corresponds to the max of the values of all continuations. This loss function is similar to the temporal difference loss function in traditional RL to estimate the value of the best plan going forward. Hence at decoding time, the LLM does not need to search for an optimal continuation and does not worry about average continuations since the resulting policy is trained to select tokens that are greedy with respect to the value function, which already accounts for the best continuation.

6. Typos and Corrections

Thank you for pointing these out. We will correct them in the final version.

Thank you for taking the time to review our paper and numbering your comments.

W1:Performance Advantage on larger models

We would like to point out the result on the 7 billion model (Table 5 Appendix) is on a different dataset, Ultra Feedback, and the absolute reward values on different datasets cannot be compared to each other. Additionally we observe that FaRMA on average performs better than the baselines including DPO. Therefore, based on the result we can not conclude that the advantage of FaRMa is diminished on larger models.

Additionally we evaluate this setting using GPT-4 and provide the win-tie rate of FaRMA against DPO and ARGS for this experiment below:

We can observe that FaRMA has a higher winning rate and does well in this setting.

W2: Smaller Reward Model

Please note that even at the same reward model size FaRMA has a computational advantage over DPO and PPO. Please refer to point 1 of the rebuttal for Reviewer wvoK. We note that FaRMA at the same model size trains $3\times$ faster. Therefore a smaller reward model would trade-off the performance against even more training efficiency.

We also note that the smaller reward model is significantly better than $\pi_{ref}$ and comparable to RGTG baselines (at half the reward model size).

W3: Tokenwise Optimization and Optimality

The best way to understand FaRMA is from a reinforcement learning (RL) perspective. In RL, we train a policy to select actions that greedily maximize cumulative future rewards captured by the state-action value function. Similarly, FaRMA chooses the next token greedily to maximize the value function. The fact that FaRMA does not do a look-ahead search during decoding is consistent with RL since RL training ensures that the resulting policy should not need a look-ahead search during execution. The look-ahead search is implicitly done during training (Eq. 17) by the estimation of the value function that captures cumulative rewards for all future steps. Furthermore, the loss function in Eq. 17 naturally accounts for compounding of errors since the value of a partial sequence is trained with respect to the value of its continuations, including any error.

W4: $\beta$ Ablation for FaRMA

We present an ablation on changing the value of $\beta$ for FaRMA on the TLDR dataset. A $\beta$ value of 1.5 is optimal in this case. We want to point out that the inference time is independent of $\beta$ for FaRMA. However, for CARDS the hyper-parameter controls the trade-off between higher reward and higher inference time. Therefore, we presented the result in the appendix to justify the threshold that we chose for CARDS.

W5: Computational Cost

Our approach does not increase the training time since, in practice, instead of training over the entire sequence we sample some steps from the sequence. Note that this is standard in many ML algorithms, e.g. diffusion models [1].

The sampling is done to keep the training set comparable to the preference dataset for full-sequence reward learning. Note that on Table 9 of the Appendix, on the same GPUs, the training time for ARGS (full sequence reward model) and Ours is comparable.

Furthermore, your suggestion of matching the reward of a partial sequence to a longer trajectory instead of just the next token is an interesting variation. However, an exponential search in terms of vocabulary size is required to find such a trajectory with maximum reward, and this would lead to significantly higher training cost.

1. Bishop, Christopher M., and Hugh Bishop. Deep learning: Foundations and concepts. Springer Nature, 2023. Algorithm 20.1

W6 and W7: Typos and Corrections

Thanks for pointing these out. We will correct these in our paper.