While the method is reasonable and useful, the reviewer identified several limitations in the manuscript:

• Novelty: The proposed method primarily aligns with existing offline RL literature [1], with its unique technical contribution being the estimation of the normalization factor. However, there is no ablation study assessing the impact of these design choices on performance. Additionally, similar offline RL techniques have been utilized for LLM alignment, such as [2]. A comprehensive comparison with related work in LLM alignment is needed to highlight the true novelty of this study.

• Contribution: Although the paper emphasizes the low cost of the proposed method, this has already been achieved by several other studies, e.g., [3]. Furthermore, several works have accomplished training-free alignment [2][4], suggesting that the contribution of this study may not be as significant.

• Writing: The overall writing lacks clarity. Sections 2.2 and 3 include large portions of well-known derivations and theorems that could be moved to the appendix to allow for greater emphasis on the study's unique contributions. Additionally, two algorithms are presented, but only the latter one is used. Deleting the first or moving it to the appendix would be beneficial. There is also a typo: $Q_{\phi}$ in Algorithm 2 is not utilized.

• Experiment Comparison: (1) The comparison is limited, with several relevant iterative fine-tuning works, such as iterative DPO and Remax [3], and SPIN [5], not included. (2) Prior iterative fine-tuning methods tend to decline after about three iterations. Why does the proposed method sustain performance through seven iterations? Are there any critical design factors?

• Ablation Study: The paper lacks a comprehensive ablation study, leaving readers uncertain about the effects of the proposed method's design choices.

References

[1] Peng X B, Kumar A, Zhang G, et al. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning[J]. arXiv preprint arXiv:1910.00177, 2019.

[2] Snell C, Kostrikov I, Su Y, et al. Offline rl for natural language generation with implicit language q learning[J]. arXiv preprint arXiv:2206.11871, 2022.

[3] Li Z, Xu T, Zhang Y, et al. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models[C]//Forty-first International Conference on Machine Learning. 2023.

[4] Mudgal S, Lee J, Ganapathy H, et al. Controlled decoding from language models[J]. arXiv preprint arXiv:2310.17022, 2023.

[5] Chen Z, Deng Y, Yuan H, et al. Self-play fine-tuning converts weak language models to strong language models[J]. arXiv preprint arXiv:2401.01335, 2024.

(Missing comparisons)

Overall, the comparisons in this paper seem incomplete:

1. Since this paper is using a method similar to RWR/AWR, comparing with RWR/AWR [4] would be important.
2. The reward model used for this task seems not super strong. Why not use GPT-4 as a judge, or one of the stronger reward models in 7B-8B parameter range from RewardBench? (https://huggingface.co/spaces/allenai/reward-bench)
3. Where are the other baselines in Figure 3?
4. If the loading of the value model/memory consumption is an issue, and the solution is to sample multiple responses, then a more appropriate comparison instead of PPO is GRPO [7]. The paper is missing this.
5. More datasets. TL;DR is well-known to be not a comprehensive dataset anymore. Experiments on some higher quality dataset, such as Ultrafeedback would make the paper stronger.
6. The paper says it is beating other baselines, but PPO still seems to be much better for most of the training in Figure 2.

(Further concerns)

The paper mentions DPO requiring a reference policy and hence becoming memory inefficient. There are many ways to mitigate this: there are reference policy-free variants of DPO, for example, SimPO [9]. Alternatively, one can just pre-compute the reference log probs when the reference model + dataset buffer is fixed, see Agent-Q [10]. The main claimed strength for this paper is memory optimization. Therefore not considering these other options/comparing with them in the paper is a big weakness for this paper, since they are also using similar tricks (only loading one model (value or policy) into memory in a single step.

(Runtime comparisons)

The paper mentions memory requirement comparisons between multiple algorithms. While that is fine, generally methods that require online sampling from the policy can be quite slow compared to methods that don’t, due to how sampling often is implemented in public codebases. I am quite curious about what is the runtime comparison, between a single run of DPO vs a single run of SPR, on the same dataset and the same hardware? Also, runtime comparison can be slower/faster based on the particular implementation, so a comparison of required FLOPs etc., is also fine. The paper would have been stronger if it discussed this trade-offs better.

(Missing citations)

The paper is missing key literature comparing online/offline data and their trade-offs for LLM alignment, such as [1].

1. I do not fully understand the advantage of using Equation (26) over Equation (23), as Equation (26) introduces an additional reference function $W_\phi$ that has an extra optimization process. This process consumes more time and memory in pursuit of a better estimation. However, whether the estimator is better depends on whether the authors have developed a quick and precise solver for Equation (26). If the solution is suboptimal, it might perform worse than the empirical estimator from Equation (22). The authors should provide a more detailed discussion on the solver and its superiority over the closed-form solution presented in Equation (23).

2. In Algorithm 1, it is unclear how the minimization problems in Steps 4, 5, and 6 are solved. Moreover, As steps 4, 5, and 6 are executed in sequence, there is a concern that if $\phi$ in Step 4 is not well-solved, it could affect the learning of $\varphi$ in Step 5.

3. Some proofs should be discussed in more detail. For example, how to obtain $\pi^*(a \mid s)$ in lines 905-906 requires further explanation. Specifically, how does the equation $\sum_{a \in \mathcal{A}} \pi(a \mid s) = 1$ imply that $\pi^*(a \mid s) = \frac{\pi_{\text{old}}(a \mid s) \exp \left(\frac{1}{\beta} A^{\pi_{\text{old}}}(s, a)\right)}{\sum_{a^{\prime} \in \mathcal{A}} \pi_{\text{old}}\left(a^{\prime} \mid s\right) \exp \left(\frac{1}{\beta} A^{\pi_{\text{old}}}\left(s, a^{\prime}\right)\right)}$? Because $\pi^*(a \mid s)$ also satisfies Equation (27b), it seems that the authors should demonstrate that $\sum_{a^{\prime} \in \mathcal{A}} \pi_{\text{old}}\left(a^{\prime} \mid s\right) \exp \left(\frac{1}{\beta} A^{\pi_{\text{old}}}\left(s, a^{\prime}\right)\right) = \frac{1}{\exp \left(-1 - \frac{\zeta_s}{\alpha d_{\pi_{\text{old}}}(s)}\right)}$ to make the desired result $\pi^*(a \mid s) = \frac{\pi_{\text{old}}(a \mid s) \exp \left(\frac{1}{\beta} A^{\pi_{\text{old}}}(s, a)\right)}{\sum_{a^{\prime} \in \mathcal{A}} \pi_{\text{old}}\left(a^{\prime} \mid s\right) \exp \left(\frac{1}{\beta} A^{\pi_{\text{old}}}\left(s, a^{\prime}\right)\right)}$ established.

4. Regarding the experiments, my primary concern is in Figure 2, where SPR N8 requires five iterations to achieve a performance comparable to DPO, which is only trained in one iteration. This does not seem like a fair comparison. It would be beneficial to see additional training iterations for DPO to ensure a more fair comparison.

Moreover, the authors do not provide standard deviation or statistical significance for Table 5. It would be advantageous to test the proposed method across different random seeds to validate its robustness and repeatability.

1. The paper dedicates three pages (pages 3–5) to preliminaries, most of which could be moved to the appendix, retaining only the essential information in the main text.
2. Equations 16 to 20 appear to be directly sourced from [1], specifically equations 37 to 39. These should also be placed in the preliminaries section.
3. While the design of component W seems to be the paper's primary contribution, there is no ablation study on this design choice. This omission makes it unclear whether the reported improvements are primarily due to the proposed approach in the methods section.

[1] Peng, Xue Bin, et al. "Advantage-weighted regression: Simple and scalable off-policy reinforcement learning." arXiv preprint arXiv:1910.00177 (2019).