Two-Step Offline Preference-Based Reinforcement Learning with Constrained Actions

1. This paper is indeed incremental, and the contributions (both theoretically and empirically) are not sufficient enough to be presented at this conference. For theory, the authors claim their method can mitigate reward hacking and reduce the complexity of RL, but no quantified analysis is presented. For the experiment, the authors do not offer indicators of the two key contributions.

2. The analysis in Section 4.2 is not enough. More content should be included, such as theoretical analysis of improved efficiency regarding the behavior policy probability density threshold $p$ and the extent to which reward hacking can be mitigated.

3. Further, the experiments need more clarity.

1. The core concept of constraining the action space to mitigate reward hacking and over-optimization is not entirely novel. The modification over conventional offline PbRL is very slignt and lacks considerations. Merely restricting the action space based on the offline dataset could introduce several issues, such as limiting the policy’s ability to explore alternative, potentially optimal actions. This paper does not account for the potential negative consequences of such a simplistic restriction, which may lead to unintended limitations in policy performance and adaptability.
2. The paper lacks a clear structure, with an excessive focus on background information and well-known techniques. The authors should place greater emphasis on detailing their own methods, providing both theoretical insights and empirical analysis to strengthen the contribution of their approach.
3. The experiments are not convincing and comprehensive, as the authors compare their method only to basic baselines in a limited set of simple environments. Additionally, the presentation of the results is unclear. For example, the authors show the performance of the best policy learned by a method during training, which is actually improper. In Table 1, certain values are bolded without explanation. Moreover, the final scores should be reported using normalized rewards rather than raw sums to enable more meaningful comparisons. Furthermore, Figures 1, 2, and 3 lack consistency in their x-axis labeling, making cross-comparisons difficult. Consolidating the results for each task into a single figure and displaying all methods would improve clarity and readability.
4. Overall, this paper does not meet the standards expected for this conference. Significant improvements are needed in both the methodology and experimental evaluation to adequately address the mentioned challenges in offline PbRL.

1. Insufficient Novelty. The contributions and novelty of using a two-step learning framework in the PBRL problem are limited. The primary innovation in this work lies in the use of a constrained action space. However, 1) this modification to the original PBRL objective appears to be marginal, and 2) the paper lacks both theoretical and principled justification for the overall performance improvements.
2. Ambiguous Experimental Results. Some of the experimental results and figures are difficult to interpret, leading to confusion. There is a lack of in-depth analysis regarding the model's performance and the underlying reasons for its superior results. A more detailed study is necessary to clarify these points (please refer to the specific question below).
3. Technical Issues. The core of the proposed method is to constrain the action space to include only actions with a high probability of being sampled by a behavior cloning policy. However, it remains unclear how the method guarantees that policies with higher performance than those in the dataset can always be found. This raises concerns about the robustness of the approach.

1. The authors should expand the related work section, as recent key studies on both online and offline PbRL, as well as RLHF for LLMs, are currently missing.

2. Especially, DPPO [1] is a two-step algorithm for offline PbRL that learns a preference predictor followed by direct policy learning through a newly proposed preference score based on policy-segment distance, thereby avoiding reward learning and RL. Additionally, CPL [2] directly learns a policy without both reward modeling and RL. Both approaches address the same challenges as this paper, indicating that this is not the first study to tackle these issues.

3. I am concerned that with unbounded regularization, as mentioned in line 240, optimization might become unstable. Additionally, could excessive conservatism risk resulting in a policy that is unable to achieve high rewards?

4. Further discussion on function $f$ and $\mathcal{P}^\prime$ are necessary to improve clarity and avoid confusion.

5. I recommend including training and evaluation configurations, such as hyperparameters for each method and details on the evaluation process (e.g., number of runs over random seeds), in the appendix for reproducibility.

6. The paper lacks quantitative results for experiments on pessimism effectiveness. Additionally, if these results are based on a single run, I suggest conducting multiple runs and illustrating the findings. This suggestion applies to all the experiments in section 5.3, 5.4, 5.5.

7. Several SOTA baselines in offline PbRL, such as PT [3], DPPO [1], and CPL [2], are missing from the evaluation.

8. Sections 5.4 and 5.5 include observations but provide few discussion of the results.

[1] An, Gaon, et al. "Direct preference-based policy optimization without reward modeling." Advances in Neural Information Processing Systems 36 (2023): 70247-70266.

[2] Hejna, Joey, et al. "Contrastive preference learning: Learning from human feedback without rl." arXiv preprint arXiv:2310.13639 (2023).

[3] Kim, Changyeon, et al. "Preference transformer: Modeling human preferences using transformers for rl." arXiv preprint arXiv:2303.00957 (2023).