POIL: Preference Optimization for Imitation Learning

Thank you for your detailed and thoughtful review of our submission. We appreciate your positive assessment of our work and the recognition of the strengths in our approach, experimental design, and presentation.

Reply to Weaknesses:

The only criticism that I can produce is that it would be nice to know that this method scales to tasks that are more realistic than the MuJoCo benchmark environments, but the authors have already performed a set of experiments that should be considered sound in this particular research topic (fundamental IL algorithms).

Thank you for the comment. For this, we have conducted additional experiments on the Adroit tasks from the D4RL benchmark suite, which involve dexterous manipulation using a simulated robotic hand, as shown in the section of "Reply to All Reviewers" (above).

We believe these additional results address your concern and further validate the potential of POIL in practical applications beyond standard benchmarks. We will include these results in the Appendix of our revised paper.

---

Having tried similar approaches in the past, I was surprised that this method worked as well as it did. But, as shown in the authors' ablations, the scaling factor β is crucial, and in general should be significantly < 1. The authors mention "A smaller β value tends to smooth the preference function, which leads to more stable gradients and improved training dynamics", but do not say/show more about this (most results only show the final return); it would be nice to have more exploration of this.

Thank you for the comment. Indeed, the scaling factor $\beta$ plays a crucial role in POIL's performance. To intuitively illustrate why $\beta$ should be significantly smaller than 1, we provide a simple visualization of the POIL loss function $-\log(\sigma(\beta(\log \pi_\theta(a_E|s) - \log \pi_\theta(a|s))))$.

While we have included a visual representation in the figure (accessible via the link below), OpenReview cannot directly display images.

Visualization of POIL Loss Function

The figure demonstrates how the POIL loss varies with different values of $\beta$. The x-axis represents the log likelihood ratio between expert and policy actions. As $\beta$ decreases from 2.0 to 0.1, the loss curve becomes increasingly smooth, leading to more stable gradient signals. This explains our empirical finding that smaller $\beta$ values (typically $<1$ in our experiments) consistently lead to better performance through more stable training dynamics.

Thank you for your positive review and your valuable feedback on our submission. We are pleased that you found the experimental results compelling, particularly our method’s performance in the low-data regime.

Reply to Weaknesses

There is no explicit theoretical justification in this work, but this is minor to me. I think DPO and its variants have strong theory as is when it comes to solving the KL-regularized RL problem.

Thank you for the question. We have addressed some theoretical aspects in our response to Reviewer xCF9. In that response, we provide mathematical insights in more detail into the underpinnings of POIL, elaborating on its relationship with Behavioral Cloning (BC) and preference-based methods.

---

I feel like with less than 1 expert trajectory (which some offline imitation learning methods look into, albeit with additional suboptimal data) the method fails, but maybe this is not necessary in real-world settings.

You are correct that our current experiments did not evaluate the scenario with less than one expert trajectory. Our focus was primarily on settings where a moderate amount of expert data is available, as we believe this reflects many practical real-world applications where collecting at least some expert demonstrations is feasible.

However, we agree that exploring the performance of POIL in ultra-low-data regimes is an important direction for future research. In scenarios with less than one expert trajectory, methods that can effectively leverage additional suboptimal data or incorporate data augmentation techniques might be necessary to achieve satisfactory performance. While POIL in its current form may face challenges in such settings, we are optimistic that extensions of our method could address these challenges.

---

I am curious as to whether an RLHF-centric approach to this problem can be good to compare to (e.g. in LLMs there is the PPO w/ RM vs. DPO debate), where one trains a reward model on policy data (low reward) vs expert data (high reward) as the agent trains. In some sense, I feel like this method is similar to DAC, which is outperformed by IQ-Learn anyway, so maybe this is unnecessary.

Your suggestion to explore an RLHF-centric approach is insightful, especially considering the similarities between preference learning in RL and LLM contexts. While RLHF typically requires training a reward model, POIL avoids this by using direct preference comparisons, which simplifies the process in offline settings where expert data is limited. We see this as a promising direction for future research and appreciate your suggestion.

We appreciate your insightful observation regarding the similarities between RLHF (Reinforcement Learning from Human Feedback) approaches and traditional imitation learning techniques like DAC (Discriminator Actor-Critic). In fact, this connection was one of the key inspirations for developing POIL.

Your recognition of this connection highlights a core insight that drove the development of POIL. By leveraging preference optimization without the overhead of adversarial training or online interactions, POIL is able to achieve efficient and effective imitation learning, particularly in data-limited scenarios.

---

Reply to Weaknesses

From Section 2, we know that SPIN is the most closely related to the proposed POIL, however, the POIL objective is directly adapted from CPO instead of SPIN, which is strange and may cause confusion. I believe CPO is chosen because it avoids using a preference model. In this sense, maybe CPO is the most related work to POIL?

Thank you for your question regarding the relationship between POIL, SPIN, and CPO. Let us clarify as follows.

• Clarifying the Relationship: While SPIN and CPO both inspire our work, they contribute in different ways:
  • SPIN (Self-Play Fine-Tuning) introduces a mechanism where a model improves itself by comparing its current outputs with those from a previous version (the reference model), continuously updating the reference model during training. This concept of self-improvement without external data aligns with our goal in POIL.
  • CPO (Contrastive Preference Optimization) provides a reference-free loss function by eliminating the need for a reference policy, which simplifies the optimization process. This aligns with our objective to avoid the complexities associated with maintaining a reference model.
• Why We Adopted the CPO Objective: In POIL, we adapt the reference-free loss function from CPO because it allows us to directly compare the agent’s actions with expert actions without needing a reference model or preference dataset. This simplifies the training process and reduces computational overhead.
• How SPIN Influences POIL: Although we use the CPO loss function, the concept of self-improvement in POIL is inspired by SPIN. Specifically, we draw from SPIN’s idea of refining the policy by comparing its outputs, but we modify it to compare the agent’s actions with expert actions instead of previous outputs.

---

SPIN can generate its training data and refine itself by distinguishing between current and previous outputs, continuously updating its reference model. In Section 3.2, Equation (3) is explained to maximize the divergence between the agent’s current behavior and its previous, which tries to link POIL to SPIN. However, it is not clear how this goal has been achieved.

Thank you for pointing out the problem. We will update accordingly in our next revision, and briefly summarize as follows.

The primary goal of Equation (3) in POIL is to maximize the preference of expert actions over the agent’s current actions, not to maximize the divergence between the agent’s current behavior and its previous behavior.

• Actual Mechanism in POIL: In POIL, we directly compare the agent's actions with the expert's actions at the same state. The loss function encourages the agent to assign higher probabilities to expert actions and lower probabilities to its own sampled actions. This is achieved through the preference loss:

$$\mathcal{L}\_{\text{POIL}}(\pi\_{\theta}) = -\mathbb{E}\_{(s, a\_E, a) \sim \mathcal{D}\_E} \left[ \log \sigma\left( \beta \left( \log \pi\_{\theta}(a\_E \mid s) - \log \pi\_{\theta}(a \mid s) \right) \right) \right].$$

- The term $\log \pi_{\theta}(a_E|s)$ encourages the policy to prefer expert actions.
- The term $\log \pi_{\theta}(a|s)$ penalizes the probability of the agent's own (potentially suboptimal) actions.

• Link to SPIN: While we do not maximize the divergence between the agent’s current and previous behaviors as in SPIN, we are inspired by SPIN’s self-improvement through preference comparisons. In POIL, instead of using a previous policy as a reference, we use the expert’s actions as the preferred behavior, and the agent’s own actions as the less preferred behavior. This adaptation allows us to refine the agent’s policy by learning from the discrepancies between its actions and the expert’s actions.

---

Reply to Questions

1. How does Equation (3) maximize the divergence between the agent’s current and previous behaviors?

The loss function in Equation (3) encourages the agent to:

• Increase the probability of expert actions $\log \pi_{\theta}(a_E|s)$.
• Decrease the probability of its own sampled actions $\log \pi_{\theta}(a|s)$.

By doing so, the agent learns to align its policy more closely with the expert's behavior while reducing the likelihood of current actions.

---

2. It seems POIL works quite well when $\lambda = 0$ in Table 1 (In this paper), and POIL is sensitive to different $\lambda$ in Figure 3. Does it mean that there is no need to introduce the BC regularization to the POIL loss?

As we mentioned in Discussion Section (line 508-520), the experimental results in Table 1 (In this paper) show, in data-scarce settings (e.g., learning from a single demonstration), setting $\lambda = 0$ yields better performance. This suggests that the BC regularization term may not be necessary in such scenarios, as it can overly constrain the policy, limiting exploration when data is limited.

Conversely, as our experimental results in Table 2 (In this paper) indicate, in environments with more abundant data, the impact of $\lambda$ is less pronounced. Both $\lambda = 0$ and $\lambda = 1$ perform competitively, indicating that the BC regularization can stabilize training without significantly impacting performance in richer data scenarios.

The inclusion of the BC regularization term $\lambda \cdot \mathcal{L}_{\text{BC}}$ is intended to encourage the policy to stay close to the expert actions, similar to how previous works like [1] and [2] incorporate behavior regularization in offline reinforcement learning. This helps prevent bootstrapping on out-of-distribution actions, which can be critical in data-rich environments. However, in data-limited scenarios, this term might restrict the policy too much, preventing adequate exploration and generalization.

Therefore, while the BC regularization can be beneficial in certain cases, it is not always necessary. The sensitivity to $\lambda$ highlights the importance of tuning this hyperparameter based on the amount and quality of the available expert data. We will expand on this point in the revised manuscript and provide more detailed guidance on setting $\lambda$ based on different data availability contexts.

[1] Fujimoto, S., & Gu, S. S. (2021). A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34, 20132-20145. [2] Wu, Y., Tucker, G., & Nachum, O. (2019). Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361.

Thank you for your mathematical analysis and the theoretical connections you've drawn between POIL and BC. Your analysis prompts us to better explain these technical aspects in depth. We address your concerns point by point below:

Reply to Weaknesses

If we ignore the term involving $(\log \sigma)$ in the POIL loss for a moment, the Behavioral Cloning (BC) gradient is simply the "infinite sample" estimate of the POIL loss. In other words, Maximum Likelihood Estimation (MLE) in exponential families already includes a "negative gradient," so there is no need to add one explicitly.

To elaborate on the relationship between POIL and BC, we would like to share several key observations:

First, a lot of simulation environments, e.g. Adroit Hand (Door, Hammer, etc.), Gym-Mujoco (HalfCheetah, Walker2d, etc.), Franka Kitchen, Ant Maze, and so on, constrain action spaces to $[-1, 1]$. Generally, to handle such bounded action spaces, people would use tanh to map unbounded Gaussian samples to bounded actions (squashed Gaussian), as demonstrated in SAC (Soft Actor-Critic) [SAC] and f-IRL (Inverse Reinforcement Learning via State Marginal Matching) [f-IRL]. Compared to direct clipping actions which distorts the Gaussian density at boundaries, this approach maintains a well-defined probability distribution. Specifically, an action $a$ is generated by: $a = \tanh(a_{in}), \text{ where } a_{in} \sim \mathcal{N}(\mu_\theta(s), \sigma_\theta(s))$

As detailed in SAC's Appendix C [SAC], this transformation introduces a probability correction term $-\log(1-\tanh^2(a_{in}))$ in the log-likelihood due to the change of variables.

For basic BC with squashed Gaussian policy, the objective is:

$$L\_{BC}(\theta) = -\mathbb{E}\_{(s,a\_E)\sim D\_E}[\log \pi\_\theta(a\_E|s)]$$ $$= -\mathbb{E}\_{(s,a_E)\sim D\_E}[\log \mathcal{N}(a\_{in,E}; \mu\_\theta(s), \sigma\_\theta(s)) - \log(1-\tanh^2(a\_{in,E}))]$$ $$= -\mathbb{E}\_{(s,a_E)\sim D\_E}[-\frac{1}{2}\log(2\pi\sigma\_\theta(s)^2) - \frac{(a\_{in,E}-\mu\_\theta(s))^2}{2\sigma\_\theta(s)^2} - \log(1-\tanh^2(a\_{in,E}))]$$

The gradients are:

$$\nabla\_\mu L\_{BC} = \mathbb{E}\_{(s,a\_E)\sim D\_E}\bigg[\frac{a\_{in,E} - \mu\_\theta(s)}{\sigma\_\theta(s)^2}\bigg] = \mathbb{E}\_{(s,a\_E)\sim D\_E}[g\_\mu]$$ $$\nabla\_\sigma L\_{BC} = \mathbb{E}\_{(s,a\_E)\sim D\_E}\bigg[-\frac{1}{\sigma\_\theta(s)} + \frac{(a\_{in,E} - \mu\_\theta(s))^2}{\sigma\_\theta(s)^3}\bigg] = \mathbb{E}\_{(s,a\_E)\sim D\_E}[g\_\sigma]$$

Let $g_\mu$ and $g_\sigma$ denote the terms inside the expectations of BC gradients. We would use it later.

While the probability correction term $-\log(1-\tanh^2(a_{in,E}))$ appears in the objective, it does not affect the gradients in BC since it doesn't depend on the policy parameters $\theta$. The key issue lies in how BC handles the pre-tanh space directly: when the same action space correction is needed, the required parameter updates vary dramatically based on position in the action space. To illustrate this:

Consider two cases where we want the same 0.001 correction in action space:

1. Center: $μ = 0.0, a_E = 0.001$

   • In action space: $\tanh(0.0) = 0.0$, $a_E = 0.001$
   • Required update in pre-tanh space: $\tanh^{-1}(0.001) - 0.0 = 0.001$

2. Near-boundary: $μ = 1.832, a_E = 0.951$

   • In action space: $\tanh(1.832) = 0.95$, $a_E = 0.951$
   • Required update in pre-tanh space: $\tanh^{-1}(0.951) - 1.832 = 0.010358$

This analysis shows that while both cases need the same 0.001 correction in action space, the required update in pre-tanh space near boundaries is over 10 times larger than in the center region. This difference arises from the non-linear nature of the tanh transformation: near boundaries (|a| ≈ ±1), the derivative of tanh approaches zero, requiring much larger pre-tanh updates to achieve the same action space correction compared to central regions (|a| ≈ 0) where the derivative is close to one. Since BC operates directly on the pre-tanh parameters, this leads to disproportionately large parameter updates near boundaries. More critically, even small adjustments in action space near boundaries demand substantially larger parameter updates, which can cause potential oscillations and instability during training - the policy might overshoot the target behavior near boundaries and require multiple corrective updates, leading to inefficient learning and potential convergence issues.

In contrast, POIL's sampling-based approach naturally handles these non-linear effects of the tanh transformation, providing more appropriate parameter updates across the entire action space.

---

One might then ask about the effect of $(\log \sigma)$. If we refer to the original MaxEnt IRL/DPO derivations, this term in the loss is meant to ensure closeness to the prior/reference policy. However, the POIL loss does not include any regularization toward the prior (i.e., there is no $\pi_{\text{ref}}$ in the denominator). Thus, at best, it acts as entropy regularization (i.e., increasing the variance for a Gaussian policy). This could be achieved without requiring any samples from the policy, which raises the question of what additional benefit this more complex procedure provides compared to Behavioral Cloning.

Regarding the reviewer's point about log sigmoid potentially serving only as variance regularization, we analyze BC with log sigmoid:

$$L\_{BC-LS}(\theta) = -\mathbb{E}\_{(s,a\_E)\sim D\_E}\big[\log \sigma(\log \pi\_\theta(a\_E \mid s))\big]$$ $$= -\mathbb{E}\_{(s,a\_E)\sim D\_E}\big[\log \sigma(\log \mathcal{N}(a\_{in,E}; \mu\_\theta(s), \sigma\_\theta(s)) - \log(1 - \tanh^2(a\_{in,E})))\big]$$

Let $g_\mu$ and $g_\sigma$ denote the terms inside the expectations of BC gradients as defined above. Then for BC with log sigmoid:

$$\nabla\_\mu L\_{BC-LS} = -\mathbb{E}\_{(s,a\_E)\sim D\_E}\big[(1 - \sigma(\log \mathcal{N}(a\_{in,E}; \mu\_\theta(s), \sigma\_\theta(s)) - \log(1 - \tanh^2(a\_{in,E})))) \cdot g\_\mu\big]$$ $$\nabla\_\sigma L\_{BC-LS} = -\mathbb{E}\_{(s,a\_E)\sim D\_E}\big[(1 - \sigma(\log \mathcal{N}(a\_{in,E}; \mu\_\theta(s), \sigma\_\theta(s)) - \log(1 - \tanh^2(a\_{in,E})))) \cdot g\_\sigma\big]$$

Let's examine two cases (same as mentioned above) where policy mean $\mu$ and expert action $a_E$ differ by approximately 0.001 in action space (assuming $\sigma_\theta(s) = 1$ for simplicity):

1. Center: $\mu = 0.0$, $a_E = 0.001$

   • Total log probability: $-0.919$
   • Weight term = $1 - \sigma(-0.919) \approx 0.715$

2. Near-boundary: $\mu = 1.832$, $a_E = 0.951$

   • Total log probability: $1.429$
   • Weight term = $1 - \sigma(1.429) \approx 0.193$

While BC-LS introduces a weighting mechanism that scales down the large pre-tanh updates near boundaries (weight ≈ 0.193 vs 0.715 in center), the policy still computes gradient updates based on Maximum Likelihood Estimation (MLE) in the pre-tanh space, where updates near boundaries are over 10 times larger (0.010 vs 0.001) than in the center region for the same action space correction. Although the weight term partially mitigates the instability by reducing the effective update size, it does not address the fundamental issue that the policy optimization still operates in a space requiring disproportionately large updates near boundaries.

In contrast, POIL approaches this challenge through a fundamentally different mechanism:

$$L\_{POIL}(\theta) = -\mathbb{E}\_{(s,a\_E)\sim D\_E}\big[\log \sigma(\beta(\log \pi\_\theta(a\_E \mid s) - \log \pi\_\theta(a \mid s)))\big] , \text{where } a \sim \pi\_\theta(\cdot \mid s)$$

The key distinction lies in how POIL handles action probabilities. While both POIL and BC-LS use a $(1-\sigma(\cdot))$ term that moderates updates, POIL's probability ratio approach compares expert actions with sampled actions from the current policy, allowing it to assess the relative preferences across the entire action distribution rather than just at the mean. This leads to partial cancellation of the Jacobian terms ($\log(1-\tanh^2(\cdot))$) between expert and sampled actions, reducing their impact. Additionally, only the Jacobian terms from sampled actions contribute to the gradient computation, providing a more balanced approach to handling bounded actions compared to BC-LS's direct compensation mechanism.

Reply to Questions

1. Could you please add standard error bars to all the plots / tables in the paper?

For Table 1 (in this paper), we will provide the result with standard error bars in the next revision.

For Table 2 (in this paper), as mentioned in the paper, we report the maximum scores achieved across the training process, normalized to expert performance. Since these are best-case results rather than averages, standard errors/deviations are not applicable in this context.

---

2. Could you test out your method on something other than the three easiest Mujoco environments?

Thank you for this suggestion. In response, we have conducted additional experiments on the Adroit tasks from the D4RL benchmark suite, which involve complex, high-dimensional robotic manipulation tasks using a simulated robotic hand, as shown in the section of "Reply to All Reviewers" (above).

---

3. Could you try the "linear" version of the loss I discuss in the weaknesses section? Maybe try tacking on a variance bonus?

As we discussed in our response to your weaknesses section, the use of the log-sigmoid function in the POIL loss is crucial for stabilizing gradients and effectively handling the non-linearities introduced by the tanh transformation in squashed Gaussian policies.

We acknowledge that exploring the linear version and assessing its impact on performance is a valuable direction for future research.

Reply to the Second Comment

1. Differentiable Sampling via Reparameterization Trick

Observe that if I sampled some set of actions from the current learner policy, there is no gradient between these sampled actions and the mean. So, if as you say above, the log det Jacobian term doesn't affect the BC loss, I don't see why the log det Jacobian term would affect the linear POIL loss.

In POIL, we sample actions from the current policy using the common reparameterization trick, which ensures that these sampled actions are differentiable with respect to the policy parameters $\theta$. Specifically, the sampling process is:

$$\text{Sample } \epsilon \sim \mathcal{N}(0, I),$$ $$a_{\text{raw}} = \mu_\theta(s) + \zeta_\theta(s) \cdot \epsilon,$$ $$a = \tanh(a_{\text{raw}}).$$

This means that the sampled action $a$ depends on $\theta$ not just through the mean $\mu_\theta(s)$ and standard deviation $\zeta_\theta(s)$, but also through the sampled noise $\epsilon$. In order to make it clearer, we will mention this in our next revision.

Note: To avoid confusion between the standard deviation (previously denoted as $\sigma$) and the sigmoid function $\sigma(\cdot)$ used in the loss function, we now use $\zeta_\theta(s)$ as the symbol for the standard deviation, instead.

2. Impact of the Log-Determinant Jacobian Term

So, taking the above two points together, I would argue that POIL should at best match the performance of BC without the $\log \sigma$ -- the 'negative samples' / core algorithmic change on top of base BC give you nothing without it.

The log-determinant Jacobian term, $\log(1 - \tanh^2(a_{\text{raw}}))$, contributes to the gradient in POIL but not in BC, leading to different update directions. While it is true that this term does not affect the gradient when evaluating at the expert action $a_E$ (since $a_E$ is fixed), it does affect the gradient when evaluating at the sampled action $a$, because $a_{\text{raw}}$ depends on $\theta$.

3. Regarding Ablations in Figure 4 and Performance Relative to BC

"While I do indeed believe that adding in an arbitrary nonlinearity would make the gradients different, it need not make things better (e.g. some of the ablations in Fig. 4 underperform BC)."

First of all, we would like to clarify that the ablation study presented in Figure 4 is a comparison of alternative preference learning methods (SimPO, SLiC_HF, RRHF, ORPO, POIL), and our POIL (olive-green in the figure) clearly outperform others (definitely NOT UNDERPERFORM).

Through these experiments, we selected CPO-based loss for POIL because it consistently outperforms others. This indicates that the specific nonlinearity chosen in POIL is not arbitrary but rather empirically validated as effective.

4. Sufficient Sample Argument

So, if you sampled enough, you would exactly get the BC gradient. Thus, in this simplified setting, I don't think there is any reason to believe POIL should out-perform BC.

Your analysis focuses on a simplified setting without log sigmoid, which fundamentally differs from our method. Our previous response (in the first reply to BC-LS as well as item one in this reply) has shown that POIL is different from BC.

In all of our experiments, POIL consistently outperforms BC across all benchmarks, including the extra one for Adroit tasks (presented in the section of "Reply to All Reviewers")

From above, both theoretical and empirical evidences show that POIL outperforms BC.

5. Reporting Maximum Scores

Re: reporting maximum scores. Thank-you for bringing this to my attention, I did not catch this on my initial read. Unfortunately, this is not standard practice, at least in imitation learning. Doing this can completely obscure any training instabilities of the method and is a huge red flag. Furthermore, given one needs online interaction to do the rollouts to select the best-performing checkpoint, this significantly detracts from the purported "offline" nature of the method. I would strongly suggest reporting the performance of the last checkpoint / using a purely offline model selection procedure (e.g. picking the minimum validation error).

Thank you for requesting. However, since this requires additional experiments for a purely offline model selection procedure and checkpoints, we are not able to provide these soon. In any case, we would like to emphasize that in all of our experiments, including the newly added Adroit benchmarks, POIL consistently outperforms other methods. This robust performance across multiple datasets and benchmarks reinforces the effectiveness of POIL.

Sorry, let me make sure I'm understanding this correctly: you don't "detach" the action tensor from the computation graph before throwing it into the POIL loss -- like,$\frac{\partial a}{\partial \theta}\neq 0$?

Yes, we use PyTorch's torch.rsample(), which implements the reparameterization trick we described above. You may refer to the official documentation here for details. Through reparameterization, sampled actions maintain differentiable paths with respect to policy parameters.

Also, when you're calculating the likelihood, $(log(\pi_{\theta}(\cdot|s)))$, do you pass in $a_{raw}$ or $a$? I'm trying to see whether cancels out in the gradient calculation.

When calculating the likelihood $\log(\pi_{\theta}(a|s))$ under squashed Gaussain policy, we use $a_{\text{raw}}$. Specifically, we compute:

$$\log(\pi_{\theta}(a|s)) = \log \mathcal{N}(a_{\text{raw}}; \mu_{\theta}(s), \zeta_{\theta}(s)^2) - \sum_i \log\left(1 - \tanh^2(a_{\text{raw}, i})\right)$$

Here, $a_{\text{raw}}$ is used in the Gaussian log-probability and the Jacobian adjustment for the squashing function, while $a = \tanh(a_{\text{raw}})$ is the action applied in the environment.

Relatedly, have you tried the "detached" version where you treat the learner samples as another "BC" dataset, but one you want to move away from? This is the way things are actually implemented in SPIN / online DPO, where there is no gradient w.r.t. the parameters of the model of the learner samples.

We have not tried the "detached" version. In our implementation, we allowed gradients to flow through the learner samples back to the policy parameters, the same as in SAC's squashed gaussian policy. However, exploring the "detached" version is an interesting idea, and we will consider it in future work.

Reply to Weaknesses

This work is missing comparisons to (or atleast discussions in related work comparing to) sample-efficient BC approaches that can work with one or very few demonstrations like ROT [1] and MCNN [2].

Thank you for your suggestions. While ROT is online learning approach makes direct empirical comparisons challenging given POIL's offline setting, we conduct new experiments with MCNN on the Adroit environment. Using the same D4RL dataset, our preliminary results show competitive performance, which has been shown on the section of "Reply to All Reviewers".

---

A comparison (or discussion about) CPL [3] and other baselines in the CPL paper, another RLHF method for robotics, is also missing.

While both POIL and CPL address robotic control through RLHF approaches, they operate under different learning settings - CPL leverages suboptimal demonstrations and preference learning, while POIL focuses on learning from expert demonstrations. Given these different learning conditions, establishing a fair empirical comparison would be challenging.

---

the return tabulated for different methods is not normalized –- this makes it hard to determine its performance between random and expert and hard to compare with other papers.

We understand the concern regarding normalization of returns for easier comparison. In our paper, we specifically noted that the scores are not normalized to expert data because we cannot directly obtain the expert scores from this dataset. Given this constraint, we report the raw scores to maintain transparency across all experimental conditions.

---

The evaluation environments are very simple vector-observation mujoco environments. It would be helpful to extend to either more environments from D4RL that testing stitching like the ant maze environment, or more complicated image-based environments like Atari, or more dexterous robotics environments like Robosuite and Adroit.

We have extended our experiments to include the Adroit tasks from the D4RL benchmark suite, which has been shown on the section of "Reply to All Reviewers".

Due to the limited time for the rebuttal, we focused on the expert datasets within these tasks, using the same settings as in the MCNN paper to ensure consistency and fair evaluation. We set $\lambda=0$ and $\beta=1$ in our experiments. Our updated results, presented in Tables 1 and 2 above, demonstrate that POIL outperforms other state-of-the-art methods, including those discussed in the MCNN paper, across all evaluated tasks.

These results validate the scalability and effectiveness of POIL in more complex and realistic environments, addressing your concern. We appreciate your suggestion, as it has allowed us to further demonstrate the strengths of POIL. We will include these results in the Appendix of our next revised paper.