Improving Reward Model Generalization from Adversarial Process Enhanced Preferences

Q1: The method is dependent on that later policies are better than earlier ones, but this may not always hold due to training fluctuations. A discussion on how this randomness impacts learning would be valuable.

A1: This randomness indeed has a negative impact on the accuracy of the generated preference. Nevertheless, APEC additionally leverages the Wasserstein distance metric to help choose the policy pair, which enhances the robustness of preference generation to randomness. Experimental results in Section 4.4 show that APEC can generate more accurate preference data compared to baselines.

Q2: In Eq. 1, the loss calculation uses a section of the trajectory, but the impact of the parameter l on performance is not explored.

A2: We apologize for missing this ablation study. Here we perform the ablation study on segment length for reward learning, across both feature-based HalfCheetah and pixel-based walker_run tasks. The "default" refers to the values used in our original experiments, and we show the overall task performance here. The findings show that APEC is not sensitive to this parameter.

Q3: Comparisons are limited to older methods. Additionally, including a baseline with an oracle reward would help illustrate the gap to the upper bound. Additionally, including a baseline with an oracle reward would help illustrate the gap to the upper bound.

A3: To address your concern, we add two recent baselines, BC-IRL [1] and DRAIL [2] and the method with oracle reward. We use the recommended hyperparameters from the source code and slightly tune them. The task performance results are shown in the table below, where we find that APEC still achieves the best performance compared to BC-IRL and DRAIL.

[1] BC-IRL: Learning Generalizable Reward Functions from Demonstrations. ICLR’23.

[2] Diffusion-Reward Adversarial Imitation Learning. NeurIPS’24.

Q4: Limited discussion with related domains: The adversarial imitation learning framework shares similarities with GAN training, yet there is minimal discussion of techniques from this domain. Furthermore, the work is closely tied to preference learning in LLMs, and a discussion of related works, which also focus on improving reward model generalization, is recommended.

A4: Thank you for your insightful feedback. We agree that there are notable similarities between the AIL framework and GAN training because both of them involve an adversarial learning procedure. We will replenish the discussion on GAN in the revision.

Besides, while preference learning in LLM is indeed relevant to our work, it is crucial to highlight that APEC focuses on the automatic generation of diverse and accurate preference data through AIL, whereas the paper referenced by the reviewer centers on improving reward generalization by improving preference learning algorithms with fixed preference data. We will elaborate on these distinctions in the revised version.

Q1: Need to check whether there are more recent and suitable baselines available

A1: Thank you for your valuable suggestions. We add two more recent reward learning baselines, BC-IRL [1] and DRAIL [2], and evaluate them on the MuJoCo benchmark. We use the recommended hyperparameters from the source code and slightly tune them. The results are shown in the table below, where we find that APEC still achieves the best performance.

[1] BC-IRL: Learning Generalizable Reward Functions from Demonstrations. ICLR’23.

[2] Diffusion-Reward Adversarial Imitation Learning. NeurIPS’24.

Q2: It would be valuable to explain whether there are other autonomous pipelines that operate without relying on human preference data, and if such approaches exist, what advantages the method using suboptimal data offers in comparison. Including this discussion would provide important insights for understanding the significance of this study.

A2: There are other autonomous methods that can operate without human preference data. Notably, LLM-based labeling [1][2] has recently emerged as a popular approach for generating preference data. However, the accuracy of LLM-generated labels is not always guaranteed due to the potential hallucinations of LLMs. Moreover, for vector-based tasks, LLMs are currently incapable of providing accurate labels because LLMs cannot understand such abstract vectors. In contrast, constructing preference pairs from suboptimal data offers a rather principled approach to generating accurate preference data. For instance, APEC builds on the theoretical property of AIL that the value gap bounds decrease as the iteration number increases.

[1] Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning. arXiv preprint ,2025

[2] Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model. arXiv preprint ,2025

Q3: Regarding the graph in Figure 2, are there specific environments where the AIL process does not match the performance of the demonstrations? If so, is there an analysis of the performance loss resulting from that gap?

A3: In our experiments, there is no environment where AIL fails to match the performance of the demonstrations.

Q1: Whether the claim that a progressive improvement in policy value over successive iterations can be achieved is correct or not.

A1: We apologize for this confusing claim. Here we want to claim that Proposition 3.1 can indicate that policies from the AIL process will establish a preference relationship when their corresponding iteration indexes differ significantly. To see this, on one hand, Proposition 3.1 essentially proves a lower bound on $V^{\pi^k}$ ($V^{\pi^k} \geq f(k)$). On the other hand, due to the proximal update nature in AIL, it is easy to prove an upper bound on the value difference between $\pi^k$ and the initial policy $\pi^1$. In particular, we can prove that

$$V^{\pi^k} - V^{\pi^1} \leq H^2 (\ln(|\mathcal{A}|))^{\frac{1}{4}} K^{\frac{3}{4}}.$$

This provides an upper bound on $V^{\pi^k}$ ($V^{\pi^k} \leq g(k)$). By combining the lower and upper bounds, we can establish that for two policies $\pi^{k_1}$ and $\pi^{k_2}$ whose indexes $k_1$ and $k_2$ differ sufficiently such that $f(k_1) \geq g(k_2)$, the preference relationship $V^{\pi^{k_1}} \geq V^{\pi^{k_2}}$ holds. This insight inspires us to leverage the AIL training process to generate preference data.

We appreciate your valuable question and will fix this claim in the revision.

Q2: Whether such a claim can theoretically guarantee a board coverage of the preference data with significantly different iteration indices.

A2: We would like to clarify that we do not claim that Proposition 3.1 can theoretically guarantee a board coverage of the preference data. Rather, it establishes that policies from the AIL training process will hold a preference relationship when their corresponding iteration indexes differ sufficiently. This provides the rationale for leveraging the AIL in our preference data construction methodology.

For the coverage property, we empirically validate that the preference data constructed based on the AIL process can provide a wider coverage than previous methods; see Table 4 and Figure 4 for details. This improved coverage likely stems from AIL's inherent advantages over BC—specifically, AIL-trained policies typically achieve better values by avoiding the compounding errors that plague BC approaches. Therefore, the preference data collected by policies from the AIL could have a better coverage on high-return regions.

Q3: Some more recent baselines should be compared and discussed.

A3: We add a more recent baseline DRAIL [1]. We use the recommended hyperparameters from the source code. APEC still achieves a better performance than DRAIL.

[1] Diffusion-Reward Adversarial Imitation Learning. NeurIPS’24.

Q4: The work is only verified on some simple benchmark.

A4: As noted by the reviewer, most works use these benchmarks as testbeds. We emphasize that our setup is more challenging compared to prior works. Specifically, we test on pixel-based control tasks and use fewer demonstrations. APEC consistently achieves strong performance across these challenging tasks, showing the potential benefits for real-world tasks.

Q5: Can only the Wasserstein distance criterion be utilized for the preference generation?

A5: We use the L2 distance as an alternative. We find that APEC with L2 distance can still have a satisfying performance.

Q6: How about the performance of a random policy collection with the combination of the Wasserstein distance criterion?

A6: We conduct experiments using only the Wasserstein distance criterion for constructing preference pairs. The results show that this leads to significantly lower performance across all five tasks. This further reinforces the importance of epoch interval filtering.

Q7: Does the number of the demonstrations have a high impact on the final performance?

A7: We conduct experiments with different numbers of demonstrations. The results show that the number of demonstrations has a limited impact on the final performance.

Q8: Curious on achieving better-than-demo performance.

A8: In certain tasks with structured true reward (e.g., linear w.r.t the state feature), PBRL could identify the underlying structure, resulting in a generalizable reward. Such a generalizable reward can guide policy learning in unseen regions, thereby achieving better-than-demo performance.

Thank you for your valuable suggestions!

Q1: Could the authors provide a comparative analysis of computational costs between APEC and baseline methods?

A1: The table below shows the computation costs (training time) of all methods on the MuJoCo task. We can see that APEC and the other methods require similar training time, with the majority of time spent on policy learning using the learned reward. Additionally, we present the training time of each stage in the reward learning process. We find that calculating the W-distance incurs only a slight increase in training time.

Q2: Have the authors explored APEC's performance on tasks beyond continuous control, such as discrete action spaces or goal-oriented environments? Dependence on DAC specifically might limit generalizability to other AIL algorithms.

A2: We conducted an additional experiment on the Atari game Pong, which features a discrete action space. Results show that APEC consistently outperforms both DREX and LERP in terms of task performance, demonstrating its broader applicability. Notably, for the Pong experiment, we used an alternative AIL method, IQLearn [1], to generate preference data, which further highlights APEC's generalizability across different AIL algorithms.

[1] IQ-Learn: Inverse soft-Q Learning for Imitation. NeurIPS’21.

Q3: How sensitive is APEC's performance to hyperparameter choices?

A3: We perform ablation studies on key hyperparameter choices, including the training epoch interval, Wasserstein distance threshold, and segment length for reward learning, across both feature-based HalfCheetah and pixel-based walker_run tasks. The "default" refers to the values used in our original experiments, and we show the overall task performance here, as suggested by the reviewer. The results show that APEC is not sensitive to these hyperparameters.

• Epoch Interval

  	5	10 (default)	50
  HalfCheetah	11957	12232	10019
  	5	10 (default)	20
  walker_run	710	701	591

• Distance Threshold

  	0.05	0.1 (default)	0.2
  HalfCheetah	13291	12232	8848
  	0.005	0.01 (default)	0.02
  walker_run	709	701	736

• Segment Length

  	100	500	1000 (default)
  HalfCheetah	12658	13434	12232
  	5	10 (default)	20
  walker_run	712	701	696

Q4: The method assumes relatively monotonic improvement during AIL training, which might not hold in highly stochastic environments.

A4: We would like to clarify that the theoretical analysis (Proposition 3.1) does not assume monotonic improvement during AIL training. Instead, we indeed prove that the value gap bounds of policies learned during AIL process increase as the iteration index $k$ decreases. This suggests that the preference relationship between two policies holds if their corresponding iteration indexes differ sufficiently, which further inspired our algorithmic design.

To handle with the case where the environment is highly stochastic, APEC leverages the Wasserstein distance metric to help choose the policy pair, which enhances the robustness of preference generation.

Q5: Relatively high-quality demonstrations despite claims of challenging setup and exploration of APEC's performance with varying amounts of demonstration data.

A5: To better understand how the quality and quantity of demonstrations impact overall task performance, we conduct experiments on four MuJoCo tasks under two conditions: lower quality demonstrations and more demonstrations. In each value pair “x/y”, x represents the performance of the demonstrations, while y is the performance of the policy trained with the reward model learned from APEC. The results demonstrate that APEC can still achieve a satisfying performance in tasks with low-quality and scarce demonstrations.