Online-to-Offline RL for Agent Alignment

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

[W1] Utilization of online environment during offline alignment in Game AI

In Game AI applications, a key characteristic is the availability of a game core that facilitates efficient agent training and deployment. During alignment, it is reasonable to assume the continued availability of the game core, as it is typically integral to real-world deployment. For instance, many commercial games rely on persistent game engines to manage online interactions, making such environments accessible during the alignment phase [1,2]. We acknowledge the limitation that this assumption may not hold in scenarios where the online environment is unavailable during alignment. Future work could explore extensions of ALIGN-GAP to fully offline settings, such as using simulated environments or learned models of the environment.

[W2] Extension to wider domains for agent alignment

Game AI provides an accessible and evaluable testbed for human alignment methods due to its structured environments and measurable human preferences (e.g., strategies or play styles). Additionally, Game AI remains one of the primary application domains for reinforcement learning agents. That said, ALIGN-GAP is not inherently restricted to Game AI. Its design can be adapted to other domains, such as autonomous driving or healthcare, with appropriate consideration of domain-specific human preferences, their representation, and evaluation. These broader applications are considered as interesting future works and could also address some the domain-specific challenges of defining and aligning with human preferences in more complex real-world scenarios.

[Q1] Explanation for reward calibration in agent alignment

Reward calibration (Equation 5) serves to explicitly identify and prioritize behaviors preferred by humans but not yet adopted by the agent. This calibration differentiates between trajectories with high human preference scores ($R_\phi(\tau)$) and those where the agent's actions deviate from human preferences ($R_\phi(\tau')$). When the agent becomes aligned with human preferences, $R_\phi(\tau)$ and $R_\phi(\tau')$ converge, resulting in negligible reward shifts and halting further training—exactly as intended. This mechanism ensures that calibration effectively guides the agent during misalignment and stops adjustments once alignment is achieved, complementing the curriculum approach.

[Q2] Analysis on the unlearning phenomenon when shifting between rewards

We agree that rapid shifts in reward signals can destabilize agents and cause unlearning. This phenomenon is explored in Appendix A.2, where we analyze the EPIC distance [3] between the environmental and alignment rewards. The EPIC distance is a metric designed to directly quantify the difference between two reward functions. By designing a reward curriculum, we demonstrate that the EPIC gap can be reduced to $\frac{2}{3}$ of its original value, thereby smoothing the transition and mitigating unlearning. We appreciate your suggestion regarding studies on plasticity loss and will incorporate insights from this field to deepen our understanding of reward signal switching [4,5]. This will guide future refinements to ALIGN-GAP’s design and theoretical analysis, and any new results will be further updated in the final version.

[Q3] Visualization on the alignment results

Using videos indeed provides a more intuitive demonstration of the agent's alignment with human preferences. To supplement the results shown in Figure 9, we have created corresponding visualizations and uploaded them to an anonymous GitHub repository (https://anonymous.4open.science/r/test2-08C1/video.mp4) to showcase ALIGN-GAP's effectiveness in aligning agent behavior with human preferences. In the final version, we will include a more comprehensive collection of video comparisons across additional environments and a broader range of human preferences. This will offer a clearer and more persuasive demonstration of ALIGN-GAP's performance and its ability to align agents with diverse human behaviors. Thank you for your valuable suggestion.

[1] Serious games in digital gaming: A comprehensive review of applications, game engines and advancements.

[2] Game engine architecture.

[3] Quantifying Differences in Reward Functions.

[4] A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning.

[5] Loss of Plasticity in Continual Deep Reinforcement Learning.

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

[W1] Clarification on preference learning for agent alignment

In Section 4.1, we use human trajectories as "winner" samples and agent trajectories as "loser" samples to train the reward model. This binary labeling framework is designed to encode human preferences, a common approach in preference learning literature [1,2]. This setup allows us to encode human preferences effectively, but we acknowledge the concern that some agent trajectories, while not perfectly aligned with human preferences, may still represent successful behavior that should not be penalized.

To address this, ALIGN-GAP employs reward calibration (Equation 5) to distinguish between human-preferred behaviors and agent-learned behaviors. Calibration compares the reward scores of human-preferred and agent trajectories under the same states. When the agent's behavior aligns with human preferences, the scores converge ($R_\phi(\tau) \approx R_\phi(\tau')$), indicating no need for adjustment. In contrast, for misaligned behaviors, calibration amplifies the difference in scores, prioritizing areas that require improvement while preserving effective behaviors. Furthermore, the reward curriculum (Equation 6) ensures that the alignment process proceeds gradually. Early in training, environmental rewards dominate, which helps retain the agent's existing skills, even those not fully aligned with human preferences. As training progresses, the focus shifts towards aligning with human preferences, encouraging the agent to refine and adapt its behaviors. This gradual transition helps prevent unlearning during the early phases of alignment while progressively emphasizing human-aligned behaviors.

[W2] Impact of pretraining for agent alignment

Our design choice of aligning pre-trained agents aligns with real-world Game AI applications, where substantial resources are often already invested in training high-performing agents [3, 4]. Fine-tuning these agents for human preference alignment serves as a post-training iterative step, ensuring that the alignment process builds upon the agent’s foundational skills (similar to LLM pretraining and post-training alignment). In contrast, training a new agent from scratch involves learning both the environmental dynamics and the alignment simultaneously. This dual learning process could result in slower convergence and higher computational demands compared to fine-tuning a pretrained agent that has already mastered the environmental tasks. As an experimental support, we update a comparison in Appendix A.5 between ALIGN-GAP and an RLPD-like online alignment with offline data [5], without prior environment pretraining. Our results show that with equivalent training steps, ALIGN-GAP achieves superior alignment performance. Even when extending the training duration of the RLPD-like approach to match the combined steps of pretraining and ALIGN-GAP fine-tuning, the alignment outcomes remain inferior to ALIGN-GAP. Besides, in the final version, we will include additional ablation studies to systematically analyze the impact of agent pretraining levels on alignment outcomes.

[W3] Computational cost for components in ALIGN-GAP

The additional computational cost of ALIGN-GAP is minimal relative to the overall training process. Reward calibration involves lightweight operations such as forward passes through the reward model, and curriculum learning primarily entails reweighting the reward function, introducing negligible overhead. In our updated Appendix A.6, we compared the number of iterations per second achievable with and without reward calibration or curriculum learning under the same hardware conditions. The results show that incorporating these components does not introduce significant additional computational overhead. To enhance scalability, we could benefit from techniques like model quantization and pruning, which can optimize the reward model for inference during alignment (considering the twice inference in reward calibration) [6,7]. Additionally, grouping similar human preferences through clustering could allow for reusable alignment processes, reducing the need for frequent retraining.

[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

[2] Provable Offline Preference-Based Reinforcement Learning.

[3] A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges.

[4] Pre-trained Language Models as Prior Knowledge for Playing Text-based Games.

[5] Efficient Online Reinforcement Learning with Offline Data.

[6] Towards Efficient Post-training Quantization of Pre-trained Language Models.

[7] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.

Dear Reviewer AHtP,

We appreciate all of the valuable time and effort you have spent reviewing our paper. As the discussion period is ending, we gently request that you review our reply and consider updating your evaluation accordingly. We believe that we have addressed all questions and concerns raised, but please feel free to ask any clarifying questions you might have before the end of the discussion period.

Best,

Authors

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

[W1] Justification for fine-tuning Instead of re-training, and comparison with AlignDiff

1. Justification for Fine-tuning Instead of Training a New Agent: Our work addresses the challenge of aligning well-trained agents with human preferences as a post-training iterative step, which assumes the existence of a pre-trained policy. This approach reflects the practical realities of real-world Game AI applications, where significant computational and resource investments are made to develop high-performing agents [1,2]. Fine-tuning these agents for alignment leverages their existing understanding of the environment, enabling efficient adaptation to human preferences. Re-training an agent from scratch like RLPD may suffer from slower convergence and longer training times compared to fine-tuning an existing agent that has already mastered environmental dynamics. To validate this claim, we update a comparison in Appendix A.5 between ALIGN-GAP and an RLPD-like online alignment with offline data settings. Our results show that with equivalent training steps, ALIGN-GAP achieves superior alignment performance. Even when extending the training duration of the RLPD-like approach to match the combined steps of pretraining and ALIGN-GAP alignment, its performance remain inferior to ALIGN-GAP.

2. Comparison to AlignDiff and Attribute Design: In AlignDiff, attributes are explicitly used as language commands to train both the attribute model and the diffusion model. In contrast, we employ data clustering techniques (e.g., using reward oracles or UMAP) to group human preference data by styles during preprocessing. As described in Section 4, ALIGN-GAP requires only preference data with specific styles for alignment and does not rely on additional attribute information. The use of reward oracles or UMAP is solely for preprocessing the data. This design makes ALIGN-GAP more flexible and better suited for aligning agents with specific user groups or individual preferences, as it avoids reliance on domain-specific attribute definitions. In applications, if a game designer aims to align an agent with the play style of a particular user group or even a single player, ALIGN-GAP can use the collected data from that user as human preference data without additional attribute designs.

[W2] Expand the range of experiments to include more game environments

We agree that diverse environments are important to demonstrate generality. While our experiments primarily focus on MuJoCo locomotion tasks and Atari games, these are widely used benchmarks in reinforcement learning research for Game AI due to their well-defined task dynamics and popularity in prior studies [3,4]. To address the concern, we have extended our experimental evaluation to include the Alien game to examine the alignment capability of ALIGN-GAP, shown Appendix A.3. In future work, we aim to extend ALIGN-GAP's applicability by exploring alignment in more complex, real-time strategy games with real-time human interactions. We will update the observations in the final version and future iterations of our work to broaden the applicability of ALIGN-GAP.

[Q1] Explanation on $\lambda$

In our experiments, $\lambda$ is set to control the transition of $\alpha(t)$ from 0 (environment reward focus) to 1 (reward model focus) as curriculum learning progresses. Specifically:

• Linear $\alpha(t)$: As detailed in Section 4.2, $\alpha(t) = \frac{\text{current\ step}}{\text{total\ steps}}$ ensures a smooth, linear increase.
• Exponential $\alpha(t)$: We empirically set $\lambda = 5$, which is a positive constant to ensure that $\alpha(t)$ follows the trend of increasing from 0 to approaching 1. We have updated the setting for $\lambda$ in the revised version for clarity.

[Q2] The scale of $R_\text{calibrated}$

Yes, empirical results demonstrate that $R_{\text{calibrated}}$ obtained via subtraction performs robustly across environments without additional scaling. The reward model's score ranges, observed to be approximately [-8, 8] across tasks (e.g., Walker2D in Figure 4(b)), remain consistent in different environments, which aligns with findings in related reward model alignment studies [5]. We acknowledge that exploring environment-specific scaling RM scores adaptively could enhance performance in future work.

[1] A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges.

[2] Pre-trained Language Models as Prior Knowledge for Playing Text-based Games.

[3] Offline Reinforcement Learning with Implicit Q-Learning.

[4] A Review for Deep Reinforcement Learning in Atari:Benchmarks, Challenges, and Solutions.

[5] Secrets of RLHF in Large Language Models Part II: Reward Modeling

I deeply appreciate the author’s effort to address my question and conduct additional experiments.

[finetuning vs retraining]

I apologize for not explaining clearly earlier, which might have caused some misunderstandings. "RLPD-like" was just one example I mentioned, but what I was more curious about was what happens if, in addition to using an RLPD-like method, we train from scratch offline using AlignDiff. (I understand the differences between the proposed method and the "RLPD-like" method. Thank you for the response.)

As I understand it, the current method involves training a reward model by combining agent trajectories generated by the online-trained agent with offline human trajectories. This reward model is then used to fine-tune the online-trained agent. However, my thought was: why not use the combined data of the same agent trajectories and offline human trajectories to train a new agent with freshly initialized weights directly using AlignDiff? Alternatively, if offline human trajectories are available, wouldn't it suffice to simply train AlignDiff on those?

I agree with the author’s comment that training from scratch is naturally more costly than fine-tuning, especially in environments like game applications. However, if fine-tuning causes issues such as an irrecoverable performance drop, as discussed in many existing offline-to-online RL studies, then training from scratch might be a stable alternative worth considering despite the cost. Furthermore, while the papers cited by the author deal with truly costly domains, I don’t believe the MuJoCo or Atari experiments presented in this paper represent environments where the tradeoff would definitively justify the conclusion that fine-tuning is always preferable.

I would like to understand the differences between and the pros and cons of the following approaches:

(1) RL agent training → Fine-tuning the RL agent using its rollout data + human preference data

(2) RL agent training → Training a new agent using AlignDiff (with reward oracle or UMAP attributes) on the agent's rollout data + human preference data

(3) Training AlignDiff (with reward oracle or UMAP attributes) using only human preference data

 

[methods for defining attributes]

Additionally, in MuJoCo, the environment preference reward is defined based on three dimensions: maintaining health, moving forward, and controlling cost. It would be ideal to extract and use reward oracles solely from offline data without domain knowledge. However, if that’s not feasible, it seems this would also require domain knowledge to define, making it essentially similar to defining attributes in AlignDiff. I now clearly understand the differences in attribute definitions between AlignDiff with utilizing UMAP.

 

I find the problem of aligning trained agents with human preferences both important and fascinating. However, the proposed method seems somewhat intricate in its design to address issues from online fine-tuning, which makes me wonder whether this complexity is necessary or if alternative approaches might be possible. While offline and online fine-tuning can be seen as fundamentally different, I feel that a more comprehensive comparison with prior works focused on aligning agents with human preferences would be valuable.

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

[W1,Q2] Range of experimental environments and theoretical analysis for switching between different rewards

We evaluates ALIGN-GAP on two widely-used reinforcement learning benchmarks: D4RL locomotion tasks (e.g., HalfCheetah, Walker2D, and Hopper) and Atari games (e.g., Pac-Man and Space Invaders). These environments were chosen due to their prevalence in prior research and their diverse characteristics, allowing us to test ALIGN-GAP across a range of alignment challenges [1,2]. To further strengthen our argument, we have now extended ALIGN-GAP on the Alien game, as detailed in Appendix A.3. In future work, we aim to explore alignment in more complex, application-oriented game scenarios with real-time human interactions to expand the scope of our method. We will update the observations in the final version and future iterations of our work to broaden the applicability of ALIGN-GAP.

In Appendix A.2, we present an analysis of the impact of our curriculum learning strategy on alignment. Specifically, we formally demonstrate that introducing a curriculum reduces the expected Equivalent-Policy Invariant Comparison (EPIC) distance [3] between the environmental reward and the alignment objective to $\frac{2}{3}$ of their original distance. The EPIC distance is a metric designed to directly quantify the difference between two reward functions without requiring a policy optimization step. It is proven to be invariant within an equivalence class of reward functions that induce the same optimal policy, making it a robust measure for this purpose. This reduction in EPIC distance supports the theoretical stability of switching between environmental and human-aligned rewards and highlights the effectiveness of the curriculum in guiding alignment.

[Q1] Data requirements for alignment with ALIGN-GAP

ALIGN-GAP is designed to work effectively with small amounts of human preference data, as motivated in Section 1, addressing practical challenges of data collection costs and availability. For example, in Section 5.1, we show that ALIGN-GAP achieves effective alignment with as few as 10 episodes of human data. This efficiency is attributed to the reward model's ability to encode human preferences even from sparse data. Regarding noisy or low-quality data, ALIGN-GAP's reliance on pairwise comparisons in training the reward model provides inherent robustness (Section 5.4). However, excessively noisy or poorly representative data could degrade alignment performance, as with any preference-based RL methods.

[Q3] Computational cost and efficiency for components in ALIGN-GAP

The additional computational cost of ALIGN-GAP is minimal compared to the overall training process. Key components such as reward calibration and curriculum learning primarily involve reward computations, which add negligible overhead relative to the neural network training for the agent. As shown in Appendix A.6, we compared the number of iterations per second achievable with and without reward calibration or curriculum learning under the same hardware conditions. The results show that incorporating these components does not introduce significant additional computational overhead. For resource-constrained applications, techniques such as model quantization and pruning can further reduce the computational burden of the reward model, making ALIGN-GAP practical and efficient even in such scenarios [4,5].

[Q4] Evaluation metrics for degree of alignment

We agree that alignment evaluation inherently involves some subjectivity due to the nature of human preferences. Currently, we assess alignment by measuring performance similarity to human proxies (Table 1 and 2) and visualizing agent behavior styles (Figures 5–9). These methods align with standard practices in LLM and agent alignment research, which often rely on reward model scores or human evaluations (e.g., win rates, subjective ratings) [6]. Besides, we have created corresponding visualizations for the observations in Figure 9 (https://anonymous.4open.science/r/test2-08C1/video.mp4) for a clearer presentation. To enhance evaluation robustness, we will explore combining subjective measures with more objective metrics in future work.

[1] Offline Reinforcement Learning with Implicit Q-Learning.

[2] A Review for Deep Reinforcement Learning in Atari:Benchmarks, Challenges, and Solutions.

[3] Quantifying Differences in Reward Functions.

[4] Towards Efficient Post-training Quantization of Pre-trained Language Models.

[5] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.

[6] Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment.