Learning to Reuse Policies in State Evolvable Environments

Thank you for the constructive comments and suggestions. We provide our responses below and present the additional experimental results in https://anonymous.4open.science/r/Lapse-7846.

Q1. Performance of baselines

The baselines underperform because they cannot anticipate all possible state evolutions during training. Instead, they must adapt to new state space using only a small set of paired transitions collected by $\pi_n$. In general, domain adaptation methods require a large amount of data to extract generic representations for adaptation. CUP cannot enumerate all possible evolutions beforehand, leading to failure for direct reuse of source policies. We will expand this analysis in Sec. 4.2.

Q2. Linear Mixing

Linear mixing is an efficient ensemble method, which has been applied in policy ensemble [1, 2]. We employ linear mixing because determining the weight via Eqn. 9 and performing linear mixing is cost-free and highly-efficient. We conduct experiments on how the way of ensemble affects the performance of Lapse. Alternatives like hypernetworks showed no significant improvement (Figure 1). We also train stacking but it failed in 4 tasks (score < 1000 after 1M steps).

Q3 TV divergence

Since $D_{TV}^2\leq \frac{1}{2}D_{KL}$, both divergences can achieve robustness. On the other hand, the loss is derived from Prop. 3.3, where performance gap is bounded by TV divergence. As discussed in Appx. D.1, the methods we use [4] do not directly optimize over TV divergence but take more efficient ways.

Q4 New data, cost and algorithms

In SERL formulation, the policy is deployed after training on the initial MDP $M_0$. During deployment, the state space may evolve for various reasons such as scheduled maintenance. Access to the information is motivated by the fact that real-world systems undergo scheduled sensor checks or replacements. For instance, if a sensor is nearing the end of its lifespan, a replacement can be installed in advance, allowing the agent to observe features from both old and new state spaces. The new dataset $D_n$ is collected by $\pi_n$ utilizing only state features $s_n$ from the old state space, but paired states $s_{n+1}$ can also be reserved.

Our method will not incur significant cost as the policy for data collection is $\pi_n$ which performs well in $M_n$. Avoiding costly trial-and-error during policy adaptation is the primary contribution of Lapse.

Since $\pi_{n+1}^{off}$ is learned using data collected by $\pi_n$, we choose offline RL algorithms. Online RL including off-policy algrorithms is not suitable because trial-and-error is not allowed in SERL.

Q5 Continuously learn a robust policy or direct map to evolution 0

When the state space evolves to $S_{n+1}$, the depolyed policy $\pi_{n+1}$ is composed of both $\pi_{n+1}^{recon}$ and $\pi_{n+1}^{offline}$, where the latter does not require state reconstruction and maintains robustness via regularization.

In Appx. E.2, we employ pruning strategies and verify that the initial policy can be omitted via proper pruning in some scenarios, without the need for iterative map. Meanwhile, we conduct experiments by mapping evolution n to 0 via cycleGAN. Figure 2 shows our method performs better than directly aligning in most circumstances as it can better utilize the paired data.

Q6 Paired data and possible enhancement

Despite the limitations in some scenarios, it is still applicable for many real-world systems. Please refer to our response in Q2 to Reviewer Uqfd for details due to word limit.

The idea that automatically pairing data is a valuable direction for further research. Cycle-GAN [6] is widely-used for domain adaptation by introducing cycle consistency loss. It implicitly pairs data from different domains. However, as the experimental analysis presents in Figure 2, it fails to achieve satisfying adaptation performance with direct application. In future works, developing relevant methods for automatically paring using a small set of data will help extend our method to more general scenarios.

Q7 State features, space and observation space

In our work, state space is the fundamental element $\mathcal S_n$ in the MDP $\mathcal M_n$, while state features $s_n \in \mathcal S_n$ are the input to the policy for decision making. The change of sensors lead to the evolution of state space, together with state features. There is no distinguishment between “observation” and “state” because we do not consider the partially observable setting. We are sorry for confusion caused by some abuse of these concepts. We will replace observation into state for consistency and clarify two concepts in the revised version.

Q8 Nitpicks

Thank you for pointing out the issues.

[1] Towards Applicable RL: Improving the Generalization and Sample Efficiency with Policy Ensemble

[2] Seerl: Sample efficient ensemble RL

[3] Efficient adversarial training without attacking: Worst-case-aware robust RL

Thank you for careful review our paper and providing constructive comments and suggestions. We offer some clarification to your questions here, and we would appreciate any further comments you might have.

Q1 Changes are directly signaled to Lapse

Yes, the change in the environment should be directly signaled in our current SERL problem formulation. Access to sudch information is motivated by the fact that many real-world systems undergo scheduled sensor checks, replacements, or recalibrations, where developers can prepare beforehand. For example, large-scale industrial systems often have planned maintenance schedules for updating or replacing sensors [1]. These scenarios allow developers to prepare for changes in advance, making our signaling assumption reasonable for practical deployment.

Q2 Lack of essential references

Thank you for bringing the references to our attention. The works referenced by the reviewer [2, 3, 4] and our Lapse all focus on the open-world/environment decision making problem. The sudden changes of environments are defined as “novelty” in these works, which emphasizes the abruptness of the change, while the “evolution” of state space in Lapse, although still unpredictable before deployment, can be scheduled beforehand for a short period. Specifically, [2, 3] detect the sudden change via knowledge graph or neuro-symbolic world model and achieve policy adaptation with imagination trajectories generated from the updated world model. These methods fail to be directly applied to high-dimensional, continuous and complex environments, where dynamics and rewards are difficult to be depicted with simple rules. [5] addresses this problem by extending it to latent-based novelty detection via DreamerV2 but does not study how to achieve policy adaptation in complex environments, mainly because the accuracy of imagination trajectories cannot be guaranteed. Our method takes a further step for the high-dimensional, continuous and complex environments via a novel solution, and we hope it could fill in gaps for open-environment.

Aside from the works mentioned by the reviewer, some other works focus on dealing with sudden changes in decision making, involving environmental dynamics [5] and teammate in multi-agent systems [6]. We will add the corresponding discussions to the paper in the revised version. Thank you again for pointing out the problem.

[1] A Methodology for Online Sensor Recalibration

[2] Detecting and Adapting to Novelty in Games

[3] Neuro-Symbolic World Models for Adapting to Open World Novelty

[4] Novelty Detection in Reinforcement Learning with World Models

[5] Adapt to Environment Sudden Changes by Learning a Context Sensitive Policy

[6] Fast Teammate Adaptation in the Presence of Sudden Policy Change

Thank you for your careful review and thoughtful comments. We hope our responses can relieve your concern. We present the additional experimental results in https://anonymous.4open.science/r/Lapse-13F8.

Q1: Prior knowledge about sensor changes

While this assumption does not hold universally, it is practical in many real-world systems. For instance: Wear-and-tear or battery depletion in industrial sensors is often predictable, allowing preemptive deployment of replacements.

Meanwhile, the assumption of accessing a large amount of test-domain data is common in domain adaptation [1]. In our work, the prior knowledge about sensor changes only offers a small set of paired test-domain samples.

Q2: Access to maintenance information

Access to maintenance information is motivated by the fact that many real-world systems undergo scheduled sensor checks, replacements, or recalibrations [2]. The developers are thus allowed to capture paired data before sensor changes, which is essential for our method to adapt policies effectively with minimal online exploration. For example, large-scale industrial systems often have planned maintenance schedules to update or replace sensors [3]. The system should keep operating during the aforementioned periods.

The problem pointed by the reviewer is valuable as there are many abrupt changes in the real world, which is also pointed out as the limitation of Lapse in conclusion part. Some works have also focused on dealing with sudden changes in dynamics [4]. However, these methods are only applicable for in-distribution changes and fail to cope with totally unseen scenarios. Online fine-tuning based adaptation is an alternative direction without the need of maintenance information, but it inevitably introduces costly trial-and-error. In future work, combining Lapse with these methods by restricting the policy adaptation in a safe and trustworthy region is of promise.

Q3: Changes of dynamics and rewards

Our work mainly focuses on the continual evolution of state space, which is built on SERL formulation. Paired states cannot be created if dynamics/rewards change, so we fail to directly apply Lapse to such scenarios. However, to investigate the potential of Lapse, we simply extend it to a more complex multi-agent scenario, SMAC, by combining it with a multi-task MARL method, MaskMA [5]. As the paired transitions do not exist, we assume access to a limited offline dataset when encountering new tasks. Figure 1 demonstrates Lapse's potential for adaptation to unseen tasks when combined with multi-task/meta learning methods.

Q4: Real-world experiments

To further investigate how Lapse performs in real-world applications, we here choose Highway-env as alternatives for real-world applications including autonomous driving. As shown in Figure 2, Lapse consistently outperforms other baselines in adaptation performance. For more realistic physical embodied scenarios involving robotics, we leave it for future work.

Q5: Lack of complete training curves, compute analysis and ablation study

1. We list out the complete training curves in the Figure 3, where blue dashed lines represent the performance when performing new paired data collection. It should be noticed that the policy is deployed to the environment for the next evolving stage until finishing the training, thus preventing costly trial-and-error during adaptation.
2. We present the compute analysis during training in Table 1 by computing the number of parameters required for training and FLOPs during each stage. Our method demonstrates better performance with no significant extra computation cost.
3. The ablation study that demonstrates what the addition of each component does is presented in Figure 4, verifying the necessity of each component.

[1] A Comprehensive Survey on Source-free Domain Adaptation

[2] Open-world machine learning: applications, challenges, and opportunities

[3] A Methodology for Online Sensor Recalibration

[4] Adapt to Environment Sudden Changes by Learning a Context Sensitive Policy

[5] MaskMA: Towards Zero-Shot Multi-Agent Decision Making