Test-time Adapted Reinforcement Learning with Action Entropy Regularization

Q1. The improvements reported on most tasks are relatively marginal. This raises questions about the practical significance and robustness under different conditions and task complexities.

A1. Thanks for your valuable feedback. The marginal improvements should be interpreted through four aspects.

1. Our performance reflects a trade-off between efficiency and performance. TARL updates only LN parameters during testing, requiring no gradient updates to the full network.

2. Unlike online RL, TARL operates purely on test samples without reward signals, making it suited for safety-critical applications where exploration is prohibited. TARL preserves the stability and safety of conservative offline RL.

3. TARL shows improvements across discrete (Atari) and continuous (D4RL) control tasks, which indicates its robustness.

4. We study more extensively across multiple datasets to validate our effectiveness. From Table D, TARL achieves significant improvement.

Table D. Comparison of TARL against the baseline.

Q2. The motivation for only fine-tuning the layer normalization parameters is not entirely clear. Why was this particular choice made?

A2. The test-time phase requires efficient model parameter updates to ensure the model can adapt to the new environment promptly. Thus, TARL only updates parameters in layer normalization layers of the model, ensuring computational efficiency without full model gradient updating. This allows for rapid adaptation while maintaining model stability.

Q3. In Table 1 and 2, are the baselines also fine-tuned with test data?

A3. The baselines do not fine-tuned with test data.

Q4. This work addresses the issue of the transfer gap between offline learning and online environments. There are several offline-to-online algorithms[1][2] that seem very related to this work. Shouldn't these algorithms be compared within the study?

A4. Comparing TARL directly with online RL methods isn't appropriate as TARL is an offline RL method. TARL and offline-to-online RL operate at different levels.

When offline RL methods run in online environments, they will suffer from OOD issues. The core motivation of TTA is that environmental dynamics and distribution shifts cause performance drops in offline-trained models. TARL aims to maintains stability amidst data distribution changes. TARL fine-tunes the policy during the test-time phase through entropy minimization, without needing environment feedback. It effectively adapts to changes in test data distribution and ensures efficient updates. In contrast, offline-to-online RL algorithms[1][2], depend on environment feedback for policy updates. However, in some environments acquiring rewards can be timeconsuming or even infeasible. Therefore, offline-to-online RL cannot update policies.

We will include and discuss relevant literatures in the revised manuscript.

[1] Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. NeurIPS, 2021.

[2] Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. PMLR, 2022.

Q5. While CQL is a classic offline RL algorithm, it would be valuable to apply the proposed method to more powerful algorithms, such as DT[3] and IQL[4].

A5. Accoding to your suggestion, we present a comparative analysis of IQL[4] performance in Table E. Even on the expert, we still achieve improvement.

Table E. Effectiveness of TARL applied to IQL.

However, applying TARL to DT[3] would require higher workload due to the inherent differences in the Transformer architecture. Given the time constraints of this submission, we are unable to complete the experiments. We will explore this direction in future work.

[3] Decision transformer: reinforcement learning via sequence modeling. NeurIPS 2021.

[4] Offline reinforcement learning with implicit q-learning. ICLR 2022.

Q6. Curiou about the sensitivity of the proposed TARL to the hyperparameters. The detailed hyperparameter settings for different tasks should be included for clarity.

A6. We use two distinct sets of hyperparameters for discrete control and continuous control tasks, respectively. All environments of the same type of task share the same hyperparameters. This further highlights the strong generalization capability of TARL. We have show hyperparameter settings in Section 4.4 and would make them more detailed and clearly.

For important hyperparameters entropy threshold $E_0$ and the KL divergence $\lambda$, we have conducted ablation studies in Section 4.7. The TARL is sensitive to $E_0$ because a small $E_0$ is crucial for selecting high-confidence samples.

---

We sincerely hope our clarifications above have addressed your questions.

Q1. In Line 70, the authors claim that they propose a novel offline reinforcement learning paradigm. However, the method belongs to a test-time adaptation method. Is there a conflict between offline reinforcement learning and test-time adaptation? I suggest further clarification on this.

A1. Thank you for your valuable feedback. Offline reinforcement learning and test-time adaptation are not conflicting concepts. TARL is a framework designed for test-time adaptation of offline reinforcement learning methods. Offline RL learns from pre-collected data without interacting with the real-time environment. TARL fine-tunes the offline-trained policy model using test data to better fit real-world conditions. It is important to note that TARL requires no environment interaction or feedback during adaptation. As a result, Offline RL algorithms optimized by TARL with test-time adaptation retain the offline characteristic, maintaining its efficiency and safety while improving real-world performance.

Q2. Continuous control tasks assume that the action obeys a normal distribution. However, the actions in real applications may not be strictly normal. Can the TARL method based on the normal distribution assumption still work?

A2. In practice, TARL remains effective even when the action distribution is not a strictly normal distribution. The core principle of TARL is to reduce policy uncertainty during action selection, thereby improving stability and adaptability. This process does not depend on any specific state-action distribution. For non-normal distributions, entropy minimization still encourages the policy to favor more confident actions, leading to better performance.

Q3. This paper mentions that there is a distribution bias between offline training and online testing in the abstract. In the experiments, where is this bias specifically reflected?

A3. In our experiments, the distributional shift primarily manifests as a mismatch between state-action distributions in offline training datasets and evaluation environments. This OOD stems from the sampling incompleteness problem: While the state-action space is theoretically infinite, offline datasets can only capture a finite subset through sampling. Consequently, the policy inevitably encounters unobserved state-action combinations during real-world deployment, highlighting the critical need for test-time adaptation mechanisms like TARL.

Q4. The details of the Atari and D4RL benchmarks are not clear. For example, the environment and the agent setting are unknown. It would be better to add more descriptions about the two benchmarks.

A4. We provide additional details regarding the benchmarks used in our experiments.

Atari Benchmark: For discrete control tasks, we evaluate policy performance on five representative Atari environments: Qbert, Seaquest, Asterix, Pong, and Breakout. These games provide diverse challenges through distinct mechanics, such as spatial reasoning in Qbert and reactive control in Pong.

D4RL Benchmark: For continuous control tasks, we employ three standardized D4RL dataset configurations: Medium Policy (suboptimal policy-generated data), Medium Replay Buffer (mixed-quality trajectories), and Medium-Expert (hybrid expert-novice demonstrations). We evaluate TARL on three locomotion tasks: bipedal locomotion (HalfCheetah), monopedal jumping stability (Hopper), and dynamic balance maintenance (Walker2D).

Consistent Experimental Settings

Our implementation strictly follows original offline RL methodologies for fair comparison. For example, CQL(TARL) maintains identical network architectures and training hyperparameters to vanilla CQL.

Q5. The performance of TARL depends on selecting appropriate entropy thresholds ($E_0$) and KL divergence weight ($\lambda$).

A5. The selection of appropriate entropy thresholds $E_0$ and KL divergence weights $\lambda$ is critical to the performance. This underscores the necessity of the data filtering mechanisms and debiasing regularization introduced in our method. A suitable $E_0$ ensures that only high-confidence samples are retained, thereby preventing the introduction of noise from out-of-distribution data and preserving the stability and reliability of policy updates. Similarly, the selection of an optimal KL divergence weight $\lambda$ plays a key role in facilitating a smooth transition of the offline policy to the online environment.

---

We sincerely hope our clarifications above have addressed your questions.

Q1. Your paper seems to rest on the hypothesis that selective training on low-entropy states enables beneficial knowledge transfer to other similar states. The paper should provide ablation studies comparing selective entropy minimization against global entropy minimization.

A1. Thank you for your valuable feedback. The rationale for focusing on low-entropy states operates on two levels:

1. Theoretical Motivation. Low entropy directly reflects reduced action uncertainty. States with low-entropy correspond to more deterministic and reliable decision regions in the policy space. By prioritizing these low-entropy states during test-time adaptation, the policy can establish stable decision boundaries while avoiding OOD extrapolation.

2. More study on the advantage of selective training on low-entropy states. The experiments were conducted in the walker2d-medium-v2 task to evaluate the hypothesis that selective training on low-entropy states improves TTA performance. The experimental setup included three conditions:

• Global Entropy Minimization, where all available data were used for test-tiem adaptation.
• Low-entropy Selective Training, where only samples with entropy below a predefined threshold were used for training.
• High-entropy Selective Training, where only samples with entropy over a predefined threshold were used for training.

Our results demonstrate that selective training on low-entropy states improves TTA performance more effectively than global entropy minimization. Meanwhile, if we only select high-entropy samples for tta, the performance actually becomes worse. This further indicates that selective training on low-entropy states enables beneficial knowledge transfer.

Table C. Effectiveness of low-entropy selective training.

Q2. Have you conducted experiments comparing TARL to simpler alternatives that also reduce policy entropy: a) argmax action selection, b) globally reduces temperature in the Boltzmann distribution? If so, what were the results? If not, why were these fundamental comparisons omitted?

A2. Our study does not include comparative experiments between TARL and simpler entropy-reduction approaches like boltzmann or argmax action selection, primarily due to their incompatibility with test-time adaptation requirements.

1. Characteristics of Test-Time Adaptation

During the test-time phase, the model has already been trained. The primary goal is fine-tuning the model using test data to adapt to the actual environment. The key characteristics of this phase is that the model cannot access environmental reward signals or other forms of feedback. This means the model cannot rely on environment feedback to guide policy updates.

2. Limitations of Boltzmann and Argmax Policies

The Boltzmann policy is an action selection method that uses the softmax function to convert action values into a probability distribution. During training, the Boltzmann policy requires environment feedback to adjust the temperature parameter and action value function. The argmax policy is an action selection method that directly selects the action with the highest value. During training, the argmax policy needs environment feedback to evaluate and update the action value function. In the test-time phase, the lack of environment feedback makes them impossible to effectively update these parameters.

3. Advantages of TARL

TARL has no reliance on environment feedback. It uses entropy minimization as the objective function for policy updates during the test-time phase, effectively updating the policy. By minimizing entropy, TARL enhances the selection of high-confidence actions and reduces reliance on uncertain actions, better adapting to changes in the distribution of test data.

Q3. Can you provide controlled experiments in simpler environments that explicitly demonstrate the hypothesized learning transfer from low-entropy states to related states?

A3. We have conducted ablation studies on the effect of low-entropy states selection in Section 4.7. $E_0$ controls the selection of samples for updating. Specifically, samples with entropy below this threshold are selected for updating. As $E_0$ decreases, the selected samples have lower entropy and fewer in number. When the entropy threshold increases, the performance of test-time adapted RL declines, which shows that it is necessary to filter some samples with relatively high confidence to update the offline policy to adapted the online environment.

---

We sincerely hope our clarifications above have addressed your questions.

Q1. Implementing the TARL framework on top of other offline RL algorithms, such as IQL.

A1. Thank you for your valuable feedback. The state-action out-of-distribution (OOD) issues stem from data distribution shifts between training and testing phases, rather than being specific to particular offline RL algorithm. Our proposed TARL method is specifically designed to mitigate OOD interference in offline RL algorithms, thereby significantly enhancing stability during testing phases.

As a foundational algorithm in offline reinforcement learning, IQL [1] will suffer from this fundamental challenge during the testing phase. In Table A, we present a comparative analysis of IQL performance on Walker2d-medium-v2 environment and Walker2d-expert-v2 environment before and after TARL. Our approach achieving improvement in terms of average episode return.

Table A. Effectiveness of our TARL applied to IQL.

[1] Offline reinforcement learning with implicit q-learning. ICLR 2022.

Q2. Conducting experiments on other datasets included in the D4RL benchmark, such as the AntMaze environments and the replay and expert datasets in Gym environments.

A2. Accoding to your suggestion, we study more extensively across multiple benchmark datasets, including the AntMaze environments and the replay and expert datasets in Gym environment, to rigorously validate the effectiveness of our method. From Table B, our TARL achieves better performance than original RL algorithms, with particularly notable improvements in the complex environment Antmaze-medium-diverse-v0 and the expert-level task Hopper-expert-v2. These results further demonstrate the advantages of our method in handling data noise, distribution shift, and high-complexity tasks.

Table B. Average episode return comparison of TARL against baseline methods on D4RL benchmarks over the 10 evaluations.

---

We sincerely hope our clarifications above have addressed your concerns.