Mitigating Distribution Shifts: Uncertainty-Aware Offline-to-Online Reinforcement Learning

Is there a performance or computational advantage of UARL over direct processing of $E_\omega$'s using the Robust RL algorithm or the algorithm with domain randomization techniques? Can this be illustrated experimentally?

UARL is designed to handle OOD scenarios without relying on direct interactions with the target environment. Unlike Robust RL or domain randomization, which often require extensive data or access to the full distribution of environments, UARL focuses on modeling uncertainty to generalize effectively across diverse conditions. The goal is to minimize dependence on refining a policy on real-world data, making it safer and more practical for applications where direct interactions with the target environment are risky or infeasible.

Notice that in offline training (1st iteration), EDAC performs much worse than CQL and TD3+BC in many environments, which doesn't seem to match the experimental results in the EDAC article?

We believe this discrepancy may be due to EDAC's potential overfitting to the original D4RL dataset. As mentioned in Section 5, for all experiments and all baselines, we used the implementation and hyperparameters provided at https://github.com/corl-team/CORL, applying the same consistency across other algorithms like EDAC, CQL and TD3-BC.

The experiments in this paper were all performed in Mujoco, how do we obtain the real-world demonstration dataset $\mathcal{D}_\omega$ in the simulation environment like Mujoco?

Since Mujoco is very different from real-world scenarios, there is no real data. In our current experiment, the real-world demonstration dataset $\mathcal{D}_w$ is obtained by running the “expert” policy on the environment with a wide range of values for randomized parameters that differ from the original Mujoco environment, which acts as “real-world data”.

We should have clarified this point further in the paper. Thanks for bringing it into our attention. We included it in the revision, Section 5.

Thank you for your thorough and thoughtful review of our submission. We greatly appreciate the time and effort you put into providing detailed feedback and suggestions to enhance our work.

It should be further shown, either theoretically or experimentally, why the diversity term $\mathcal{L}^{\text{RL}}_{\text{div}}$ in Eq. 5 allows the ensemble Q network to learn the value of diversity.

Indeed, in the worst case, it would be possible for all members of the ensemble to converge to the same value Q’(s,a) which would differ from r + Q(s,a), and the loss would be satisfied. However, because Q is high dimensional and each member of the ensemble was initialized independently, this case is extremely unlikely and does not happen in practice. Note that this was already true in DENN. Therefore, we do in practice obtain a high diversity among the members of the ensemble. You can verify this directly in Fig. 4: the critic variances for UARL are all >> 1 OOD and are much larger than the critic variances ID, indicating that in practice all the members of the ensemble differ. Note that if they did not differ, the critic variance would be closer to 0, and if UARL was not useful, the critic variance OOD would be much closer to the critic variance ID, as is the case for AWAC (first row of Fig 4). Finally, this result remains consistent as we change the definition of ID and OOD (columns 2 and 3).

Further clarification is needed for how to calculate $V_Q$ in Algorithm 2 and how to calculate critical variance in uncertainty estimation experiments

$V_Q$ is calculated simply as the variance of the outputs from the ensemble of critics. For any given state-action pair $(s, a)$, each critic in the ensemble $(Q_1, ..., Q_N)$ will output an estimate of the Q-value, represented as $Q_1(s, a), ..., Q_N(s, a)$. The variance ($V_Q$) is then computed from these Q-value estimates, reflecting the disagreement or uncertainty among the critics for that specific state-action pair. To calculate the variance, we just follow the variance calculation equation: $V_Q = \frac{1}{N-1} \sum_{i=1}^{N} (Q_i(s, a) - \bar{Q}(s, a))^2$ where $\bar{Q}(s, a)$ is the average of the Q-values over the ensemble, $\bar{Q}(s, a) = \frac{1}{N} \sum_{i=1}^{N} Q_i(s, a)$. Given this basic variance formula, the calculation is straightforward and well-established in statistics.

The ablation experiments in Appendix B.3 are not detailed enough

We understand the concern about the readability of the curves and appreciate the suggestion to differentiate the parameter combinations more clearly. Our primary goal with the plots was to convey the effect of different hyperparameter settings on the algorithm's performance without performing extensive hyperparameter tuning. To avoid clutter, we chose to represent all curves with the same color and made the line corresponding to the chosen hyperparameters slightly darker to make it distinguishable. Given the large number of curves (15 in total) with standard deviation as shaded areas, introducing additional colors would likely make the plot overwhelming and harder to interpret.

For each $E_i$, the randomized environmental hyperparameter range is determined without a common metric but as a hyperparameter, which may require a lot of time for online tuning for complex scenarios?

In UARL, the randomization ranges for environmental hyperparameters are not manually tuned or determined via trial and error. Instead, we rely on a dynamic and adaptive approach where the range of each parameter is adjusted based on the agent’s uncertainty in the environment. This process allows the system to automatically adjust the randomization ranges without requiring extensive online tuning. In the paper, we ensure that the parameters are varied in a way that introduces meaningful uncertainty for OOD detection while avoiding excessive complexity or destabilization of the agent. Therefore, while the approach may seem complex, it does not require significant manual intervention or exhaustive online tuning.

According to Eq. 5, $R(s, a)$ is not sampled from the dataset $\mathcal{D}$, but the general Q-function update for offline RL, as in Eq. 1, is to use the sampled $r$. Is this a mistake here?

Thanks for bringing this to our attention. It is indeed a mistake. We fixed it in the revision.

in Equation 5, you wrote $R(s,a)$ in the bellman error, while $r$ in the diversity term $\mathcal{L}_{div}^{RL}$. I think they should be identical, right?

Thanks for bringing this to our attention. It is indeed a mistake. We fixed it in the revision.

the authors do not discuss the limitations of their method in the main text or the appendix

We addressed the limitations of UARL in the Conclusion and provided a more detailed discussion in Appendix D of the revised version.

the performance improvement of UARL seems limited and incremental on some tasks (e.g., see Figure 3)

That is a valid observation, as already noted in lines 454 and 455. However, we would like to emphasize again that UARL is not primarily focused on performance improvements. Its main purpose is to assist with OOD detection. The point made in Section 5.1 is to demonstrate that the introduction of UARL does not negatively impact the performance of the underlying algorithms.

UARL can still suffer from performance degradation during the fine-tuning phase (e.g., see Figure 5)

This is also a valid observation; however, the balanced replay buffer mechanism in UARL (Section 4.3) results in less performance degradation compared to baseline methods, as shown in Figure 5.

In Lines 184-185, you wrote, disagreement among ensemble models, particularly at the boundary of the training distribution, what do you exactly mean by at the boundary of the training distribution?

By "at the boundary of the training distribution," we refer to data points that lie near the edge of the training data distribution, where the model has less certainty and potentially higher variance in its predictions. These points are typically where the model might struggle to generalize, as they are far from the majority of training data, and thus, where disagreement among ensemble models is most valuable. Introducing diversity at these boundary points (or in OOD regions) leads the ensemble models to be more diverse OOD. We enhanced Section 3.2 in order to make this point clearer.

what are the advantages of the diversity term in Equation 5 compared to other diversity terms?

EDAC aims to prevent over-estimation of Q-values when training with OOD samples, for a conservative training process. This is achieved by diversifying the gradients of Q-values.

UARL enforces the diversity explicitly in the critic output space, which is the space of interest since this is where we compute the critic variance. In this paper, we focused on an empirical evaluation of our approach. Notably, Fig. 4 illustrates the benefits of UARL-enhanced AWAC compared to the base AWAC method. The diversity of UARL leads to a higher ability to discern ID from OOD samples, while maintaining a high performance (Fig. 3).

How can the authors tell that the uncertainty measurement provided in this paper is valid?

Our primary goal with UARL is to detect OOD events rather than to provide an exact estimate of uncertainty. While uncertainty estimation plays a role in guiding the detection process, the focus is on identifying regions where the policy is likely to encounter novel dynamics. We acknowledge that uncertainty calibration is an important consideration for uncertainty estimation methods, but due to the nature of UARL, we do not expect perfect calibration. Instead, we focus on using the uncertainty measurements to highlight areas of high risk and improve OOD detection. We will consider further comparisons with other uncertainty estimation methods in future work, but the current approach is primarily validated through its ability to detect OOD events effectively in our experiments.

do you have any parameter study on the threshold parameter in Algorithm 2?

Threshold at Algorithm 2 acts as a stopping criterion in fine-tuning, allowing the agent to expand the state space until the variance ($V_Q$) of the critic ensemble on the real-world dataset ($D_w$) falls below it. A lower threshold enforces a stricter certainty requirement, enhancing safety by requiring more consistent value estimates but potentially increasing training time. In contrast, a higher threshold could lead to earlier deployment, risking premature exposure to uncertain scenarios. Tuning the threshold may indeed vary per task, as each environment’s dynamics can impact the balance between safety and efficiency. While $V_Q \le \text{threshold}$ provides a proxy for confidence, it is not a formal safety guarantee. True safety would benefit from future work on approaches such as formal verification, explicit safety constraints in learning, or conservative policy updates, which go beyond the current paper’s scope. However, per the reviewer’s suggestion, we will conduct a study on the role of the threshold and its impact on performance.

We sincerely thank the reviewer for their insightful and detailed assessment of our work. We acknowledge that our initial submission did not effectively convey our ideas, and we would like to provide clarification here. Our primary goal is to develop reliable OOD detection capabilities without direct interaction with the target domain. Policy robustness is not the main focus of our work.

Offline-to-online RL or off-dynamics RL?

We thank the reviewer for highlighting this aspect of domain randomization. We acknowledge that it was not clearly explained in our initial submission. Indeed, off-dynamics is a better description of our work.

The prior work [1-5] assumes that the target domain is accessible and relies on refining the policy on the target domain. However, this presents significant safety risks when applied to complex, real-world systems. For example, refining a policy for a self-driving car involves deploying a potentially suboptimal policy to a real system, where the stakes of failure or "termination" are exceptionally high.

In contrast, our work does not rely on refining the policy by interacting with the target domain. Instead, we use an ensemble of critiques as a proxy to evaluate whether the policy is “in-distribution” of the target domain.

Furthermore, I am a bit confused about the benefits of parameter randomization against domain randomization

In conventional domain randomization, there is often no clear guidance on which parameters to randomize or the extent of randomization needed. The process tends to be somewhat arbitrary and relies on iterative validation, comparing the performance of a randomized policy to real-world scenarios. However, this trial-and-error approach also presents significant safety risks when applied to complex, real-world systems, similar to our reasoning above.

Our research seeks to improve domain randomization by introducing a clear evaluation criterion to assess whether a policy is ready for real-world deployment without directly deploying it. We approach this as an out-of-distribution detection problem. We continuously randomize the parameters until the policy is “in-distribution” of the real-world data. By utilizing an ensemble of critics, we evaluate whether the randomization effectively encompasses the true dynamics of the environment. Our concept of safety is based on the fact that we could avoid real-time interactions with the environment to refine the policy.

Our algorithm indeed builds upon the general form of the diversity term introduced in DENN. However, we note that it adapts it to the RL setting by identifying the correct function to repulse from (in our case, $r + \gamma Q(s’, a’)$). Moreover, we extend the original formulation of DENN as we do not require to train in a first phase a “reference function”, making training less cumbersome.

Lacking baseline algorithms.

Thank you for this comprehensive list of suggested baselines. We appreciate the thoroughness of your review and would like to clarify several important distinctions between our work and the suggested comparisons:

• Fundamental Objective Difference: Our primary goal is to develop reliable OOD detection capabilities, not just policy robustness. While the suggested baselines (H2O, VGDF, PAR) focus on making policies robust to distribution shifts, our work aims to explicitly identify when the system encounters novel situations that require intervention without direct interaction in the novel environment. This is a crucial safety feature for real-world deployment.
• Real-world Data Usage: Several suggested baselines (H2O, VGDF, Off2On, PROTO) require real-world data during training or policy refinement. Our method deliberately avoids this requirement for safety reasons, using real-world data only for evaluation. This design choice makes our approach more practical for safety-critical applications where real-world training data may be scarce or risky to collect.
• Complementary Rather Than Competitive: Our method is complementary to many of these baselines rather than directly competitive. While RLPD and other offline-to-online methods focus on policy improvement, our work addresses the crucial preceding question: when is it safe to deploy a policy in the first place? This detection capability could actually enhance the safety of these existing methods.

That said, we acknowledge that adding some robust RL baselines could help demonstrate the indirect benefits of our approach to policy robustness. We will expand our evaluations to include appropriate baselines that do not require real-world training data, focusing on comparing OOD detection capabilities where possible.

Lacking theoretical justifications

While we agree that such analysis could strengthen the paper, our primary focus here is on the empirical evaluation of UARL. We believe the experiments provide strong evidence of its effectiveness, and we will consider theoretical analysis in future work.

Does this algorithm fall within the scope of Offline Reinforcement Learning?

We believe this topic is already thoroughly addressed in the Related Work section (Lines 110-114), where we explicitly discuss the distinctions between our approach and existing Offline RL methods. However, we recognize the importance of ensuring that this contextualization is evident throughout the paper. To address this, we further emphasized these points in the abstract.

We sincerely thank you for your insightful feedback and detailed comments on our work. Your suggestions have significantly contributed to improving the clarity and presentation of our research.

The paper could better highlight its unique contributions compared to existing OOD and ensemble-based offline RL methods. A clearer differentiation of UARL's specific advancements would help underscore its novelty within the landscape of similar approaches.

We understand the reviewer’s concerns regarding the framing and narrative of the paper in its current format. We will work to adjust the structure and focus to clarify our main contributions and streamline the discussion around key topics. Specifically, we will refine the emphasis on our central problem domain to create a more cohesive and targeted narrative, reducing the scope of secondary discussions where possible. The changes are reflected in Sections 1 and 2 in the revision.

The experimental validation, limited to few environments such as the Ant-v4 and HalfCheetah-v4 environments, may not fully capture the method’s effectiveness across a diverse range of tasks. Extending the experiments to include more varied environments would provide a more comprehensive assessment and enhance the generalizability of the results.

While we believe our experiments provide a solid foundation for validating our method's core principles, we acknowledge the value of broader testing. That is why we are working on providing results on more environments to improve the generalizability of our findings.

A comparison with recent state-of-the-art methods, such as PBRL[1], RORL[2], would strengthen the empirical evaluation.

We want to clarify an important distinction between our work and RORL/PBRL that we should have emphasized more clearly in the paper. While RORL and PBRL represent significant advances in robust offline RL, they tackle a fundamentally different problem than our work. These methods focus on building robust policies that can maintain performance despite encountering OOD scenarios, primarily through uncertainty-based penalization during training. This is distinct from our core objective: developing reliable mechanisms to detect when a system is operating in OOD conditions.

The distinction becomes clear when considering real-world applications: a robust policy might continue operating in OOD conditions (as RORL/PBRL aim to achieve), but this could be undesirable in safety-critical systems where we need to explicitly recognize such situations and potentially halt operation or seek human guidance. Our method's progressive environmental randomization serves to build this detection capability, training the uncertainty estimator to recognize the boundaries between familiar and novel situations.

While our approach does yield some robustness benefits through its iterative fine-tuning, this is secondary to its main purpose of reliable OOD detection. We revised the paper to better articulate this fundamental difference in objectives and clarify how our method specifically targets the detection challenge rather than just robustness, reflected in Appendix C.

We deemed PBRL and RORL are not directly comparable to our work; nevertheless, we included results for a limited experimental setting comparing our method with PBRL and RORL in Appendix B.5 of the revised paper for the readers’ reference.

Comparing the computational overhead of your method with that of baseline algorithms would strengthen your work. Could you include this information to provide a clearer understanding of its efficiency?

Our method does introduce additional computational complexity, primarily through the repulsive dataset, which impacts memory usage more significantly than computation time. Most baseline methods (CQL, AWAC, TD3+BC) already use dual-critic architectures, so our additional computational overhead stems from maintaining separate nominal and repulsive datasets, the diversity loss calculation, and the replay buffer balancing mechanism.

Our measurements suggest the memory usage was increased by approximately 50% compared to baseline methods, primarily due to storing both nominal and repulsive datasets. That is also subject to change given the desired ratio of nominal data points to repulsive ones. The computational time overhead is relatively modest, in the 10-20% range, and is equivalent to training a regular actor-critic method with a larger batch size

Our results suggest that uncertainty awareness provides significant value that outweighs the additional computational requirements. We included these results in Appendix D of the revision.

Thank you for your feedback. After revisiting the reviews, I believe my score aligns with the feedback and will keep it. I appreciate your effort and encourage you to continue developing your ideas.

the convergence property of the proposed algorithm should be discussed, especially line 8 in Algorithm 2 - what if this condition is never voilated?

If the condition in line 8 of Algorithm 2 is never violated, it likely means that the domain randomization is ineffective. High uncertainty ($V_Q$) despite diverse training environments suggests that the agent has not learned a policy that generalizes to real-world conditions. This indicates that either the domain randomization strategies are inadequate or the real-world data ($D_w$) is too different from the simulated environments. In such cases, continuing training would not be productive. Instead, it would require revisiting the domain randomization process and real-world data to ensure they are sufficiently representative of the target environment. It is important to note that this is not a shortcoming of our method, but rather a reflection of the limitations of the domain randomization process or the quality of the real-world data. Our approach assumes that these factors are appropriately addressed, and failure in this regard would require improvements outside the scope of our method. We highlighted these points in the revision at the end of Section 4.4.

Thank you for your detailed review and for engaging with our work. We appreciate the time and effort you put into your thoughtful feedback. We realize there may have been some miscommunication about the problem we are addressing, and we have taken steps to clarify our goals and contributions in the revised manuscript. Your comments have helped us identify areas where clearer explanations were needed.

The core problem in this paper, i.e., distributional shift, and many important concepts are not discussed in detail.

We acknowledge that the relationships between distributional shift across different RL paradigms deserve a more thorough treatment. Our work primarily addresses detecting distributional shift during offline-to-online transition, which differs from purely offline RL in a crucial way: offline RL faces fixed distributional gaps between training and deployment data, while offline-to-online transition must handle dynamic shifts as the policy begins interacting with the environment. Our method addresses distributional shift through progressive environmental randomization that systematically expands the policy's exposure to different dynamics. Unlike robust RL which typically optimizes for worst-case performance across a fixed distribution, we actively shape the distribution of experiences to build reliable uncertainty estimates. This helps the policy identify when it's encountering novel scenarios and adapt appropriately. We will revise the introduction to clarify these conceptual relationships and better motivate how our approach bridges the gap between offline training and online adaptation.

In offline (to online) RL, the uncertainty quantifier is defined clearly as the upper bound of the error produced by the empirical Bellman operator (see [1], Eq.(4.1)).

In both papers, the goal of quantifying uncertainty is to identify states and actions where the learned policy might not be reliable due to limited data or distributional shifts. While Jin et al.’s uncertainty quantifier provides a theoretical upper bound on Bellman operator errors, our variance-based metric (Equation 5) offers a practical and adaptive approach for estimating uncertainty, particularly effective in offline-to-online RL. Both methods aim to identify unreliable state-action regions, but our ensemble variance highlights critic disagreement, serving as a scalable proxy for uncertainty under distributional shifts. Though not grounded in formal bounds like Jin et al., our method has demonstrated robust empirical performance. Future work could further investigate its theoretical properties, potentially bridging the gap between heuristic and formal uncertainty quantification.

In offline RL, there have been many uncertain-aware methods to deal with the distributional shift problem, such as [2] and [3].

While MOBILE and PBRL make valuable contributions to robust offline RL, there is a fundamental difference in objectives and methodology. MOBILE and PBRL focus on robustness against OOD scenarios by penalizing uncertain actions during offline training. Their primary goal is to learn conservative policies that avoid OOD situations. This is achieved through uncertainty-based penalties in Q-value estimation and specialized sampling techniques. In contrast, our work addresses a distinctly different challenge: explicit OOD detection during deployment, without direct interactions with the OOD scenarios. Rather than just avoiding OOD actions, we aim to actively identify when the agent encounters novel scenarios. This capability is crucial for:

1. Maintaining awareness of when the current policy might be unreliable
2. Enabling informed decisions about when to request human intervention
3. Guiding targeted data collection for policy improvement

Our progressive environmental randomization approach specifically builds up the agent's ability to distinguish between in-distribution and OOD states, rather than just being robust to them. While robustness emerges as a beneficial side effect of our method during iterative fine-tuning, it is not the primary objective. We provided a discussion in Appendix C of the revision to more clearly articulate this distinction between robustness-focused approaches (like MOBILE and PBRL) and our detection-focused methodology.

As the discussion period draws to a close, we wanted to follow up to check if you have had a chance to review our rebuttal. If you have any remaining questions or concerns, we would be glad to address them. Thank you.

We sincerely thank you for your assessment of our work. Your comments have helped us further strengthen the clarity and presentation of our contributions. Following are our responses to your comments.

Repulsive locations discussion could be written more formally and thus more concisely. The current version is a bit too dense verbally.

The section aims to establish a clear conceptual framework for how our approach progressively expands the exploration space through targeted environmental randomization. While we appreciate the suggestion for potential rephrasing, we believe the current format is necessary as we introduce the concept of repulsive locations in the RL context, detail our specific implementation through hyperparameter randomization, illustrate the progressive expansion of the exploration space through Fig. 2, and distinguishes our targeted approach from standard domain randomization. However, we welcome specific suggestions for improving the formality or conciseness of particular passages while maintaining these core explanatory elements. If the reviewer has specific areas where you feel the language could be tightened, we would be happy to address them in revision.

I'm also a bit confused by Figure 2, while it's a nice visual, is it something derived from the experiments or is it just a conceptual illustration?

It is a conceptual visualization for illustrating the idea. It did not derive from actual experiments but is meant to provide an overview of the approach. We mentioned that explicitly in the revision.

Lack of related works

UARL differs fundamentally from curriculum RL, which relies on a structured progression of tasks, helping the agent learn incrementally by tackling simpler tasks before harder ones. In curriculum RL, the policy is refined on new tasks as they are introduced. However, this sequential approach may fall short in real-world scenarios where unexpected, OOD events can disrupt performance. In contrast, UARL dynamically detects and adapts to OOD shifts based on real-time uncertainty rather than a fixed curriculum. Notably, UARL assumes that policies cannot be refined in the target environment. By quantifying and addressing uncertainty, our approach improves safety and robustness in unpredictable environments, making it better suited for real-world deployment where adaptability is key. To highlight these differences further, we will provide a discussion in the Related Work section.

How do you define "progressively expanding the randomization range" for different environment parameters?

In UARL, "progressively expanding the randomization range" does not involve a simple, uniform increase in each parameter. Instead, we dynamically adjust the randomization based on environmental uncertainty, without requiring expert knowledge for each parameter. Specifically, the expanded dataset serves as a repulsive dataset, introducing uncertainty to help detect OOD cases. During the verification process on the real-world environment ($\mathcal{D}_w$), two outcomes guide us: if low uncertainty is detected, it suggests that the dataset adequately captures the real-world dynamics, meaning the parameter range is reasonable. If high variance remains, it indicates that the dataset is still "too narrow," signaling that the randomization range should be further expanded to better encompass real-world variability. This iterative process ensures that we avoid training in environments that are too far from the actual dynamics, stopping before the agent encounters destabilizing conditions.