Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning

We sincerely thank the reviewer for the time and feedback. Please refer to the link https://anonymous.4open.science/r/ICML-2663/README.md for our newly conducted experiments.

Negative transfer Our initial environments were relatively simple, so negative transfer was not evident. To support our claims, we ran a new Cartpole experiment (Fig 1). The source domains are with randomly generated parameters (length of pole), and the target is the default one. As shown, MDTL-Avg suffers from negative transfer, while MDTL-Max successfully mitigates it. We will elaborate further in the final version.

Experiments on E and robust baselines We validate the trade-off in Fig. 2 and Table 1. As E increases, convergence slows (under fixed step size) and performance declines. However, computational time also decreases, indicating reduced communication.

We also add two robust baselines (see Table 2 and reply to R rfmi).

Identical reward assumption We made this assumption to simplify presentation and analysis, but our framework can extend to settings with differing rewards.

1. If the (unknown) target domain reward differs from the sources, additional knowledge on its relation to the source reward should be known, e.g., $|r_t(s,a)-r_s(s,a)|\leq R(s,a)$, otherwise the target may have different goals with source domains, making transfer impossible. With such knowledge, we additionally define a reward uncertainty set for source: $\mathcal{R}^k_{s,a}=\{r': |r'-r(s,a)|\leq R(s,a) \}$ and modify the robust Bellman operator to $\mathbf{T}^k(Q)=(r-R)+\sigma(V)$, preserving pessimism and effectiveness.

2. If source rewards also differ from each, we can still apply different local operator $\mathbf{T}^k(Q)=(r^k-R^k)+\sigma(V)$ to similarly ensure conservative.

We will discuss more in the final version.

Extended to the case with different similarities parameter in source domains We thank the reviewer for this insightful point. While most proofs are independent of the radius, some technical adjustments are needed. For example, in extending Thm. 6.4, we can adjust (89) to yield a bound $\gamma\lambda\frac{E-1}{1-\gamma}\max_i\{\Gamma_i\}$. We will discuss this further in the final version.

Claim on the scalability Thank you for this valuable comment. We will revise the claim. By scalability, we refer to the local updates, which can leverage scalable, model-free methods. While global aggregation is necessary in tabular Q-learning, our approach is not limited to this setting. Our method can be paired with function approximation or policy gradient approaches to improve scalability. For instance:

1. With function approximation for robust RL [1], or
2. Robust policy gradient [2], the global step only requires parameter aggregation (e.g., Jin et al., 2022), making it scalable in practice. We further numerically verify this by develop experiments under Cartpole and DVRP (please see our response to #tyX8). We will revise the statement and leave further extensions for future work.

[1] Zhou, R. et al., Natural actor-critic for robust reinforcement learning with function approximation, 2023 [2] Wang, Y. & Zou, S., Policy gradient method for robust reinforcement learning, 2022

Proxies in Line 220 Our proxies outperform options (1)-(3) due to reduced conservativeness. Though proxy (4) is less conservative, it presents practical challenges:

1. Its uncertainty set lacks a ball structure, making standard robust RL methods inapplicable.
2. Intersection-based methods require accurate knowledge of each local model and set, which is not available via raw data communication alone. Thus, our method has two advantages over (4): lower communication requirements and easier implementation.

E=1 in Theorem 5.6/6.4 When E = 1, the global Q-table is updated every step using $Q \leftarrow \frac{1}{K} \sum_k \mathbf{T}^k(V)$. This operator is a $\gamma$-contraction, so convergence rate is independent of $\Gamma$. Even in the presence of unbiased noise, the expected update preserves this rate. Technically, Eq. (34) bounds $\|\sigma(V) - \sigma(V')\| \leq \|V - V'\| \leq \frac{1}{1 - \gamma}$, independent of $\Gamma$. However, $\Gamma$ still affects the pessimism level $\zeta$, which impacts the effectiveness, though not the convergence rate.

When E > 1, agents perform multiple local updates with their own robust operators. The global update then reflects local differences, making it depend on $\Gamma$. A similar effect is observed in Wang et al., 2024 (Theorem 1), where heterogeneity does not affect convergence when E = 1.

Learn locally and average We apologize for the possible misunderstandings here. $Q_{AO}$ is the intended solution, and $Q_k$ is the algorithm’s output. The reviewer may have confused $Q_k$ with $Q^*\_k$, the robust value function for local domain k. Importantly, $Q_{AO}$ is different from $\frac{\sum Q^*\_k}{K}$, so simply learning locally and averaging does not yield the correct solution.

We sincerely thank you for your time and feedback, and appreciate the reviewer identifying our contribution. Please refer to the link https://anonymous.4open.science/r/ICML-2663/README.md for our newly conducted experiments.

Hyperparameter sensitivity; Benchmark like contextual MDPs. We appreciate the reviewer’s suggestion.

Two hyperparameters are included: E and $\lambda$.

1. We implement algorithms with different E in Fig 2, Table 1. A larger E results in slower convergence but less computation, matching our theoretical results.
2. We run experiments with $\lambda=0.2$ in Table 2. Together with results in our paper for 0.1, it shows our methods are uniformly better and robust to it.

We present 2 more experiments. (1). Cartpole Experiment (Fig. 3): We apply our methods to the Cartpole environment, a common example of a contextual MDP. Results show that even when updates introduce bias, our methods outperform the baseline, confirming robustness. (2). Dynamic Vehicle Routing Problem (DVRP): We design a real-world scenario—DVRP [Iklassov, Z, et al., Reinforcement Learning for Solving Stochastic Vehicle Routing Problem, 2024]—which extends the classic vehicle routing problem with real-time requests and environmental uncertainty. Three important objectives are considered: Minimizing overall routing cost, Enhancing route smoothness, and Improving route stability. As shown in Table 3, our methods consistently outperform baselines across all criteria, and are more effective.

We will add more experiments in the final version.

Additional references. We thank the reviewer for recognizing our contributions and providing constructive suggestions.

We briefly discussed contextual RL (cRL) for zero-shot generalization in Appendix A. cRL seeks to optimize performance under a contextual distribution to enable transfer without fine-tuning. However, theoretical guarantees remain limited. For instance, [1] provides a suboptimality bound, but it requires samples from the contextual distribution, potentially including target domain data. Furthermore, it remains unclear whether cRL can mitigate negative transfer, especially when the context distribution is assumed to be uniform. We also appreciate the reviewer’s mention of safe and risk-sensitive RL, which aim to optimize reward while maintaining low cost under specific criteria [2]. However, most of these works are developed for single-environment settings, and do not address transfer scenarios. We will incorporate a more thorough discussion in the final version.

[1] Wang, Z., et al., Towards Zero-Shot Generalization in Offline Reinforcement Learning, 2025 [2] García, J., et al., A Comprehensive Survey on Safe Reinforcement Learning, 2015

Intuition or summarizing remarks. Thank you for the suggestion. Our algorithm consists of two main parts: local updates and global aggregation. Each agent independently updates its local Q-table using its data, and after every E steps, a global aggregation step unifies the local Q-tables. Our convergence proof mirrors this structure: it is decomposed into a local part (governed by the local Bellman operator) and a global part (governed by the aggregation mechanism). We will include further discussion in the final version.

Unbiased Estimation Assumption. The unbiased estimation assumption is introduced to simplify convergence analysis. However, our convergence results can be extended to settings with controlled bias, such as the bias introduced by max aggregation. This bias can be mitigated using techniques like threshold-MLMC [1]. Importantly, even in the presence of bias, if the expected proxy remains pessimistic, our transfer learning framework and theoretical guarantees still hold. We thus argue that our method is robust to such biases. To support this, we include an experiment on Cartpole (Fig. 3) analyzing the impact of bias. Results show that our proxy remains conservative, and the pessimism-based framework continues to outperform the baseline, confirming robustness.

[1] Wang, Y., et al., Model-Free Robust Reinforcement Learning with Sample Complexity Analysis, 2024

Comparisons with baselines. As noted in the paper, our primary advantage lies in theoretical guarantees: we optimize a lower bound on the target domain’s performance, avoiding overly optimistic decisions that may fail under domain shift. In contrast, domain randomization (DR) lacks such guarantees and can be overly optimistic, especially when the randomized training environments fail to capture true uncertainty.

We acknowledge that when the uncertainty set is constructed using inaccurate prior knowledge, our method may become overly conservative, potentially leading to suboptimal performance compared to DR. Nonetheless, the ability to guarantee robust performance across all scenarios is a central contribution—especially important for safety-critical or high-stakes applications, where conservativeness is not only expected but necessary.

We sincerely thank you for your time and feedback, and appreciate the reviewer identifying our contribution. Please refer to the link https://anonymous.4open.science/r/ICML-2663/README.md for our newly conducted experiments.

Upper bound on $\max_\pi\zeta^\pi$.

We first clarify that $\|\zeta\|$ has an upper bound and can be controlled via the construction of the uncertainty sets. With our proxies, it holds that $\|\zeta\|\leq \mathcal{O}((1-\gamma)^{-2}\Gamma)$. Moreover, when adapting our Max proxy with different radii $\Gamma_i$, it holds that $\|\zeta\|\leq \mathcal{O}((1-\gamma)^{-2}\min\{\Gamma_i\})$. Hence the suboptimality gap can be upper bounded by the domain similarities, which is reasonable and subject to the nature of the problem. This dependence also motivates one of the potential extensions of our framework -- a safe-to-improve principle: once a conservative proxy is constructed, any improvement to it directly improves target performance, without the loss of theoretical performance guarantees. For instance, if additional information is available (e.g., a small set of target domain data or allowance of explorations), we can further reduce radii $\Gamma$ to ensure a better transfer performance.

Formally define the problem setting. It would be better to specify the setting and the assumptions needed explicitly in this section.

Our formulation is as follows. Consider the target domain $\mathcal{M}\_0 = (\mathcal{S},\mathcal{A},P\_0,r,\gamma)$ with the unknown target kernel $P_0$. We have no data from it, but instead have access to multiple source domains $\mathcal{M}\_k = (\mathcal{S},\mathcal{A},P\_k,r,\gamma)$. Our goal in zero-shot multi-domain transfer learning is to optimize the performance $V_{P_0}^\pi$ under $\mathcal{M}_0$. For simplicity, we consider the case of identical reward (please see our response to Reviewer QZdx for extensions).

The only assumption is that, there exists an upper bound $\Gamma\geq D(P_0||P_k)$, that is known by the learner. This assumption is reasonable, as such information is essential to obtain performance guarantees. Even if in the worst-case, we can set $\Gamma=1$ (in total variation) to construct an (overly) conservative proxy, which yet still avoids decisions with severe consequences in the target domain, preferred in transfer learning settings.

Theorem 7.2. seems to be too abrupt to appear, the $\Delta_T$ is used without definition.

We apologize for missing the definition of $\Delta_T$. It is the difference between the algorithm output and the fixed points of the aggregated operators: $\Delta_T := Q_{\rm AO} - Q^T$ for MDTL-Avg, and $\Delta_T := Q_{\rm MP} - Q^T$ for MDTL-Max. We will clarify these.

The proposed methods borrow some ideas from federated learning, it would be better to briefly introduce some related work in federated learning in the Related Work section (in appendix A. Additional Related Works).

We thank the reviewer for the helpful suggestion. We will include related work on federated reinforcement learning (FRL). We note, however, that directly extending FRL to our setting poses non-trivial challenges: (1) FRL has linear update rules (non-robust operators), while ours involve non-linear robust operators; (2) FRL focuses on optimizing average performance across local environments, whereas we address the more challenging max-based multi-domain transfer; and (3) more importantly, FRL is not designed for transfer learning, while our pessimism principle enables transfer with theoretical guarantees.

In Proposition 5.4., you prove the results under total variation or Wasserstein distance, do similar results hold under other metrics?

According to our proof, the result holds as long as $\sigma\_{\bar{\mathcal{P}}}(V) - \frac{1}{K}\sum_{k=1}^K\sigma\_{\mathcal{P}\_k}(V) \leq 0$. We note that additional for $l_p$-norm, the support function has a duality (Clavier et al., 2024. Near-Optimal Distributionally Robust Reinforcement Learning with General $L_p$ Norms.): $\sigma(V)=\max_{\alpha} ( PV\_{\alpha}+f(\Gamma,V\_{\alpha}))$ for some function $f$. Thus $\sigma\_{\bar{\mathcal{P}}}(V) - \frac{1}{K}\sum_{k=1}^K\sigma\_{\mathcal{P}\_k}(V)=\max\_{\alpha} ( \bar{P}V\_{\alpha}+f(\Gamma,V\_{\alpha}))-\frac{1}{K}\sum_i \max_{\alpha} ( P_iV\_{\alpha}+f(\Gamma,V\_{\alpha}) )\leq 0$. It is our future interest to extend to other uncertainty sets.

We sincerely thank you for your time and feedback, and appreciate the reviewer identifying our contribution and novelty. Please refer to the link https://anonymous.4open.science/r/ICML-2663/README.md for our newly conducted experiments.

Other relevant robust RL approaches We thank the reviewer for pointing these methods out. We first want to highlight that our method is indeed based on distributionally robust RL, where our uncertainty set is constructed to account for the potential distributional shift. The adversarial robust RL, although commonly studied in experiments, generally lacks theoretical guarantees on, e.g., convergence, and tends to be overly conservative. Our methods, however, enjoy both convergence and performance guarantees. We will include more discussion in our final version.

We then numerically verify that our method is effective, and present the results for the recycling robot problem in Table 2 in the above link. We use adversarial action-robust RL [1] and distributionally robust RL [2] as new baselines. As the results show, although adversarially robust and distributionally robust RL outperform non-robust methods, their performance remain significantly inferior to ours.

[1] Tessler et al., 2019. "Action Robust Reinforcement Learning and Applications in Continuous Control."
[2] Panaganti & Kalathil, 2022. "Sample Complexity of Robust RL with a Generative Model."

Cost of MLMC MLMC is primarily designed for proof under the tabular settings, and is essential for obtaining an unbiased update. As we discussed, the computational cost of MLMC can be viewed as a trade-off of transfer performance, thus such a high cost may be acceptable in safety-critical applications such as robotics and autonomous driving.

Its cost can be reduced along several potential directions: (1) The level number $N$ of MLMC can be controlled through techniques like threshold-MLMC [1], which applies an upper bound on $N$. Although it results in a biased estimation, the bias can be controlled and hence still implies convergence (numerically verified in Fig 3 in the link). (2) Techniques to reduce the computational cost of the robust Bellman operator can also be applied. One potential approach is to relax the uncertainty set constraint, which would result in a conservative and efficient solution [2]. (3) Reducing the aggregation frequency by increasing $E$ is also another way to reduce the computational cost, but introduces a trade-off in the convergence rate, as we have shown in our theoretical results (further numerically shown in Fig 2 of the link).

[1] Wang, Y. et al. Model-Free Robust Reinforcement Learning with Sample Complexity Analysis, 2024
[2] Kumar, N.et al. Policy gradient for rectangular robust Markov decision processes, 2023

No robustness tests We developed some robustness testing experiments in Appendix, where our methods are shown to be more robust to target domain uncertainty. We also conduct additional experiments to verify robustness. Firstly, as the results in Table 2 in the link show, our methods are more robust to uncertainty in the target domain. Under different levels of uncertainty, our methods outperform baselines and is hence robust. Secondly, in Fig 3, we showed that even with noise or bias in the update, our algorithms still outperform the baseline and are also robust.

We will include more robustness tests in the final version.

Assuming prior knowledge of domain similarity Generally, such knowledge can be obtained through domain experts, or estimated from a small amount of target data.

Moreover, our method remains applicable even without prior knowledge, by setting $\Gamma$ to a known upper bound on distributional distance (e.g., total variation between any two distributions is at most 1). While this yields an over-conservative proxy, it still helps prevent significant drops in transfer performance due to our built-in pessimism principle. We believe this is the most reliable approach in settings where no similarity information is available.

We also note that this limitation applies to other methods as well. For instance, domain randomization (DR) assumes prior access to a set of environments that includes the target domain (Chen et al., 2021). Without such prior knowledge, our pessimism principle helps prevent undesirable outcomes, whereas DR provides no such guarantees.

Can ($\zeta$) be adaptively tuned?
We first note that $\zeta$ depends on $\Gamma$ in our algorithm. Since we consider a zero-shot setting, $\Gamma$ is pre-set and fixed, and it cannot be tuned without any additional information. However, if any additional knowledge is available, e.g., a small amount of target domain data, or we are allowed to fine-tune via exploration, we can use it to shrink the uncertainty radius and enhance the effectiveness.

Negative transfer We sincerely refer the reviewer to our response to Reviewer QZdX. The results are in Fig 1.