BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping

We thank reviewer SDzM for their thoughtful review and helpful feedback. We are glad that they found our approach to studying pseudo-rewards to be very original and interesting, and our theoretical results insightful.

most of the performance guarantee results are proven at the scale of meta-learning, and it is not clear to me whether this actually solves the “reward-hacking” problem at the scale of individual MDPs

Thank you for flagging that this aspect of our contribution is unclear. We have updated our language in section 4.3 in the revised manuscript to make this clearer- our result there is proven for the regular RL setting. We also added an experiment in this section in the MountainCar environment to empirically demonstrate how reward-hacking at the MDP level is avoided with a BAMPF. We show a BAMPF pseudo-reward helping PPO learn an optimal goal-reaching policy more quickly, whereas a similar non-potential based IM term initially helped exploration but eventually led to a reward hacking policy that avoids the goal to collect more pseudo-rewards. See the revised manuscript and the overall response for more details of the new experiments.

it remains unclear what are the advantages/disadvantages of choosing a specific BAMPF, which limits the practical relevance of the paper.

We appreciate this feedback for making our paper more practical. We have included discussion about how we chose effective BAMPFs for the new experiments (see the overall response or sections 4.2.2 and 4.3.1 in the revised manuscript) which we hope will be a helpful example for practitioners seeking guidance on how to design BAMPFs for their own settings.

Questions:

how IM is related to VOI or VOO quantitatively

We discuss further in Appendix C how RL algorithms can underestimate the value of information in their base objective, which can explain how many IM terms help exploration by adding approximations of VOI back into the objective. But we agree that this section is mostly qualitative; it is generally not possible to calculate the true VOI and VOO, but it can still be a useful framework for understanding and designing IM terms. The Mountain Car (section 4.3.1) and Bernoulli Bandits (section 4.2.2) experiments we added provide more concrete examples of how the framework can guide the design of and understand the effects of IM.

what are the definition of attractive/repulsive signals?

We briefly explained attractive and repulsive signals after the definition of VOO, but we see how that could be easy to miss so we now reiterate it in the caption of Table 1. An attractive signal provides (usually positive) rewards to correct value underestimation, and a repulsive signal provides (usually negative) rewards to correct value overestimation

Questions 2 and 3:

Thank you for flagging the lack of clarity and precision in the wording here. By environments with finite information, we meant to intuitively describe environments in which potential functions measuring the amount of information gathered cannot increase indefinitely. We did intend “settling” to describe a phenomenon that depends on both the RL agent’s trajectory and the potential function, but have now reworked the theory in this subsection to be simpler and more direct. Realizing that every example of Settling we thought of involved a bounded monotone potential function, we cut out the intermediate definition of Settling and the policy-dependence. Instead we directly prove that BAMPFS based on bounded monotone potentials preserve approximate optimality for any RL agent (see section 4.3 of revised manuscript)

In figure 3, what is None? Does it make use of any exploration method?

None corresponds to the DQN agent without any pseudo-rewards. The DQN agent we ran in this demonstration uses epsilon-greedy exploration, with epsilon=1 for the first 20 steps, then decaying to 0.05 over the next 100 steps. We have clarified this in the caption and text and added these details to the appendix.

We thank reviewer SDzM for considering our response.

(We realized that latex sometimes doesn't display properly unless broken into separate lines, hence the multiple line breaks in our reply here:)

Theorem 4.3 does not really address my concern. My question is that whether we can guarantee the performance of the optimal BAMDP policy for an individual MDP

Apologies for misunderstanding your concern. It is indeed possible that for some $M$ in the support of $p(M)$, $\mathbb{E}_{\bar{\pi}^{*}}[G|\mathcal{M}]$ is significantly lower than the expected performance over $p(M)$.

To say more about $\mathbb{E}_{\bar{\pi}^{*}}[G|\mathcal{M}]$ would involve doing some type of worst-case robustness analysis.

We do not address this problem in our paper, but this is not what we mean by reward hacking. We address reward hacking as defined in [1] as rewards that “mislead the agent into learning suboptimal policies”. This is where an RL algorithm is incentivized to generate a policy $\pi^{*'}$

that maximizes expected shaped return $\mathbb{E}_{\pi^{*’}}[G|\mathcal{M}’]$,

but achieves suboptimal expected real return $\mathbb{E}_{\pi^{*’}}[G|\mathcal{M}]$.

Our theorems state what forms of pseudo-reward can be added to a BAMDP (4.2)/MDP(4.3) without changing which RL algorithms/MDP policies maximize expected BAMDP/MDP return, respectively, and thus can't cause reward-hacking by definition.

Regarding attractive/repulsive signals, I was asking for mathematical definitions of them

Thank you for flagging that these concepts are still unclear. Here is a more precise mathematical definition: a pseudo-reward is an attractive signal towards a metric M if the reward increases in the metric, i.e., $f(h1) \geq f(h2) \iff M(h1) \geq M(h2)$ (where e.g., f is state coverage and M is the number of states visited) and it is a repulsive signal if it decreases in the metric, i.e., $f(h1) \leq f(h2) \iff M(h1) \geq M(h2)$ (e.g., f is the joint angle violation penalty and M is the angle of a robot’s joint). This isn't a full classification of how pseudo rewards could incorporate metrics, just an intuitive typology for many reward shaping or IM terms that are designed to encourage or discourage specific behaviors or outcomes. We have updated the manuscript to make this clearer.

to define a monotone function $\phi(h)$, you'll have to first define an order over $h$.

We thank the reviewer for pointing this out. The sequence of histories is ordered by time, i.e., for all training steps $t_1<t_2$, either $\phi(h_{t_1}) \leq \phi(h_{t_2})$ for a monotone increasing $\phi(h)$, or $\phi(h_{t_1}) \geq \phi(h_{t_2})$ for a monotone decreasing $\phi(h)$. We have updated the manuscript to specify this explicitly.

Thanks again for taking the time to engage with our work, and please let us know if you have any more questions

[1] Ng, Andrew Y., Daishi Harada, and Stuart Russell. "Policy invariance under reward transformations: Theory and application to reward shaping." Icml. Vol. 99. 1999.

We thank the reviewer for taking the time to consider our responses again, and appreciate their patience in explaining their concern-- it is a challenge to write clearly about concepts at both the RL and meta-RL level within the same work, and this discussion is valuable for highlighting ways that we could improve the clarity of our writing.

It's possible that some confusion may be arising from the fact that we're discussing reward hacking in meta-RL, but the Noisy TV problem[1,2] (and most well-known cases of reward hacking) is an instance of reward hacking in regular RL. This is where the pseudo-rewards cause RL algorithms (which may or may not be Bayes-optimal BAMDP policies themselves) to converge on MDP policies that maximize shaped return $\mathbb{E}_{\pi}[G|\mathcal{M^\prime}]$,

but perform suboptimally with respect to the true return $\mathbb{E}_{\pi}[G|\mathcal{M}]$ in an individual MDP. E.g., in the Noisy TV problem, the eventual policy $\pi$ learnt by the RL algorithm $\bar{\pi}$ maximizes the shaped return by watching TV, while this behavior doesn't maximize the true return. This empirically documented problem, which matches the definition of reward hacking from [4], is the major motivation of the paper, which we address with theorem 4.3.

Now we return to the discussion about reward hacking in meta-RL (which is the purview of theorem 4.2).

is it not the case that there might exist a BAMPF that makes the performance arbitrary bad for some environments?

It cannot make performance arbitrarily bad - if we guide a meta-RL agent with BAMPF pseudo-rewards and the resulting Bayes-Optimal RL algorithm $\bar{\pi}^*$ performs worse in one MDP, it must perform correspondingly better in other MDPs such that the lost performance balances out, preserving the optimal expected performance $\mathbb{E}_{\bar{\pi}^*,\mathcal{M}\sim p(\mathcal{M})}[G]$. Thus, the learnt RL algorithm still behaves rationally with respect to the true distribution of MDPs $p(\mathcal{M})$, and this is not an instance of reward hacking where the pseudo-rewards mislead the meta RL agent to optimize for a different reward distribution.

The problem here is that the Bayes-optimal RL algorithm might not find the optimal policies for some environments.

It is true that a Bayes-optimal BAMDP policy might not find an optimal MDP policy for some MDPs. But this is not a problem caused by BAMPFs or any sort of reward hacking - this is a consequence of how the optimality of an RL algorithm (and the meta-RL objective) is defined in terms of its expected performance over $p(\mathcal{M})$, rather than worst-case performance. Reimagining the mainstream RL objective to optimize for best or worst-case performance has been explored, e.g., [3], and it is an interesting question but orthogonal to our work.

We hope that these explanations help resolve the reviewer's uncertainty, but please let us know if there are remaining concerns.

[1] Burda, Yuri, et al. "Exploration by random network distillation." arXiv preprint arXiv:1810.12894 (2018).

[2] Mavor-Parker, Augustine, et al. "How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation." International Conference on Machine Learning. PMLR, 2022.

[3] Veviurko, Grigorii, Wendelin Böhmer, and Mathijs de Weerdt. "To the Max: Reinventing Reward in Reinforcement Learning." arXiv preprint arXiv:2402.01361 (2024).

[4] Ng, Andrew Y., Daishi Harada, and Stuart Russell. "Policy invariance under reward transformations: Theory and application to reward shaping." Icml. Vol. 99. 1999.

Looking over the reviewer's questions and concerns, it seems that it might be helpful to clarify the distinction between the meta-RL versus RL settings of our results. So we restate our theorems below, trying to make the differences as explicit as possible:

Theorem 4.2: (meta-RL): we can preserve $E_{p(M)}[ \mathbb{E}_{\bar{\pi}^{\ast}}[G|M] ]$: the optimal expected (over both the MDP distribution $p(M)$, and any stochasticity within each MDP's $T$ and $R$ functions) return of the RL algorithm $\bar{\pi}$ (i.e., performance throughout learning) learnt by any meta-RL agent, if we guide the meta-RL agent with BAMPF rewards.

Limitations: doesn't guarantee worst-case performance of learnt RL algorithm in individual MDPs

Theorem 4.3: (regular RL) we can preserve $\mathbb{E}_{\pi^*}[G]$: the optimal expected (just over any stochasticity in $T,R$) return in all MDPS of the MDP policy $\pi$ (i.e., test-time performance of a fixed policy) learnt by any RL algorithm, if we guide the RL algorithm with bounded monotone BAMPF rewards.

Limitations: doesn't guarantee the rewards obtained throughout learning (but we provide principles and examples for designing BAMPFs to improve learning performance in sections 3 and 4.3)

We will further clarify the scope and limitations of each theorem in the paper, and thank the reviewer for bringing this area of improvement to our attention. But to reiterate, Theorem 4.3 does fully address the problem of reward hacking, both as defined by Ng. et al, and as it is widely observed empirically (e.g., the Noisy TV problem). We believe that this in itself is a contribution worthy of publishing at ICLR. (Theorem 4.2 addresses reward-hacking for meta-RL, which is not yet well-known since reward-shaping for meta-RL is still a nascent technique, but was recently discussed in e.g., [1].)

We hope that this has helped clear up any misunderstandings of our work, but please do let us know if there are remaining concerns, and again we appreciate the reviewer's continued patience and thoughtful engagement.

[1] Zhang, Jin, et al. "Metacure: Meta reinforcement learning with empowerment-driven exploration." International Conference on Machine Learning. PMLR, 2021

We thank the reviewer for taking the time to elaborate on their concerns again.

Addressing questions and concerns about Theorem 4.2 (meta-RL setting):

BAMPF ... will preserve the [expected] performance of RL algorithms... but I don't see how this can allow us to "design new pseudo-reward terms in this form to avoid unintended side effects"

When we talk about unintended side effects we mean decreasing expected performance, which is the standard metric used in meta-RL. (Note that if we're in a setting where optimal RL algorithms always find the optimal MDP policy, then after adding a BAMPF the resulting RL algorithm will still always find the optimal MDP policy)

If I'm understanding correctly, the present work only states that we can shift the unintended side effects from one environment to another while preserving the expected performance when averaged over environments.

Performing worse in some MDPs and better in others is not a side-effect of BAMPFs: even with no reward shaping, meta-RL generally learns RL algorithms (that may still be Bayes optimal) with this behavior. And while Theorem 4.2 doesn't guarantee performance in all individual MDPs, our work does show we can preserve optimal performance of MDP policies in all MDPs with Theorem 4.3.

Addressing questions and concerns about Theorem 4.3 (RL setting):

In Equation 23, $G$ should be replaced by $G_t$

We believe the reviewer has misunderstood our notation here. Apologies for not providing sufficient explanation, here is a full explanation which we will also add to the paper:

$\mathbb{E}_{\pi}[G_t^{\prime}]$ refers to the expected discounted sum of both BAMPF rewards and real rewards that MDP policy $\pi$ achieves when rolled out at training step $t$, which varies with $t$ because the BAMPF varies over time.

$\mathbb{E}_{\pi}[G]$ is the expected discounted sum of just the real rewards that $\pi$ achieves-- because the real reward function doesn't change over time, this return is constant with respect to any $\pi$.

the performance guarantee only holds after $t$ steps, not for earlier steps (and therefore not for $G$ as a whole)

Our performance guarantee in Theorem 4.3 is for the performance of the MDP policy $\pi_t^{* \prime}$ learnt by the RL algorithm, $\mathbb{E}_{\pi^{*\prime}_t}[G]$ (i.e., the test-time performance of the learnt policy),

which is different from all the rewards obtained by RL algorithm during learning, i.e., $\mathbb{E}_{\bar{\pi}}[G]$. So, Theorem 4.3 doesn't guarantee that RL algorithms will always achieve high returns throughout training, it guarantees the optimality of the learnt MDP policy at test-time. This is sufficient to address the problem of reward hacking both as defined by Ng et al. and observed empirically- when the RL algorithm converged on an optimal MDP policy for the shaped return (e.g., the TV watching policy) which is not optimal for the real return.

Note that, though Theorem 4.3 doesn't address performance throughout training, Section 3 (value decomposition) provides principles and Section 4.3 provides empirical examples of how to design bounded monotone BAMPFs that do improve learning.

The $H$ in lemma A.2 may be policy-dependent, and whether this can be upper bounded might depend on the class of policies considered

The sequence $\phi(h_t)$ depends on the RL algorithm, not any one MDP policy $\pi$, since the RL algorithm determines the history of experiences throughout learning $h_t$ (often through outputting a sequence of policies $\pi_1,\pi_2,...$). Thus, $H$ may certainly be algorithm dependent ($H(\bar{\pi})$). But as long as $H$ exists for any given RL algorithm (which we proved), we guarantee that it preserves optimality. Thus, we can guide any RL algorithm with a bounded monotone BAMPF while guaranteeing that optimality is preserved.

We thank the reviewer for another insightful response. We are glad to hear we have addressed their main concerns, while we greatly value their continued engagement in helping us improve our manuscript.

Currently, the authors motivate the work through examples of how bonus rewards can lead to unintended side effects in the regular RL setting, which may lead the readers into thinking that the proposed solution can address these problems in the regular RL setting rather than the meta RL setting.

We do propose a solution to reward hacking in the regular RL setting (e.g., the Noisy TV problem)- the bounded monotone BAMPF in section 4.3 addresses this setting. But we appreciate the reviewer letting us know that the narrative could be improved. We see how it may be confusing since we start with our meta-RL contribution in section 4.2 before the regular RL setting in 4.3, but we did not thoroughly motivate or introduce the problem of reward shaping for meta-RL in the introduction or abstract. We will rework the introduction and abstract to make the narrative clearer and more representative, and we have already edited the discussion in the revised manuscript to make the contribution and limitations more explicit.

Regarding Lemma A.2:...you need to show that $H(\bar{\pi})$ ($\forall\bar{\pi}$ optimal at $t$) can be bounded by $H$.

Thank you for further explaining your concern. To clarify, do you mean "$\forall \pi$ optimal at $t$"? In Theorem 4.3 we only discuss optimal MDP policies $\pi$ (where $\pi_t^{*\prime}$ is an optimal MDP policy for the shaped rewards at time $t)$, not optimal RL algorithms $\bar{\pi}$ (since this theorem is for the regular RL setting). The convergence speed is fully specified by the RL algorithm $\bar{\pi}$, and we don't have a different horizon for each optimal MDP policy at time $t$. This is because every RL algorithm $\bar{\pi}$ generates its own sequence of training histories $h_1,h_2,...h_t$ and thus its own bounded monotone sequence $S^{\bar{\pi}}_t=\phi(h_1),\phi(h_2),...\phi(h_t)$. Though the transitions and rewards in $h$ are typically collected by rolling out MDP policies $\pi_1,\pi_2,...$, note that each $\pi$ is generated by the RL algorithm based on the history ($\pi_t=\bar{\pi}(h_t)$) and only used for a finite time e.g., a batch of 100 episodes, before being updated.

But we think we understand the reviewer's concern now. To prove Theorem 4.3, we use Lemma A.2 to bound the difference in $\phi(h_{t+N})$ between two MDP policies $\pi_1,\pi_2$ if they were rolled out from the same point in training $h_t$:

$E_{\pi_1}[\phi(h_{t+N})] - E_{\pi_2}[\phi(h_{t+N})] < \epsilon$

But if $\pi_1,\pi_2$ are arbitrary policies not necessarily generated by RL algorithm $\bar{\pi}$ during training, then potential $\phi$ at the new histories $h_{t+N}$ they induce may not be part of the same sequence $S_t^{\bar{\pi}}$. So even if $S_t^{\bar{\pi}}$ has converged, $E_{\pi}[\phi(h_{t+N})]$ might still change by more than $\epsilon$ for these hypothetical MDP policies. Thus, Lemma A.2 can only be applied for MDP policies $\pi$ that were actually generated by the RL algorithm during training. Did we correctly understand the reviewer's concern?

In light of this, Theorem 4.3 can instead guarantee the ordering of all policies tried by the RL agent is preserved. The main conclusions are unchanged: preserving the ordering still prevents reward-hacking since the RL algorithm cannot generate a policy $\pi_t$ that gets higher shaped return but lower real return than any other policy it tries. (This is much closer to our original result in this section, before we restructured our theory to bypass the definition of Settling following reviewers' suggestions).

We are very grateful for the reviewer's care and attention to detail with our theory. We will revise our manuscript accordingly, taking extra care to check all the details. Please let us know if this addressed the concern!

We thank the reviewer for the clarification on this. We agree, and will update our theorem statement to make it more precise as described in our previous response.

Thanks so much again for your thoughtful responses and high level of engagement throughout this discussion period, which has been extremely helpful for improving our paper!

We thank reviewer AbDY for their thoughtful and constructive comments. We are glad that they found our paper to be well-written and our theoretical results to be insightful and interesting.

Experiments:

We agree that the paper would be strengthened with more experiments and less simplistic settings. We have added two more experiments to section 4; in both settings, as suggested, we contrast a BAMPF with other forms of pseudo-reward, showing how it speeds up learning while avoiding reward-hacking.

• In the Mountain Car environment, which is episodic and involves a continuous two-dimensional state space. We demonstrated a BAMPF pseudo-reward signalling VOI helps PPO learn a goal-reaching policy more quickly while preserving optimality. In contrast, regular potential-based shaping did not provide any improvement, and adding similar non-potential based rewards initially helped exploration, but eventually led to a reward hacking policy that avoids the goal to collect more pseudo-rewards

• In the Bernoulli bandit meta-learning problem introduced by [1] we show that a BAMPF helps the A2C meta-learner find the optimal RNN-based RL agent more quickly by discouraging over-exploration, while the non-BAMPF version of the same pseudo-reward led to a reward-hacking RL agent that over-exploits.

We also explain how we design the potentials and understand the effects of our pseudo-rewards using the principles from our framework in section 3 (value signaling and decomposition). See the revised manuscript and the overall response for more details of these new experiments.

Case Study: This is a great point. We have added a case study describing how one could use our framework to convert Curiosity[2], a well-known IM term that suffers from the Noisy TV problem, into a form that preserves optimality and thus alleviates the Noisy TV problem (section 4.3.2 in the revised manuscript)

Limitations: Thank you for this suggestion. The BAMPF potential function includes all pseudo-rewards that are functions of the algorithm’s learning history, which captures virtually any form of reward shaping or intrinsic motivation. This guarantees the optimal RL algorithm will be preserved if we use this to shape meta-RL. However, to provide reward hacking guarantees for regular RL we also require the potential to be bounded and monotone, which is more limited. This includes rewards based on the entire history like state coverage or information gain, but excludes rewards purely based on the latest actions like prediction error (the predictability of the latest observations) or entropy bonus (the randomness of the latest policy). We show in the Curiosity case study that we can still convert some of these to preserve optimality, by expressing them as the sum of a PBS function and a bounded monotone BAMPF. But in general, the conversion process is manual and may not be immediately obvious. We have added discussion of these limitations to section 4.3.2 and the Discussion section.

We thank the reviewer for their feedback for making the figures and tables more legible and informative. We fleshed out the captions, added more space between the subfigures and labeled the colorbar. We have simplified figure 2 and believe it is now much easier to read. We have added citations to every term in Table 1.

[1] Wang, Jane X., et al. "Learning to reinforcement learn." arXiv preprint arXiv:1611.05763 (2016).

[2] Pathak, Deepak, et al. "Curiosity-driven exploration by self-supervised prediction." International conference on machine learning. PMLR, 2017.

I thank the authors for their response. The added empirical validation has increased my confidence in the proposed framework and I have updated my score accordingly. As per the case study, what was presented in the updated draft is too brief and not very helpful to the reader if they are not familiar with Curiosity. I encourage the authors to include an appendix that expands upon the brief paragraph included in section 4.3.2, such that it presents the conversion process explicitly, with equations and, ideally, some simple numerical experiment (note that I will not decrease my current score if the appendix is not included, however I am happy to further increase my score if it is).

We thank reviewer AbDy for their fast response. We really appreciate the suggestion to provide more background on Curiosity and include a simple experiment-- we agree this would make the case study more helpful and concrete. Thus, we have expanded on it in Appendix D in the revision we just uploaded: in D.1 we introduce Curiosity more thoroughly, in D.2 we go through the conversion explicitly with equations, and finally in D.3 we present a simple experiment where we recreate the Noisy TV problem, and show that the converted version of Curiosity is not susceptible to it. We hope this in-depth example is helpful for readers looking to convert their own problematic intrinsic motivation/reward shaping terms into a form that preserves optimality. Thank you for helping us further improve our manuscript!

We thank reviewer jfJX for their helpful comments. We are glad that they find our work to be novel and well-motivated; we agree that it addresses a critical challenge in reward modeling.

We agree that experiments in more complex and realistic settings would be helpful for demonstrating our theory. We added two deep RL experiments to section 4:

• In Mountain Car, which involves a two-dimensional continuous state space and simulates motion under Newton’s laws, we demonstrated a BAMPF pseudo-reward signalling VOI helps PPO learn a goal-reaching policy more quickly while preserving optimality. In contrast, regular potential-based shaping did not provide any improvement, and adding similar non-potential based rewards initially helped exploration, but eventually led to a reward hacking policy that avoids the goal to collect more pseudo-rewards

• In the Bernoulli Bandit meta-RL setting introduced by [1] we show that a BAMPF helps the A2C meta-learner find the optimal RNN-based RL agent more quickly by discouraging over-exploration, while the non-BAMPF version of the same pseudo-reward led to a reward-hacking RL agent that over-exploits.

See the revised manuscript and the overall response for more details of these new experiments.

We appreciate the feedback on the clarity of the writing. To make the paper easier to follow, we have gone through the paper and broken up long sentences, and reduced the use of technical jargon where possible. We also moved the “RL algorithms” paragraph to section 2.3 as suggested.

[1] Wang, Jane X., et al. "Learning to reinforcement learn." arXiv preprint arXiv:1611.05763 (2016).

We thank reviewer kw7i for their generally positive review of our paper. We are glad that they find our results to be novel, high quality and important.

Experiments: We agree that more examples, particularly larger-scale and longer-horizon empirical examples, would be helpful to demonstrating the impact of our theoretical results. We added two more experiments to section 4; in both we use our theory to understand the pathologies of various pseudo-rewards, and to design BAMPF pseudo-rewards that improve learning while avoiding reward hacking.

• In Mountain Car, which involves a 2D continuous state space and episodes up to 200 steps long, we trained PPO for 250k steps. We demonstrated a BAMPF pseudo-reward which signals VOI helps PPO learn a goal-reaching policy more quickly. In contrast, regular potential-based shaping did not provide any improvement, and similar non-potential based rewards initially helped exploration, but eventually led to a reward hacking policy that avoids the goal to collect more pseudo-rewards

• In the Bernoulli Bandit meta-RL setting introduced by [1] we used A2C to learn an RNN-based RL agent over 4000 updates/256k episodes. BAMPF helps the A2C meta-learner find the optimal RNN-based RL agent more quickly by discouraging over-exploration, while the non-BAMPF version of the same pseudo-reward led to a reward-hacking RL agent that over-exploits.

We hope these additional experiments better demonstrate the utility of our framework. See the revised manuscript and the overall response for more details of these new experiments.

Theory: We appreciate the feedback about the conceptual clarity and definition-heaviness of our theory, and have done some restructuring which we hope will make it easier to follow. We modified our theorem on preserving MDP optimality (Theorem 4.3 in the revised manuscript) to bypass the intermediate definition of Settling, which we believe makes it clearer and more direct. We also expanded the BAMPF abbreviation in the statement of Theorem 4.3 (now 4.2 in the revised manuscript), making it less jargon-heavy. Though this result might not seem surprising, it is not currently widely appreciated: to the best of our knowledge, we are the first to design potential-based reward shaping for meta-RL. Given it is natural and intuitive, we believe it is particularly valuable to share with the community.

Thank you for pointing out the blurriness of the plots, we have remade them in higher resolution.

[1] Wang, Jane X., et al. "Learning to reinforcement learn." arXiv preprint arXiv:1611.05763 (2016).