Towards Robust Zero-Shot Reinforcement Learning

We thank the reviewer for the constructive comments. Regarding the concerns of the reviewer 5D2W, we provide the following responses.

Weakness 1 & Question 5 & Limitation 1 [Comments regarding the use of attention architecture]

• We provided a discussion in Section 3 to describe why we need to deprecate standard MLP architectures in FB. This is primarily because learning a successor measure for all possible tasks is actually an extremely complex task, which requires sufficient expressivity for modeling. Moreover, as we have shown in Fig.2, simply scaling up MLP parameters does not provide performance improvement; by contrast, using an attention-based architecture greatly boosts the performance.
• The reason we choose attention-based architecture in our design is threefold:
  1. Attention mechanism and the derived transformer architecture have already demonstrated great success in terms of modeling capability and scalability. We want to build our model based on a well-verified architecture.
  2. As illustrated in Fig. 3, the attention mechanism naturally favors capturing the complex dependencies among different dimensions of state, action, and task vectors.
  3. In our preliminary exploration, we actually tested other network architectures, such as BRO [1] and OFENet [2]. We found that these structures do not provide meaningful performance improvement over the MLP-based FB architecture. Due to the page limit, we only reported the final architecture that works (i.e., attention-based architecture in Fig. 3), rather than these failed attempts.
• Regarding the concern from the reviewer that "the inputs are only 1 or 2 tokens". Each "token" here actually refers to a 512-dim embedding vector encoded by the encoder. It is different from the typical discrete token used in NLP tasks. We apologize for the inaccurate description in our initial submission and have revised it in our final paper.

Weakness 2 and Question 4 [Regulating towards the exploratory policy & how to get $\mu(a|s)$]

• It should be noted that the original FB objective (Eq. (3)) suffers from similar OOD error exploitation issues as in offline RL, where we have discussed this issue in the last paragraphs in Sections 2 and 3. Without proper OOD regularization, the FB policy learning could suffer from serious bias and learning instability. Some recent FB-based methods like MCFB/VCFB (Ref[7] in our paper) have also recognized this issue and used a much more conservative CQL-based regularization to restrict the learned policy from deviating too much from the data distribution (i.e., the exploratory policy mentioned by the reviewer).
• By contrast, as described in Section 4.1, we adopt a implicit value/policy regularization scheme in BREEZE, that is, through proper reformulation (e.g., leveraging expectile regression in Eq.(7), and using the closed-form optimal policy in Eq.(9)), we do not need to explicitly learn the exploratory policy $\mu(a|s)$ in the dataset or enforce explicit policy constraint, but instead, we can directly using dataset sample to implicit enforce behavior regularization. This scheme is called in-sample learning in offline RL literature [3,4], which is less conservative as compared to the explicit policy constraint or the CQL-based scheme used in MCFB/VCFB, while also providing better performance and learning stability.

Weakness 3 & Question 1 [Negative values in FB]

• In Section 3, for the successor measure $M^{\pi}$, as it is an expected discounted probability (see Eq. (1)), it should always be positive. But as we have shown in Fig. 1 as well as more results in Fig. 8-10 in the appendix, both FB and MCFB can produce large negative values for $M^{\pi}$, which is obviously problematic.
• As for $Q_z$, by definition of value function in RL, for a reward function with range $[0, r_{max}]$, the theoretical Q-value should fall within the range $[0, r_{max}/(1-\gamma)]$. In the Walker task in Fig. 1, the actual task reward is in the range $[0,1]$, so we should not expect any negative Q-values. However, similar to the case $M^{\pi}$, we again find FB and MCFB produce problematic Q-values with large negative values. By contrast, BREEZE produces much more reasonable Q-value estimates with proper scaling.
• The underlying cause for this phenomenon is that the FB learning objective Eq.(3) completely depends on the bootstrapped update based on learned $F$ and $B$, without any external scale supervision. This is different from the typical Bellman update used in standard RL, as we have reward values as external scale supervision. Hence, the high quality of $F$ and $B$ estimates becomes essential for unbiased FB learning, which requires sufficient model expressivity and careful handling of the impact of OOD error exploitation. These are exactly what BREEZE has addressed. The improvement in the final empirical results also demonstrates the effectiveness of our design.

Weakness 4 [Minor typo] We sincerely apologize for the typos and thank the reviewer's correction. We will revise these issues pointed out by the reviewer in the final version of our paper.

Question 2 [Question regarding Eq.(4) and (5)] The incorporation of $E[BB^T]$ is introduced in the original FB paper [5] rather than proposed by us, which is also adopted in all later FB-based ZSRL works like [6, 7]. Actually, in FB methods, $B$ is designed as a function that maps reward to a task vector $z$ (i.e., $z=E_s[r(s)B(s)]$), hence $B(s')^\top E[BB^\top]^{-1}z$ corresponds to the reward estimate based on task vector $z$. For more details, we refer reviewers to the original FB paper [5, 6].

Question 3 [Typo in Eq. (6)] We apologize for this typo. Yes, we only max $V_{\pi_z}$ over $a\in A$. We will remove $z\in Z$ under the max operator in our final version.

Question 6 [Half-time training on HILP] We apologize that HILP could only plot half-time curves. We used the official codebase with default environment settings and hyperparameters but encountered an environment error around 600k training steps in every domain. Currently, we are not fully clear about the cause of this problem and are still investigating.

To ensure the correctness, we rigorously checked HILP's original reported results when plotting curves and recording scores, confirming that our obtained peak performances are consistent with the results reported in the HILP paper.

We will continue fixing this issue and try to obtain the complete curve of HILP in the final version of our paper.

Question 7 & Limitation 2 [Concern on performance and computation speed]

• We would like to clarify that BREEZE outperforms all baselines across most domains and datasets, as shown in Table 5 in the Appendix. Notably, it achieves strong performance simultaneously on all tasks within each domain.
• Moreover, if the reviewer checks Fig. 5, BREEZE converges much faster than other baseline methods, typically reaches good performance within just 100k training steps, and enjoys stable convergence. By contrast, other baselines need much larger training steps (typically about 300k-500k steps), and often suffer from unstable convergence.
• To demonstrate a more detailed computation cost vs performance trade-off, we report the training time per 100k steps and the corresponding aggregated scores in the following table:

  Methods	Training Time (h) per 100k Steps	IQM Walker 100k Steps	IQM Jaco 100k Steps	IQM Quadruped 100k Steps	Aggregate IQM Score 100k Steps
  FB	0.4	448±12	17±2	270±15	245±6
  VCFB	1.2	373±15	18±2	300±27	231±10
  MCFB	1.2	415±16	17±1	270±20	234±8
  HILP	0.75	509±9	30±1	394±15	311±6
  BREEZE	2	615±7	45±3	542±11	404±4

• The scores reported for each domain are averaged over five random seeds and all datasets. The aggregate score denotes the overall performance by averaging all domains in ExORL. Based on the above table and the learning curve in Fig. 5, we can observe that BREEZE reaches much better performance with a reasonable increase in training cost.

Note: each experiment was conducted on a single NVIDIA A6000 GPU.

Question 8 [What happens if $\tau=1$] When $\tau=1$, expectile regression effectively discards all data, equivalent to learning nothing. In our experiments, we used $\tau=0.99$ for all ExORL tasks, and $\tau=0.7$ for all Kitchen tasks (see Section B.7 in our Appendix).

Limitation 3 [Lacking ablation on IQL loss]

• Adding behavior regularization directly on a TD-type loss will require adding explicit policy regularizations, i.e., adding $D(\pi || \mu)$ to avoid OOD action exploitation, where $D()$ is some divergence function. As we have explained in the response to Weakness 2, this will require learning an explicit $\mu$ and can be more conservative. We refer the reviewer to the in-sample learning offline RL method papers [3, 4] for more detailed information regarding the advantage of using IQL-style loss in the offline RL setting.

References

[1] Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłos. Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control. NeurIPS 2024.

[2] Kei Ota, Devesh K. Jha, and Asako Kanezaki. Training Larger Networks for Deep Reinforcement Learning.

[3] Kostrikov, Ilya, Ashvin Nair, Sergey Levine. Offline reinforcement learning with implicit q-learning. ICLR 2022.

[4] Xu, Haoran, et al. Offline RL with no ood actions: In-sample learning via implicit value regularization. ICLR 2023.

[5] Ahmed Touati, Yann Ollivier. Learning One Representation to Optimize All Rewards. NeurIPS 2021.

[6] Ahmed Touati, Jérémy Rapin, Yann Ollivier. Does Zero-Shot Reinforcement Learning Exist? ICLR 2023.

[7] Jeen, Scott, Tom Bewley, Jonathan Cullen. Zero-shot reinforcement learning from low quality data. NeurIPS 2024.

We sincerely appreciate the reviewer Arsf for the positive feedback and the constructive comments on our paper. Regarding the comments of the reviewer, we provide the following responses.

Weakness 1, 2 [Incremental contribution & extra machineries]

• Although our work is built upon the FB framework, we provide the first evidence that conventional FB-based zero-short RL (ZSRL) methods suffer from systematically biased successor measure and Q-value estimations, due to the absence of proper scale supervision on the FB learning objective. We show in our paper that enhancing the model expressivity and properly handling the ODO error exploitation are crucial to solving this problem.
• It should be noted that the extra machinery introduced in BREEZE is targeted specifically to address the above issues. For example,
  • We introduce expressive self-attention architecture for $F$ and $B$ to capture the complicated state-action-task relationships and enhance successor measure learning quality.
  • We introduce an implicit behavior regularization schemes for both $F$ and policy extraction, which enable a closed-form optimal policy (Eq. (9)) and can naturally be modeled using a weighted diffusion regression (Eq.(10)).
• As the reviewer can notice, the above design is a very logical combination. Removing either component will reintroduce the original limitation in the conventional FB-based method.

Weakness 1, 2 [Performance comparison with HILP]

• The baseline HILP is actually not a FB-based ZSRL method. It is constructed upon a goal-conditioned RL framework, by learning a Hilbert representation for state space, and using the normalized distance between the current state and the goal state under the Hilbert representation as the task vector. The final policy in HILP is learned by running an off-the-shelf offline RL algorithm as a sub-routine.
• As the underlying mechanism of HILP and FB-based method is very different, thus it is not very suitable to directly compare with BREEZE's variants without FB enhancement or diffusion component. These variants are more suitable for comparison with FB-based methods like FB or VCFB/MCFB, and as we have reported in Table 2, even these incomplete variants could have stronger performance than conventional FB-based methods.
• Lastly, as we have reported in Table 1 and Fig. 5, in almost all tasks, our complete BREEZE algorithm achieves stronger performance and learning speed as compared to HILP.

Question 1 [Interpretation of Figure 1]

• In Fig. 1, for the successor measure $M^{\pi}$, as it is an expected discounted probability (see Eq. (1)), it should always be positive. But as we have shown in Fig. 1 as well as more results in Fig. 8-10 in the appendix, both FB and MCFB can produce large negative values for $M^{\pi}$, which is obviously problematic.
• As for $Q_z$, by definition of value function in RL, for a reward function with range $[0, r_{max}]$, the theoretical Q-value should fall within the range $[0, r_{max}/(1-\gamma)]$. In the Walker task in Fig. 1, the actual task reward is in the range $[0,1]$, so we should not expect any negative Q-values. However, similar to the case $M^{\pi}$, we again find FB and MCFB produce highly problematic Q-values. By contrast, BREEZE produces much more reasonable Q-value estimates with proper scaling.

Question 2 [Orthonormalization regularization]

• We'd like to clarify that orthonormalization loss can only regularize the scale and orthogonal property of $B$, but cannot help to address the scaling problem of the successor measure $M$ and Q-value $Q$. Note that $M$ is modeled as $M=F^\top B$, solely normalizing $B$ is insufficient to enforce proper scaling of $M$.
• Also, the FB learning objective Eq.(3) completely depends on the bootstrapped update based on learned $F$ and $B$, without any external scale supervision. This is different from the typical Bellman update used in standard RL, as we have reward values as external scale supervision. Hence, the high quality of $F$ and $B$ estimates becomes essential for unbiased FB learning, which requires sufficient model expressivity and careful handling of the impact of OOD error exploitation.

Question 3 and 4 [Diffusion guidance and missing citation]

• Yes, it is possible to perform guided diffusion by using a weighted term to the noise prediction at each step. We refer the reviewer to the detailed proof of Theorem 2 in a prior work [1].
• Paper [1] actually is the Ref[42] in Lemma 1 in Appendix A.2. We apologize for the missing reference. For some unknown reason, this citation was not properly compiled in our submission. We will correct this issue in the final version of our paper.

References

[1] Zheng, Yinan, et al. Feasibility-guided safe offline reinforcement learning. ICLR 2023.

We thank the reviewer for the constructive comments. Regarding the concerns of the reviewer Ytsj, we provide the following responses.

Weakness 1 [Clarity & writing] We thank the reviewer for the suggestion. We will perform thorough language editing and polish the writing in our final version.

Weakness 2 [Preliminaries & task formalization]

• We apologize for the relatively brief introduction on the problem setting of zero-short RL (ZSRL) due to the page limit. In short, ZSRL trains a policy in an unsupervised manner based on a dataset collected from a specified domain without reward. The desired reward function will be specified at test time with the same domain, and the learned agent must immediately produce a good policy without fine-tuning (i.e., zero-shot). We will add more preliminary information in our final version. We also refer the reviewer to the prior papers [1,2,3] in FB-based ZSRL for more detailed information.
• Regarding the discount factor, it is actually necessary to include the discount factor $\gamma$ in FB-based ZSRL, as the central element in this problem, the successor measure $M^{\pi}$ (see Eq.(1) in our paper) is defined as the expected discounted occurrences of a future state after staring from a given state-action pair. The FB-based ZSRL methods estimate this value by solving a Bellman-like equation $M^{\pi}=\mathcal{P}+\gamma \mathcal{P}_{\pi}M^{\pi}$ (Eq.(3)), in which $\gamma < 1$ is also needed to ensure convergence to a fixed point during learning.

Weakness 3 & Question 2 [Ambiguity in zero-shot framing]

• We actually share the same feeling on the potential ambiguous term "zero-short RL", as it is essentially a class of unsupervised RL (training w/o task/reward specifications) that generalizes to arbitrary tasks at test time. However, the term "zero-short RL" has already been used extensively in the existing literature, e.g., [1,2,3,4,5,6], hence we follow this naming convention from prior works in our paper. We thank the reviewer for the suggestion and will include a clearer definition in our final version.

Weakness 4 & Question 3 & Limitation 1 [Missing limitations & assumptions & discussion on failure cases]

We thank the reviewer for this constructive comment.

• We do have a limitation section in Appendix D.4 in our paper. The main limitation of BREEZE is that it has increased computation demand due to the use of more expressive model architectures. But as it enables noticeable improvements in learning performance and stability, we believe it is a reasonable trade-off.
• Regarding the assumptions:
  • BREEZE inherits all theoretical assumptions from FB, e.g., the MDP assumption, as well as assuming for each task vector $z$, we can find a pair of forward ($F_z$) and backward ($B$) representations such that we can decompose $M^{\pi_z}$ to be $F^\top_zB$.
  • The availability and quality of offline data require minimal assumptions, as the goal of ZSRL is to learn the environment structure from arbitrary datasets and handle any downstream tasks. In practice, an offline dataset with higher diversity is likely to provide better ZSRL performance, which has been discussed in the prior paper [2].
  • Regarding the use of KL constraint: The KL-based behavior regularization is widely adopted in offline RL literature to prevent the learned policy from deviating too much from the dataset distribution. We borrowed a similar design in Eq.(9) as the FB-based ZSRL problem suffers from a very similar out-of-distribution (OOD) exploitation problem as in offline RL. A more important benefit of this design is that it can enable a closed-form optimal policy (Eq.(9)), which can be seamlessly integrated with a diffusion policy learning objective (Eq.(10)).
• Lastly, we will consider adding additional failure case analysis in the final version of our paper, for example, testing the cases with small-sample and low-cover datasets (the OOD problem will become more severe), as well as testing the possible over-conservative issue due to adding additional behavior regularizations.

Weakness 5 [Related works]

We thank the reviewer's recommendation. Reference [7] provides an effective approach to learn improved representations under partial observability. By contrast, FB-based ZSRL methods focus on learning a special type of successor representation that can facilitate zero-shot policy adaptation. The goals of these two representation learning methods are very different. However, we will add extra discussion with [7] in our final paper.

Weakness 6 & Limitation 2 [Computational details]

We appreciate the reviewer's comments and would like to provide the following details.

• In our experiments, we train BREEZE on a single NVIDIA A6000 GPU. Peak GPU memory usage varied by tasks, normally below 23,000MB.
• As BREEZE uses more expressive model architectures, it requires more computational resources for training. Training for 1M steps, as adopted in other baseline methods, takes about 20 hours for most environments. However, as we have reported in Fig. 5, BREEZE can learn much faster, typically reaches good performance within just 100k steps, and enjoys stable convergence. By contrast, other baselines need much larger training steps (typically about 500k steps), and often suffer from unstable convergence (performance drop in later training stages).
• To demonstrate a more detailed computation cost vs performance trade-off, we report the training time per 100k steps and the corresponding aggregated scores in the following table:

  Methods	Training Time (h) per 100k Steps	IQM Walker 100k Steps	IQM Jaco 100k Steps	IQM Quadruped 100k Steps	Aggregate IQM Score 100k Steps
  FB	0.4	448±12	17±2	270±15	245±6
  VCFB	1.2	373±15	18±2	300±27	231±10
  MCFB	1.2	415±16	17±1	270±20	234±8
  HILP	0.75	509±9	30±1	394±15	311±6
  BREEZE	2	615±7	45±3	542±11	404±4

• The scores reported for Walker, Jaco, and Quadruped tasks are averaged over five random seeds and all datasets of this domain. The aggregate score denotes the overall performance by averaging all domains in ExORL. Based on the above table and the learning curve in Fig. 5 of our paper, we can observe that BREEZE reaches much better performance over baselines with a reasonable increase in training cost.

Question 1 ["Ground truth" on Figure 1]

• For the successor measure $M^{\pi}$, as it is an expected discounted probability (see Eq. (1)), it should always be positive. But as we have shown in Fig. 1 as well as more results in Fig. 8-10 in the appendix, both FB and MCFB can produce large negative values for $M^{\pi}$, which is obviously problematic.
• As for $Q_z$, by definition of value function in RL, for a reward function with range $[0, r_{max}]$, the theoretical Q-value should fall within the range $[0, r_{max}/(1-\gamma)]$. In the Walker task in Fig. 1, the actual task reward is in the range [0,1], so we should not expect any negative Q-values. However, similar to the case $M^{\pi}$, we again find FB and MCFB produce highly problematic Q-values. By contrast, BREEZE produces much more reasonable Q-value estimates with proper scaling.

Limitation 3 [Domain shift consideration]

• The ZSRL problem setting [1,2,4,5,6] only considers zero-shot adapting to different tasks in the same domain (e.g., walk, stand, run, flip tasks for Walker domain), rather than focusing on solving domain shift problems. Future research can explore learning agents that have both task and domain generalization capabilities, but it is not the current focus of our paper.

Paper Formatting Concerns

We sincerely appreciate the reviewer's suggestion. We will revise these issues pointed out by the reviewer in the final version of our paper.

References

[1] Ahmed Touati, Jérémy Rapin, Yann Ollivier. Does Zero-Shot Reinforcement Learning Exist? ICLR 2023.

[2] Jeen, Scott, Tom Bewley, Jonathan Cullen. Zero-shot reinforcement learning from low quality data. NeurIPS 2024.

[3] Ahmed Touati, Yann Ollivier. Learning One Representation to Optimize All Rewards. NeurIPS 2021.

[4] Park, Seohong, Tobias Kreiman, Sergey Levine. Foundation policies with Hilbert representations. ICML 2024.

[5] Ingebrand, Tyler, Amy Zhang, and Ufuk Topcu. Zero-Shot Reinforcement Learning via Function Encoders. ArXiv 2401.17173.

[6] Frans, Kevin, et al. Unsupervised zero-shot reinforcement learning via functional reward encodings. PMLR 235, 2024.

[7] Altmann et al., CROP: Towards Distributional-Shift Robust Reinforcement Learning using Compact Reshaped Observation Processing. IJCAL 2023.

We thank the reviewer for the constructive comments. Regarding the concerns of the reviewer WnNG, we provide the following responses.

Question 1 [Compute footprint & efficiency]

• In our experiments, we train BREEZE on a single NVIDIA A6000 GPU. Peak GPU memory usage varied by tasks, normally below 23,000MB.
• As BREEZE uses more expressive model architectures, it requires more computational resources for training. Training for 1M steps, as adopted in other baseline methods, takes about 20 hours for most environments. However, as we have reported in Fig. 5, BREEZE can learn much faster, typically reaches good performance within just 100k steps, and enjoys stable convergence. By contrast, other baselines need much larger training steps (typically about 500k steps), and often suffer from unstable convergence (performance drop in later training stages).
• To demonstrate a more detailed computation cost vs performance trade-off, we report the training time per 100k steps and the corresponding aggregated scores in the following table:

  Methods	Training Time (h) per 100k Steps	IQM Walker 100k Steps	IQM Jaco 100k Steps	IQM Quadruped 100k Steps	Aggregate IQM Score 100k Steps
  FB	0.4	448±12	17±2	270±15	245±6
  VCFB	1.2	373±15	18±2	300±27	231±10
  MCFB	1.2	415±16	17±1	270±20	234±8
  HILP	0.75	509±9	30±1	394±15	311±6
  BREEZE	2	615±7	45±3	542±11	404±4

• The scores reported for Walker, Jaco, and Quadruped tasks are averaged over five random seeds and all datasets of this domain. The aggregate score denotes the overall performance by averaging all domains in ExORL. Based on the above table and the learning curve in Fig. 5 of our paper, we can observe that BREEZE reaches much better performance over baselines with a reasonable increase in training cost.

Question 2 & Limitation 1 [Pixel-based experiments]

• We appreciate the reviewer's suggestion. In our paper, we follow other zero-shot RL literature and conduct the experiments on the widely used ExORL/Kitchen benchmarks. We are implementing pixel-based experiments in DMControl and will promptly update the results if we can finish producing results within the discussion period. These results will be included in our final version.
• Regarding Atari experiments, due to the lack of multi-task settings and relevant datasets, they have not yet been used for zero-shot RL in the task setting.

Question 3, 4 & Limitation 3 [Futher ablation & analysis]

• We appreciate the reviewer's suggestion. We will further conduct relevant experiments and provide analysis on the potential impact of critic bias and diffusion sampling.

• As for the importance of the diffusion model:

  • Regarding the Kitchen environment, we have finished ablation on the setting w/o diffusion actor, as shown in the following table:

  Setting	Dataset	Average Normalized Return
  with Diffusion	kitchen-partial	72.5
  with Diffusion	kitchen-mixed	75.0
  w/o. Diffusion	kitchen-partial	33.3
  w/o. Diffusion	kitchen-mixed	37.5

  • Combining the above results with Table 2 in our paper, we can clearly observe the benefits of using diffusion policy in BREEZE for all tasks, which helps enhance policy expressivity and enables better capturing complex data distributions.

Limitation 2 [Parameter sensitivity]

• In our ablation experiments shown in Fig. 6, the y-axis labels actually start at 650 and 670 instead of 0. The performance varies in the ranges of [650, 700] for $\tau\in$ [0.5, 0.99] and [670, 700] for $\alpha\in$ [0.01, 0.5]. The performance variations are actually not as significant as they seem to be.
• Moreover, if the reviewer checks Tables 3 and 4 in the Appendix of our paper, we used the consistent expectile $\tau$ (0.99 for all ExORL tasks and 0.7 for Kitchen) and guidance weight $\alpha$ (0.05 for all tasks) for all our experiments without any hyperparameter tuning. Even using untuned hyperparameters for different tasks, BREEZE still achieves better performance. This also demonstrates the hyperparameter robustness of BREEZE.

Limitation 4 [Comparison to function-encoder lines]

• We appreciate the reviewer's suggestion. We have managed to conduct additional experiments on the baseline FRE [1] for the walker domain in the ExORL benchmark during the short 1-week rebuttal period. We use the official codebase in the FRE paper and modify the evaluation part for rollout in our task environment. The results averaged over three random seeds are shown in the table below, and we will report FRE's scores on other tasks and domains in our final version.

  Dataset	Tasks (Walker)	FRE IQM Score	BREEZE IQM Score
  RND	Walk	187.7±9	945±3
  RND	Run	88.3±3	340±16
  RND	Stand	407.3±19	950±1
  RND	Flip	242.3±5	593±18
  RND	Whole Performance	232±8	703±2
  APS	Walk	162.3±9	949±4
  APS	Run	56.7±11	278±9
  APS	Stand	376.0±19	922±33
  APS	Flip	104±5	589±15
  APS	Whole Performance	175±12	685±7
  PROTO	Walk	177.3±2	923±7
  PROTO	Run	115.7±6	274±9
  PROTO	Stand	341.3±7	930±4
  PROTO	Flip	187.7±35	722±13
  PROTO	Whole Performance	190±11	711±3
  DIAYN	Walk	83.3±12	774±21
  DIAYN	Run	99.0±7	229±10
  DIAYN	Stand	336.3±10	781±49
  DIAYN	Flip	94.0±22	395±14
  DIAYN	Whole Performance	153.2±2	545±12

• As shown in the above table, BREEZE consistently outperforms FRE on Walker tasks. Function-encoder ZSRL methods, such as FRE, rely heavily on reward prior assumptions and extensive training/tuning, but they have fewer theoretical guarantees compared to FB-based ZSRL methods.

References

[1] Frans, Kevin, et al. Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings. ICML 2024.