Unsupervised Zero-Shot Reinforcement Learning via Dual-Value Forward-Backward Representation

Q5. Have you performed any analysis to assess the sensitivity of DVFB to essential hyperparameters? This would be beneficial to evaluate the resilience of the framework across diverse contexts and circumstances.

A5. Thank you for highlighting the importance of hyperparameter sensitivity analysis. To address this concern, we conduct a series of ablation studies to evaluate the impact of the key hyperparameters $\alpha$, $\beta$, and $\eta$ on DVFB’s performance.

Ablation Study on $\alpha$. Table 5 shows the results for varying $\alpha$ in the Walker domain. The experiments indicate that while changes in $\alpha$ affect performance slightly, the overall generalization performance of DVFB remains stable.

Table 5 Ablation Study on $\alpha$

Ablation Study on $\beta$. Similarly, Table 6 reports the performance for different $\beta$ values in the Walker domain. The results reveal that DVFB exhibits only minor deviations across different values. This demonstrates the robustness of our framework to changes in $\beta$.

Table 6 Ablation Study on $\beta$

Ablation Study on $\eta$. We conduct an ablation study in the Quadruped domain to evaluate the sensitivity of the DVFB framework to the hyperparameter $\eta$ during the fine-tuning phase. As shown in Table 7, the overall average performance across all tasks remains consistent, indicating that DVFB is relatively resilient to changes in $\eta$.

Table 7 Ablation Study on $\eta$

The results of our sensitivity analysis demonstrate that DVFB is resilient to variations in key hyperparameters $\alpha$, $\beta$, and $\eta$. Furthermore, for all other neural network hyperparameters (e.g., learning rate), we adopt the default settings of URL with DDPG, ensuring consistency with prior work.

Q6. Could you elaborate on the possible limits of DVFB, including computational complexity and scalability in practical applications? Considering these criteria would yield a more equitable perspective on the approach's practical viability.

A6. Thank you for raising questions about DVFB's practical limitations. We conduct comprehensive experiments to analyze its computational complexity and scalability across different domains.

Computational Analysis.

We compare the training time of DVFB with several baseline methods on identical hardware (RTX 3090). As shown in Table 8, DVFB requires approximately 19 hours of pre-training time, which is moderate compared to other methods:

Table 8 Training time comparison across different methods

While DVFB's training time is higher than some baselines due to its dual-value architecture, we believe this computational overhead is justified by its superior performance.

Further Domain Evaluation.

We evaluate DVFB's scalability across different domains, including point-mass maze and manipulation tasks. Table 9 shows the performance comparison. The results demonstrate DVFB's generalization capabilities across different environments, consistently outperforming baseline methods.

Table 9 Performance comparison across different domains

Results marked with * are sourced from MCFB (NeurIPS 24) [3].

Practical Limitations.

In summary, although DVFB demonstrates exceptional zero-shot performance in simulation environments, it still has room for further optimization in terms of computation time. In addition, the effectiveness of DVFB in varied real-world contexts such as quadruped robot control, robot manipulation, and so on is still unknown and needs to be further explored.

---

References

[1] Yang Y, Zhou T, He Q, et al. Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning. ICLR, 2024.

[2] Touati A, Rapin J, Ollivier Y. Does zero-shot reinforcement learning exist?. ICLR, 2023.

[3] Jeen S, Bewley T, Cullen J. Zero-Shot Reinforcement Learning from Low Quality Data. NeurIPS, 2024.

Q4. In what manner does the contrastive entropy reward facilitate skill differentiation, and can you present empirical data that substantiates its efficacy? An elucidation or ablation of the role of this reward would improve comprehension.

A4. We agree that a deeper analysis of contrastive entropy reward would strengthen our work.

Contrastive Entropy Reward. Our contrastive entropy reward is specifically designed to encourage exploration while keeping skill separability through two mechanisms:

• Skill Discriminator. We employ contrastive learning to train a skill discriminator that learns representation for trajectories and skills. The dot product between trajectory representation and skill representation serves as a similarity measure. The learned discriminator is used to distinguish skill-trajectory pairs.
• Contrastive Entropy. We maximize a particle entropy computed from the dissimilarity between a trajectory and its N most related skills. This encourages the agent to keep skill separability while exploring new states.

Ablation Study. We perform an ablation study on the coefficient $\beta$ of contrastive entropy reward $r_{intr} = r_{rnd} + \beta r_{ce}$, as shown in Table 3. The results demonstrate that increasing $\beta$ from 0.1 to 0.7 consistently improves performance, validating that contrastive entropy enhances generalization by promoting skill separability. However, further increasing $\beta$ negatively impacts performance due to an imbalance between skill learning and exploration.

Table 3 Ablation study on contrastive entropy coefficient $\beta$ on Walker tasks. Results show mean ± standard deviation across three seeds.

Comparison Experiment. We further compare DVFB with its variants using alternative intrinsic rewards (ICM-APT, Proto, and CIC), as presented in Table 4.

Table 4 Comparison with alternative intrinsic rewards on Walker tasks. Results show mean ± standard deviation across three seeds.

The results reveal several key insights:

1. The scalability of DVFB. All variants achieve reasonable zero-shot generalization performance, demonstrating DVFB's compatibility with different intrinsic rewards.
2. The advantage of CE reward. DVFB with contrastive entropy consistently outperforms other variants, achieving the highest average performance.

These experiments provide evidence for the effectiveness of our contrastive entropy design, while also demonstrating DVFB's flexibility in incorporating different intrinsic rewards.

Q2. What is the performance of DVFB in relation to other contemporary zero-shot generalization methods, and what are the reasons for the selection or exclusion of specific baselines? Incorporating a broader array of comparisons or elaborating on these selections would bolster the assertion of enhanced performance.

A2. We appreciate the reviewer's valuable suggestion regarding the comparison with contemporary zero-shot methods. In this paper, we address zero-shot generalization in online URL and aim to develop a method that combines both zero-shot generalization capability and fine-tuning ability. To the best of our knowledge, no existing method achieves zero-shot generalization in online URL. Therefore, we choose the baselines as they represent the closest existing approaches to our problem setting (zero-shot generalization in online URL).

USD methods (e.g., BeCL, CeSD) are designed for online unsupervised pre-training and aim to learn transferable skills. While zero-shot generalization would be ideal, they typically rely on fine-tuning in downstream tasks due to limitations in mutual information-based skill learning[1]. Our results in Table 1 demonstrate that DVFB achieves zero-shot performance comparable to their fine-tuned performance, and DVFB outperforms them in fine-tuning.

Table 1 Comparison of Zero-shot and Fine-tune Performance

SR methods (e.g., LRA-SR,FB), while originally designed for offline settings, represent SOTA in zero-shot generalization. To ensure a fair comparison of zero-shot capabilities, we implemented their online versions following the authors' official code.

Our experimental setup ensures fair comparison by evaluating methods under identical conditions (same interaction and fine-tuning steps), demonstrating DVFB's superior performance in both zero-shot and fine-tuning scenarios.

Baseline Categories. To make it easier to understand the key properties of each method and their relationships, we categorize the methods into several groups based on their key properties as shown in Appendix C.2 (also see https://sites.google.com/view/baselinestable).

Comparison with offline settings. Original offline SR methods rely on pre-collected datasets, which diverges from our online URL setup. Hence, such methods were excluded from our primary comparisons. However, we conduct further experiments to provide a broader understanding of the relative advantages of DVFB. We compare DVFB against SR methods, including LAP, LRA-SR, LRA-P, and FB, across the three domains under offline settings. The performances of offline methods are from FB [2], where offline datasets are collected by APS, Proto, and RND exploration methods. The results, summarized in Appendix G (also see https://sites.google.com/view/zero-shot-generalization-perfo), highlight the key differences between offline and online methods:

High data sensitivity for offline methods: Offline methods exhibit significant performance variation depending on the quality of exploration data. On the one hand, the same algorithm requires different exploration datasets in different domains. For example, FB trained with Proto data in the Walker domain achieves best performance (666), while in the Quadruped domain using Proto data yields a huge performance drop (222). On the other hand, different algorithms require different exploration datasets. For example, LRA-P performs best with RND data, while FB performs best with APS data. When designing a novel algorithm, how to make sure what kind of exploration dataset is most suitable? The intuitive idea is to train the models on different exploration datasets, and compare to find the best performance models. Obviously, it will lead to high computational costs. In contrast, DVFB does not depend on offline datasets, and requires only a single agent pre-training phase to achieve zero-shot capability, significantly reducing time and computational overhead.

Performance limitation on pre-collected fixed datasets: Offline methods rely on the diversity and quality of fixed pre-collected datasets, which limits their generalization performance. In contrast, DVFB balances the exploration and exploitation online with an intrinsic reward based on contrastive learning, leading to enhanced skill learning and better zero-shot performance.

In summary, the results demonstrate that DVFB offers superior performance and efficiency compared to both offline and online methods, establishing its significance in zero-shot online URL.

Thank you for your thoughtful and detailed feedback, which will greatly help improve our work. Below are our detailed responses.

Q1. Could you furnish more detailed explanations regarding the metrics employed in the studies, especially for figures where the axes and comparisons lack clarity? Supplementary labeling and contextual information would assist readers in appropriately interpreting your findings.

A1. Thank you for your detailed feedback. We agree that these suggestions will enhance the clarity and readability of the paper. In response, we have added more detailed explanations for the figures and metrics used in the studies. Specifically, we have standardized the x-axis labels in Figure 6 and enlarged the legend fonts in Figure 7, while ensuring that the overall figure dimensions remain consistent. These adjustments aim to make the figures and their comparisons easier to interpret for readers.

---

Q3. Could you provide a detailed explanation of the practical execution of the reward mapping technique, possibly including pseudocode? Additional detail would elucidate this component's impact during fine-tuning.

A3. The reward mapping technique in our framework is designed to balance the influence of the skill value function with the downstream task value function, ensuring stable and effective fine-tuning. Below, we provide a detailed explanation of this technique and demonstrate its impact through additional experiments.

Reward Mapping Implementation. The reward mapping process involves three key steps:

1. Compute the implicit reward using the backward network (backward_net) and a latent variable (z).
2. Normalize both implicit and extrinsic rewards using running mean and standard deviation trackers.
3. Rescale the extrinsic reward to align with the scale of the implicit reward.

The detailed implementation and pseudocode is shown in Appendix K (also see https://sites.google.com/view/rewardmapping).

Potential Fine-Tuning Techniques. For fine-tuning, our objective is to ensure stable policy improvements over zero-shot performance. The reward mapping is designed to achieve stable improvement by balancing the importance of prior skill value $Q_M$ and downstream task value $Q_F$. A straightforward alternative is directly using the downstream task rewards $r_t$ for the task value, and choosing a suitable coefficient $\eta$ in Eq.11 to balance the value of $Q_M$ and $Q_F$. We call this approach DVFB w/o MAP, using a dual-value fine-tuning scheme with the downstream task value based on downstream task rewards $r_t$ rather than $r_f$. We also explore an adaptive method to dynamically adjust the $\eta$ parameter by setting $\eta = \frac{Q}{MQ}$, which we refer to as DVFB w/o MAP adaptive.

Experimental Results. We conducted comparative experiments in the quadruped domain, with the results presented in Appendix K (also see https://sites.google.com/view/figureeta) and Table 2. The findings show that DVFB w/o MAP achieves stable but limited improvement with higher coefficients, while exhibiting unstable performance with lower coefficients, ultimately constraining overall performance. It is also challenging for DVFB w/o MAP adaptive to achieve superior improvement due to the nonlinear relationship between skill value and downstream value. In contrast, DVFB with the reward mapping technique consistently delivers stable and superior improvements across all tasks. These results validate the effectiveness of our chosen approach.

Table 2 Performance Comparison of Different Fine-Tuning Approaches

Justification for the Reward Mapping Technique.
The reward mapping mechanism in DVFB helps balance the prior skill value and downstream task value, enabling stable and effective fine-tuning. Our experiments demonstrate that this approach significantly outperforms alternatives.

Dear Reviewer pWXy,

Thank you for your insightful comments. We were wondering if our response and revision have resolved your concerns. We have attempted to address your initial questions through our replies and are eager to clarify any further points you might raise. Please feel free to provide additional feedback. We greatly appreciate your continued engagement.

Best regards, Authors

Q3. Similar questions for the reward mapping technique. Could we see more justification for their approach and other alternatives explored?

We appreciate the reviewer’s thoughtful feedback on the reward mapping technique. In response, we provide a detailed justification and explore potential alternatives during our experiments.

Potential fine-tuning techniques. For fine-tuning, our objective is to ensure stable policy improvements over zero-shot performance. The reward mapping is designed to achieve stable improvement by balancing the importance of prior skill value $Q_M$ and downstream task value $Q_F$. A straightforward alternative is directly using the downstream task rewards $r_t$ for the task value, and choosing a suitable coefficient $\eta$ in Eq.11 to balance the value of $Q_M$ and $Q_F$. We call it DVFB w/o MAP, using dual-value fine-tuning with the downstream task value based on downstream task rewards $r_t$ rather than $r_f$. We also explore an adaptive method to dynamically adjust the $\eta$ parameter by setting $\eta = \frac{Q}{MQ}$, which we refer to as DVFB w/o MAP adaptive.

Experimental results. We conduct comparative experiments in the quadruped domain, with the results presented in https://sites.google.com/view/figureeta and Table 4. The findings show that DVFB w/o MAP achieves stable but limited improvement with higher coefficients, while exhibiting unstable performance with lower coefficients, ultimately constraining overall performance. It is also challenging for DVFB w/o MAP adaptive to achieve superior improvement due to the nonlinear relationship between skill value and downstream value. In contrast, DVFB with the reward mapping technique consistently delivers stable and superior improvements across all tasks. These results validate the effectiveness of our chosen approach.

Table 4: Performance comparison of different fine-tuning approaches on the quadruped domain. Results show mean ± std over three runs.

Justification for the reward mapping technique.
The reward mapping mechanism in DVFB helps balance prior skill value and downstream task value, enabling stable and effective fine-tuning. Our experiments demonstrate that this approach significantly outperforms alternatives. The consistent improvement across tasks provides strong evidence for the effectiveness of this technique. We agree that exploring other effective alternative options is a meaningful topic.

---

Q4. Can the authors provide code so that others can directly reproduce the results?

A4. Thank you for your suggestion regarding code availability to ensure reproducibility. We are committed to making all relevant code and implementation details publicly available upon acceptance of the paper to facilitate reproducibility and further research. We appreciate your understanding and support for this approach.

---

References

1. Touati A, Rapin J, Ollivier Y. Does zero-shot reinforcement learning exist?. ICLR, 2023.
2. Jeen S, Bewley T, Cullen J. Zero-Shot Reinforcement Learning from Low Quality Data. NeurIPS, 2024.

Q2. How important is their particular choice of reward to encourage exploration -- the contrastive entropy reward? How well would other rewards stand-in for this, or is it particularly well suited?

A2. Thank you for this valuable suggestion about comparing different intrinsic rewards. Our contrastive entropy reward is designed to encourage skill discrimination during exploration, which keeps skills learned by the FB mechanism. To evaluate the role of the contrastive entropy reward, we conduct two sets of comprehensive experiments.

Ablation study. We perform an ablation study on the coefficient $\beta$ of contrastive entropy reward $r_{intr}=r_{rnd}+\beta r_{ce}$, as shown in Table 2. The results demonstrate that increasing $\beta$ from 0.1 to 0.7 consistently improves performance, validating that contrastive entropy enhances generalization by promoting skill separability. However, further increasing $\beta$ negatively impacts performance due to an imbalance between skill learning and exploration.

Table 2: Ablation study on contrastive entropy coefficient $\beta$ on Walker tasks. Results show mean ± standard deviation across three seeds.

Comparison experiment. Following your suggestion, we compare DVFB with variants using alternative intrinsic rewards (ICM-APT, Proto, and CIC), as shown in Table 3.

Table 3: Comparison with alternative intrinsic rewards on Walker tasks. Results show mean ± standard deviation across three seeds.

The results reveal several key insights:

1. The scalability of DVFB. All variants achieve reasonable zero-shot generalization performance, demonstrating DVFB's compatibility with different intrinsic rewards.
2. The advantage of CE reward. DVFB with contrastive entropy consistently outperforms other variants, achieving the highest average performance.

These experiments provide strong evidence for the effectiveness of the proposed contrastive entropy reward, while also demonstrating DVFB's flexibility in incorporating various intrinsic rewards.

Thank you for your thoughtful and constructive feedback, which will greatly help in refining our work and expanding its applicability. Here are our detailed responses.

Q1. How broadly applicable is the method, particularly beyond robotic control tasks? Are there any preliminary results in other domains that the authors could include?

A1. Thank you for your insightful feedback regarding the broader applicability of DVFB beyond the DeepMind Control Suite. According to your suggestion, we conduct additional experiments on Point-Mass Maze navigation environment and Meta-World robotic manipulation environment. Although the offline methods FB-offline[1] and MCFB-offline[2] rely on pre-collected offline datasets (in contrast to online URL, high data sensitivity of offline zero-shot methods introduces a significant challenge in collecting suitable offline datasets), we provide a comparison with these offline settings for a more comprehensive evaluation of our approach. Since the Meta-World experiment is not performed in these methods, we summarize the results of FB-offline and MCFB-offline with RND offline data in Point-Mass Maze domain. As shown in Table 1, DVFB outperforms baseline methods across both domains. The additional results demonstrate that DVFB is not only effective in robotic control tasks but also generalizes well to other domains, such as navigation and robotic manipulation.

Table 1: Performance comparison across different domains. For Point-Mass Maze, results show mean ± standard deviation across three seeds. For Meta-World, results show success rates.

Results marked with * are sourced from MCFB (NeurIPS 24).

Dear Reviewer yJG4,

Thank you for your insightful comments. We were wondering if our response and revision have resolved your concerns. We have attempted to address your initial questions through our replies and are eager to clarify any further points you might raise. Please feel free to provide additional feedback. We greatly appreciate your continued engagement.

Best regards, Authors

Q5. It is unclear what guarantee of zero-shot generalization of the proposed method can have.

A5. Thank you for your insightful feedback. We agree that providing a theoretical foundation for the proposed method is crucial to support its objectives and claims. Below, we address your concerns by offering a theoretical guarantee for zero-shot generalization and a detailed analysis of how the DVFB framework improves FB’s zero-shot generalization in online URL. Theoretical guarantee for zero-shot generalization and theoretical analysis is shown in Appendix H (also see https://sites.google.com/view/theoretical-gurantee).

According to our experimental results, the DVFB framework consistently outperforms FB across diverse unseen tasks. These findings confirm the zero-shot generalization capability of the DVFB framework in online URL.

---

Q6. For Fig. 2 and Fig. 3, Please add more descriptions to the caption so that the reader can understand the main discovery and meaning of the figure.

A6. Thank you for your suggestion. We agree that the captions for Figures 2 and 3 could be clearer. We revise them as follows:

Figure 2: The $x$ axis "position" means the walking distances for different trajectories, and short-distance trajectories reflect less diverse exploration outcomes. The $y$ axis "skill index" means the index of different skills. Lines of the same color represent trajectories corresponding to the same skill. The revised caption is:

"The trajectories of the RND, FB-online, and FB-offline agents in the Walker walk task. The FB-online agent learns short-range trajectories, in contrast to the RND agent's diverse exploration and the FB-offline agent's ability to master long-range locomotion skills."

Figure 3: In (a), the x-axis represents time steps during pre-training, and the y-axis shows the episode rewards of agents. In (b), the x-axis represents trajectories categorized by episode reward ranges. The y-axis in the top figure indicates the skill values, while the y-axis in the bottom figure depicts the Spearman correlation between skill values and episode rewards. The revised caption is:

"Generalization curve and value function properties. (a) presents the agents' performance during pre-training, while (b) illustrates the normalized skill value and the Spearman correlation between skill value and return across different return ranges."

We have updated the captions and explanations of figures in the revised PDF.

---

References

1. Touati A, Rapin J, Ollivier Y. Does zero-shot reinforcement learning exist?. ICLR, 2023.

2. Yang Y, Zhou T, He Q, et al. Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning. ICLR, 2024.

Q3. A naive approach for this problem would be using a method of pure exploration or a method of skill discovery to collect an offline dataset with better data coverage than FB, then training FB on top of this dataset and testing its ability in zero-shot generalisation.

A3. We appreciate your comment for considering zero-shot offline baselines (i.e., a method of skill discovery to collect an offline dataset, then training FB on top of this dataset). Following your suggestion, we evaluate DVFB against SR methods, including LAP, LRA-SR, LRA-P, and FB, across the Walker, Quadruped, and Cheetah domains under offline settings. The performances of offline methods are from FB [1], where offline datasets are collected by APS, Proto, and RND exploration methods. The results, summarized in Appendix G (also see https://sites.google.com/view/zero-shot-generalization-perfo), highlight the following key differences between offline and online methods:

High data sensitivity for offline methods: Offline methods exhibit significant performance variation depending on the quality of exploration data. On the one hand, the same algorithm requires different exploration datasets in different domains. For example, FB trained with Proto data in the Walker domain achieves best performance (666), while in the Quadruped domain using Proto data yields a huge performance drop (222). On the other hand, different algorithms require different exploration datasets. For example, LRA-P performs best with RND data, while FB performs best with APS data. _When designing a novel algorithm, how to make sure what kind of exploration dataset is most suitable? An intuitive idea is to train the models on different exploration datasets, and compare to find the best performance models. Obviously, it will lead to high computational costs. In contrast, DVFB does not depend on offline datasets, and requires only a single agent pre-training phase to achieve strong zero-shot capability, significantly reducing time and computational overhead. It is simpler and easier to deploy than offline zero-shot methods.

Performance limitation on pre-collected fixed dataset: Offline methods rely on the diversity and quality of fixed pre-collected datasets, which limits their generalization performance. In contrast, DVFB balances the exploration and exploitation online with an intrinsic reward based on contrastive learning, leading to enhanced skill learning and better zero-shot performance. Experimental results across twelve tasks in Mujoco domains demonstrate that DVFB consistently outperforms zero-shot offline methods.

In summary, the results demonstrate that DVFB offers superior performance and efficiency compared to both offline and online methods, establishing its significance in zero-shot online URL.

---

Q4. Following the previous comment, for Table 1, it would be better to group the baselines into several categories so that it is clear from the table which property (zero-shot, online, offline, exploration or skill discovery) each method has or does not have.

A4. We agree that grouping the baselines according to their properties will enhance clarity. We categorize the methods into several groups based on their key properties as shown in the table below. We hope this categorization makes it easier to understand the key properties of each method and their relationships.

Table 2 Properties of Different Methods

Thank you very much for your insightful and constructive feedback on our submission. The following are our detailed responses.

Q1. The major contribution in this work is the combination of an exploration reward with FB learning, where the technical novelty is limited.

A1. As the reviewer pointed out, our method combines exploration rewards with FB learning. However, we would like to emphasize that this combination presents non-trivial technical challenges. Specifically, the value functions in FB [1] are inherently tied to implicit rewards for skill learning, and directly incorporating online exploration rewards could disrupt this skill learning capability. Our significant contributions include:

Scientific finding: We systematically demonstrate that insufficient exploration in FB leads to low data diversity, which in turn results in inaccurate successor measures and limits zero-shot generalization. Our key scientific finding is that the critical challenge of online FB lies in preserving efficient FB representation learning while simultaneously increasing online data diversity.

Technical novelty: To address this, we develop a novel DVFB framework that enhances exploration while maintaining effective skill learning through dual value functions and a novel intrinsic reward. Additionally, we introduce a dual-value fine-tuning scheme that achieves stable performance improvements, outperforming current SOTA methods.

To the best of our knowledge, DVFB is the first approach to successfully enable both zero-shot generalization and fine-tuning capabilities in online URL settings. We believe our insights into extending successor representation methods for online URL provide solid contributions to the field. We hope that our novelty and distinct contributions have been adequately justified.

---

Q2. Although the performance gain shown in Table 1 looks strong, I have concern on the baselines used in comparison for this setting. It is unclear why these are suitable baselines here for the problem of zero-shot online URL.

A2. Thanks for raising this important concern. To the best of our knowledge, no existing method achieves zero-shot generalization in online URL. We selected the baselines because they represent the closest existing approaches to our problem setting: zero-shot generalization in online URL.

USD methods (e.g., BeCL, CeSD) are designed for online URL and aim to learn transferable skills. While zero-shot generalization would be ideal, they typically rely on fine-tuning in downstream tasks due to limitations in mutual information-based skill learning [2]. Our results in Table 1 demonstrate that DVFB achieves zero-shot performance comparable to their fine-tuned performance (for more results, please refer to Table 2 in the attached PDF) and outperforms them in fine-tuning.

Table 1: Comparison of Zero-shot and Fine-tuned Performance Across Domains

Results marked with * are sourced from BeCL (ICML 23) and CeSD (ICML 24).

SR methods (e.g., LRA-SR, FB), originally designed for offline settings, represent the SOTA in offline zero-shot generalization. To ensure a fair comparison of zero-shot capabilities in online URL, we implemented their online versions based on the authors' official code.

Our experimental setup ensures a fair comparison by evaluating all methods under identical conditions (same interaction steps and fine-tuning steps), demonstrating DVFB's superior performance in both zero-shot and fine-tuning scenarios.

Dear Reviewer U97w,

Thank you for your insightful comments. We were wondering if our response and revision have resolved your concerns. We have attempted to address your initial questions through our replies and are eager to clarify any further points you might raise. Please feel free to provide additional feedback. We greatly appreciate your continued engagement.

Best regards, Authors

Dear Reviewer U97w,

Thank you for taking the time to review our manuscript and for providing valuable suggestions. We have further revised the main PDF to address your main concerns more clearly, with the changes highlighted in blue.

Below are the specific changes we made in response to your feedback:

1. We explain the contributions of DVFB and the scientific findings related to extending FB to online URL.
2. We discuss the limitations of the offline zero-shot method in Section 2 (RELATED WORK) and highlight DVFB’s advantages over SOTA offline zero-shot methods in Appendix G.
3. We provide a clear categorical description of the baselines in Appendix C.2 and offer guidance in Section 6 (EXPERIMENTS).
4. We present a detailed theoretical analysis of how the DVFB framework improves online FB’s zero-shot generalization in Appendix H, and provide guidance in Section 5 (METHODOLOGY).
5. We have updated the captions and explanations of the figures in Section 4.
6. We conduct additional experiments on the Point-Mass Maze and Meta-World benchmarks in Appendix G, along with detailed ablations in Appendix K (reward mapping technique) and Appendix L (sensitivity to hyperparameters) to further demonstrate the effectiveness of DVFB.

As the rebuttal period is closing, the authors would greatly appreciate it if the reviewer could consider our responses to the original review. We are more than delighted to have further discussions and improve our manuscript. If our responses have addressed your concerns, we would be grateful if you could kindly re-evaluate our work.

Best regards, Authors