Model-Based Transfer RL with Task-Agnostic Offline Pretraining

1. In section 4.4, you claimed that the feature of $G_\theta$ before output is more informative than the output of $G_\theta$, but why? And what do you mean by saying that "more informative"?
2. How do you verify that the learned weights indeed help select the most similar task?
3. The generative action guidance is trained on expert data from each source task, but tested on the training data from target task, will this distribution shift hurt?

1. Questions about task similarity weights calculation

(a) For calculating the task similarity score, why having a distillation layer $f_\text{distill}$, why cannot we just directly take $\|e_t - s_t^i\|$? The role of $f_\text{distill}$ and $f_\text{weight}$ is very unclear even though the authors are trying to explain in the text.

(b) Why not a simpler approach such as [7]? We could calculate the task similarity by simply looking at the similarity of their gradients. This should serve as a baseline to substitute the proposed architecture here if the authors want to claim that learning $f_\text{distill}$ and $f_\text{weight}$ gives a better solution.

2. Clarification of Terms

What is $G_\theta(e_t, k)$ in equation (8)? The author only mentions that $G_\theta$ consists of the encoder $E_{\theta_1}$ and decoder $D_{\theta_2}$, and what $E_{\theta_1}$ and $D_{\theta_2}$ are in (6) and (7).

3. Handling Different Action Spaces

It is not clear to me how the proposed method handles different action space dimensionalities in DMC tasks.

4. Task Selection

• (MetaWorld): The six tasks from Metaworld are all too simple. The authors should consider more challenging tasks such as Hand Insert, Stick Pull, Stick Push, Disassembly, Assembly, Shelf Place, and so on. Please refer to [1] for a rough categorization of the difficulty of tasks in MetaWorld.
• (DMC): Only two tasks during downstream finetuning are considered (Quadruped Walk and Quadruped Run). More medium-difficulty tasks in DMC should be considered. Please refer to [3] for a list of medium-difficulty tasks. Additionally, a few harder humanoid tasks would be good to demonstrate the effectiveness of the proposed pretraining approach.

5. Number of Environment Steps

For DMC, 400K environment timesteps are far too few to draw a conclusion. With only 400K steps, the performance of all methods on Quadruped Run and Quadruped Walk is poor (<350 compared to 800-900 of the convergent performance). For Metaworld, the authors use 200K, but it is mostly that the selected tasks are too easy.

6. Baselines

(a) [1] and [8] are both model-based approaches for visual RL that should serve as baselines. Maybe [1] is more relevant since it’s dreamer-based, but at least [8] should be mentioned in the paper. (b) Dreamer-v3 [2]: the author should compare with the latest dreamer model. (c) Model-free approaches [3,4,5]: model-free visual RL algorithms should also be good baselines. ([3,4,5] are missing from the related works of the paper.) If after pretraining latent dynamics from multitask offline datasets, it still does not perform as well as the model-free visual RL approach from scratch, then the pretraining would be of no value.

7. More Random seeds

(Minor point): More random seeds are needed for the empirical evaluation. Right now Vid2Act only runs 3 random seeds.

8. Missing Related Works

[6] is a latent model-based transfer RL approach that should also be discussed in the related work section.

References

• [1] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, Pieter Abbeel, "Masked World Models for Visual Control," CoRL 2022.
• [2] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap, "Mastering Diverse Domains through World Models," Arxiv Preprint.
• [3] Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto, "Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning," ICLR 2021.
• [4] Edoardo Cetin, Philip J. Ball, Steve Roberts, Oya Celiktutan, "Stabilizing Off-Policy Deep Reinforcement Learning from Pixels," ICML 2022.
• [5] Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, Furong Huang, "TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning," NeurIPS 2023.
• [6] Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew E Cohen, Furong Huang, "Transfer RL across Observation Feature Spaces via Model-Based Regularization," ICLR 2022.
• [7] Yifan Xu, Nicklas Hansen, Zirui Wang, Yung-Chieh Chan, Hao Su, Zhuowen Tu, "On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning," ICLR 2023.
• [8] Nicklas A Hansen, Hao Su, Xiaolong Wang, "Temporal Difference Learning for Model Predictive Control," ICML 2022.

• What's stopping the task-selective weight to be learned such that $w_i = 1, i=k \quad \text{and} \quad w_i = 0, i != k$ for some single $k$, where $k$ might be the offline model which is closest to the current online task. I.e The current task would only care about one of the offline models.
• Can the method also be worked with proprioceptive observations?

Addressing my comments listed in the weaknesses section is most likely to change my opinion. However, I'd also like the authors to clarify the following:

• In the case of the DMControl experiments, the authors only consider target tasks using the Quadruped embodiment which is not included in the offline dataset. Why did the authors choose these particular tasks for transfer? What would the results look like for a transfer setting in which the target task is a seen embodiment but unseen task? Intuitively, I would expect transfer to be most effective in this setting, since the dynamics are exactly the same -- only the task (reward function) is new. The Meta-World experiments don't quite cover this setting since objects and thus dynamics differ even though it is the same robot.
• Code is provided as supplementary material. Are the authors committed to open-sourcing their code in case of acceptance?