Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach

We thank the reviewer for the insightful comments. Below, we address each comment point-by-point.

Q1. The authors use Dreamer V2 as their primary baseline. However, this is an outdated baseline and it would be great if the authors could provide comparisons with Dreamer V3.

Please refer to our response to Reviewer LSCr Q1.

Q2. It would be great if the authors could include details about how Dreamer V2, which is an online RL algorithm, is adapted to the offline setting.

The core adaptation of DreamerV2 to the offline setting lies in how the replay buffer is initialized and utilized. In the original online DreamerV2, the agent interacts with the environment to collect new trajectories incrementally. These trajectories are stored in a replay buffer, and the world model and policy are trained by sampling batches from this dynamically updated buffer. In the offline setting, we instead preload the entire static dataset into the buffer once at the start of training. No new data is collected during training, and all models are trained exclusively on this fixed dataset.

Q3. It would be great if the authors could include an ablation study of the effect of different amounts of unlabeled data on policy performance. It would be interesting to see if reducing the amount of data also reduces the policy performance.

As suggested, we have conducted a new ablation study to analyze how the amount of unlabeled data impacts policy performance. The results are summarized below. As the amount of source domain data increases, the model's performance improves accordingly, demonstrating that our method effectively leverages information from source domain videos and exhibits strong scalability. Full results across different datasets will be included in the revised paper.

We sincerely appreciate the reviewer's valuable comments and have addressed each point below.

Q1. The method is compared to DreamerV2 and not V3.

Please refer to our response to Reviewer LSCr Q1.

Q2. This paper doesn't cite FICC, which is another method that tackles a very similar problem.

Thank you for pointing out the FICC paper. We will add the citation and provide a detailed comparison in the revised manuscript. Although FICC and VeoRL both employ VQ-VAE and a forward model to extract latent actions in an unsupervised manner, they are distinct in the following aspects:

(1) We focus on offline RL.

While FICC focuses on pre-training a world model with offline data and then transferring it to downstream online tasks, our work addresses offline RL, where policies must be learned purely from static datasets without online interactions. These are fundamentally distinct problem settings with different technical challenges. For instance, offline RL primarily grapples with distributional shift due to the mismatch between the fixed dataset and real environment dynamics, while FICC employs online fine-tuning to achieve fast adaptation to downstream tasks.

(2) We handle significant domain gaps.

FICC operates under the assumption that the source domain (offline data) and target domain (downstream tasks) share the same environment. In contrast, our method tackles a more challenging cross-domain transfer scenario: extracting knowledge from real-world videos and employing it in a simulation environment. The significant visual and dynamics gaps between real-world and simulated domains substantially increase the difficulty of knowledge transfer.

(3) We leverage hybrid action spaces for behavioral alignment

FICC is explicitly designed for discrete actions (tested only on Atari) and acknowledges its limitation in handling continuous control tasks ("difficult to handle the continuous action space" as stated in their paper). Our method, however, focuses on the tasks with continuous actions. The dual-path policy optimization process we designed associates discrete actions extracted from videos with the continuous action space of the target environment, thereby enhancing offline RL policies with high-level behavioral abstractions (e.g., 'reach', 'push') from video data.

Q3. Experiments on more difficult, multi-step tasks.

As suggested by the reviewer, we compare our method with DreamerV2 and VIP in Soccer from Meta-World. This task typically involves a two-stage decision-making process where the agent must first fetch a ball and then push it to a goal position.

Q4. The natural videos this method leverages contain, for the most part, high-quality trajectories. This enables the planner to simply use a behavior cloning policy. This however is a quite limiting assumption.

(1) Do the auxiliary videos contain only high-quality trajectories?

No, the videos we leverage are not curated or task-specific. Instead, they are diverse, easily accessible, and collected without explicit task filtering. While these videos may not always align closely with the target tasks (meaning they should not be considered high-quality trajectories). Even when the tasks in the videos differ from the target tasks, our method can extract useful priors for policy learning.

(2) Does the method reduce to simple behavior cloning?

No, our approach goes beyond naive behavior cloning, as real action labels are unavailable in passive videos. Instead, we learn discrete latent actions that capture abstract behavioral patterns. This allows our method to generalize across environments, as it clones high-level behavioral concepts rather than relying on low-level action sequences. By leveraging these latent structures, our method can transfer useful priors even when direct action replication is impossible.

Q5. 247 right - considering more on -> did you mean "focusing more on"?

Yes.

We appreciate the reviewer’s insightful comments and provide our responses below.

Q1. Comparison with DreamerV3.

The decision to use DreamerV2 as the backbone stems from two key considerations:

• Offline setting performance: While DreamerV3 achieves strong performance across diverse tasks with fixed hyperparameters, our experiments (as shown below) demonstrate that it underperforms in offline RL settings without hyperparameter tuning compared to DreamerV2.
• Backbone consistency: LAMPO, a key baseline in our work, employs DreamerV2 as its backbone. To enable direct and equitable comparisons between our method and LAMPO, we maintained consistency by using the same backbone.

Notably, we have conducted additional experiments on Meta-world by integrating VeoRL with DreamerV3. The results show that our approach outperforms vanilla DreamerV3, showing its ability to seamlessly integrate with different model architectures.

Q2. Offline to online transfer.

In the offline-to-online transfer setup, we used DreamerV2 as the comparison model, as it has demonstrated stable performance across multiple environments. In this rebuttal, we conduct further experiments with the second-best comparison models in each environment (VIP, LOMPO, and VPT, respectively). The results presented below consistently demonstrate that our approach outperforms existing methods. The updated Fig 6 is available at https://anonymous.4open.science/r/image-765.

Q3. Value overestimation.

(1) Does VeoRL alleviate value overestimation?

Overestimation bias is indeed a key challenge in offline RL, and our method mitigates this indirectly through intrinsic rewards derived from auxiliary videos. To further validate its impact, we compare estimated values with true values (i.e., the discounted sum of actual returns) on the Meta-World Handle Press task.

• We analyze the distribution of estimated and true values using histograms. The results below show that VeoRL's predicted values align more closely with the true values, whereas DreamerV2 exhibits significant overestimation. Specifically, in 69.12% of states, DreamerV2 predicts values exceeding 1000, despite 57.71% of true values being below 200.

Notably,

• Since our model incorporates both environmental and intrinsic rewards in value estimation, we also include the intrinsic reward when computing the true value. The negative intrinsic reward leads to a lower overall highest true value in VeoRL. As a result, directly comparing the true values of the two models is not meaningful.
• Additionally, value computation follows each model’s successful trajectories.

Corresponding figures are available at https://anonymous.4open.science/r/image-765. Our approach is shown to produce value estimates that are closer to the true values compared to DreamerV2, indicating reduced overestimation bias.

(2) How does VeoRL specifically handle the value estimation bias?

VeoRL leverages intrinsic rewards to mitigate value bias in offline RL by grounding value estimates in realistic behaviors:

• Traditional value functions in offline RL can extrapolate poorly for unseen states because they optimize based only on limited environmental feedback.
• Intrinsic rewards encourage high-value states to align with realistic source behaviors. By incorporating intrinsic rewards from video priors, our approach provides an alternative supervision signal, reducing reliance on extrapolated Q-values.
• This approach acts as a regularization mechanism, constraining the model from assigning arbitrarily high values to unfamiliar states, preventing unrealistic value spikes.

We thank the reviewer for the constructive comments.

Q1. Complete code documentation and environment configuration files.

Code and full configuration files will be open-sourced upon acceptance.

Q2. Computational complexity.

Below, we present the training time and GPU usage until model convergence. While the latent behavior extraction and dual-branch architecture in VeoRL introduce additional computations, our experiments demonstrate a favorable efficiency-performance trade-off:

• Compared with DreamerV2: VeoRL requires 1.3× GPU memory and 2× training time. This modest cost results in about 360% improved episode return in CARLA and 140% higher success rates in MineDojo, as shown in Fig 1.
• Compared with APV, VIP, and LOMPO: VeoRL shows comparable training time and offers advantages in lower memory usage.

Q3. Interpretability of latent behaviors.

(1) Semantic interpretation of latent behaviors:

Latent behaviors align with real actions through our model, guided by the reward function and optimization objectives. As visualized in Fig 7, for example, in BridgeData, distinct latent behaviors correspond to atomic actions like "grasp", "reach", or "push".

(2) How do the latent behaviors guide policy optimization?

Latent behaviors are abstract and high-level, requiring alignment with low-level actions in the target task. To enable latent behaviors to guide target policies, we modify the MBRL algorithm as follows:

• The policy network conditions on both the current state and latent actions from the behavior cloning module, enabling high-level intentions to guide low-level actions.
• State transitions are modeled using a Trunk Net (driven by real low-level actions) and a Plan Net (guided by high-level latent behaviors) for long-horizon planning.
• An intrinsic reward encourages the rollouts from the Trunk Net to progressively align with long-term states predicted by the Plan Net, bridging the target policy with behaviors extracted from auxiliary videos.

(3) Generalizability across tasks:

We conduct additional experiments on Meta-World and MineDojo to validate the transferability of latent behaviors across tasks. The results are presented below.

Training schemes: The behavior abstraction network is frozen after training on Task A, with a fixed latent codebook. For Task B, this pre-trained network is directly deployed without undergoing task-specific fine-tuning on source video data.

The minimal performance degradation confirms that our method's latent behaviors trained on Task A remain effective for Task B, consistently outperforming other baselines trained specifically on Task B. This transferability significantly reduces training costs for downstream tasks, as it spares the time for re-training on the auxiliary videos.

Q4. More seeds.

As suggested, we have conducted additional experiments on MineDojo using five different random seeds. The updated results below show consistent performance of our model. We'll include full results in the revised paper.

Q5. The distribution differences between video data and the target tasks.

We summarize the differences between the source and target domains below. These distinctions highlight VeoRL’s ability to effectively leverage real-world video data, even with notable distributional discrepancies, to enhance target domain task performance.