RLVR-World: Training World Models with Reinforcement Learning

We would like to sincerely appreciate Reviewer d8tC for the comprehensive review and insightful questions.

Q1: Clarification on Our Insight and Contribution

We would like to clarify that our central paradigm contribution is not simply performing task-specific fine-tuning after pre-training, but rather revealing RLVR as a more task-aligned fine-tuning paradigm for world models, compared to conventional supervised fine-tuning (referred to as maximum likelihood estimation, or MLE, in our paper). In other words, our key insight addresses how fine-tuning should be done, rather than whether fine-tuning should be done at all.

As we discussed in the Introduction, advanced world models, whether autoregressive or diffusion-based, often employ non-end-to-end architecture and surrogate training objectives. While such approaches are scalable and well-suited for pre-training, we point out that continuing to use the same training methods for task-specific fine-tuning is not effective enough. This is because these surrogate training methods misalign with the actual task objectives of world models. Our insight is that switching to a more aligned training paradigm, RLVR, can lead to significantly faster and more effective post-training. For example, as shown in Figure 3, training speed of RLVR can be more than 1000x faster. Such results are not already well-known, sufficiently surprising, and worth sharing with the community.

We acknowledge that the current form of Figure 1 may have caused confusion regarding our paradigm contribution. We will revise it in the next version to more clearly emphasize the differences between supervised and RLVR fine-tuning.

Q2: Enhancing General-Purpose Capability

The mechanism by which RLVR improves general capabilities in LLMs such as DeepSeek-R1 remains an open and active research question. A common belief is that the base model must be sufficiently strong for task-specific improvements (e.g., in math or code) to transfer to broader domains. Unfortunately, training LLMs with hundreds of billions of parameters exceeds our available computational resources. Moreover, at present, no video world model exists with similarly strong general capabilities as LLMs. For this reason, we focus our experiments on a specific scenario (RT-1). Nevertheless, we provide additional experiments below using a "general" video world model across two relevant domains.

Preliminary experiments: We adopt two datasets, Rope and Granular, from DINO-WM [1], a related work suggested by Reviewer BbTo. Both are collected from the same simulator, PyFlex, but involve different tasks. A shared base model is jointly trained on the two datasets, and then fine-tuned by RLVR on each dataset individually, to investigate whether improvements in one domain can benefit the other.

As shown in the table above, we observe that when the model is RL-trained exclusively on one dataset, performance on the other can improve slightly. This result aligns with discoveries in the LLM field. However, we also find that performance on the held-out dataset later degrades due to overfitting. We attribute this to the limitations of the base model, which is not pre-trained on large-scale, diverse data and thus lacks truly generalizable representations. We hope these preliminary results can encourage the community to further explore RLVR in the context of large-scale world models.

Q3: Discussion on Policy Evaluation Metrics

The goal of real2sim policy evaluation is to identify effective policies using only rollouts from simulated environments. In this setting, our focus is not on reducing the "sim-to-real gap", a topic already addressed in detail in our prediction experiments (Sections 6.2 and 6.3). Instead, we aim to assess whether the model can accurately estimate the success rate of a given policy. In this context, a smaller measured discrepancy is sufficient to demonstrate the model’s effectiveness.

This formulation aligns with the standard setup in most existing policy evaluation studies. For example, [2] introduced a benchmark using metrics like rank correlation to compare policy rankings derived from success rates in simulators and the real world. Similarly, [3] and [4] employed metrics like Pearson correlation and regret@1, both of which are computed over aggregated success rates rather than individual trajectories. SIMPLER [5], a collection of simulated environments on common real robot setups, also takes its accurately estimated success rates as evidence to demonstrate that simulation-based evaluation can be a scalable, reproducible, and reliable proxy for real-world evaluation.

[1] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. ICML 2025.

[2] Benchmarks for Deep Off-Policy Evaluation. ICLR 2021.

[3] Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization. ICLR 2021.

[4] Variational Latent Branching Model for Off-Policy Evaluation. ICLR 2023.

[5] Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL, 2024.

Dear Reviewer d8tC,

We sincerely appreciate your thoughtful response and engagement in such a meaningful discussion. We would like to offer a detailed clarification regarding the comparison you requested.

First of all, we apologize for any confusion caused by our inaccurate wording. When we use the term "task-aligned," we do not mean alignment to a specific domain of data (e.g., pre-training data vs. downstream data), but rather alignment to a specific objective (e.g., conventional cross-entropy vs. task objectives like LPIPS) within the same domain.

We can now discuss the advantages of RLVR over SFT, both principally and empirically.

Principally:

• As the reviewer correctly noted, many task objectives (e.g., F1 score) are non-differentiable and thus cannot be optimized directly. SFT can only optimize the cross-entropy loss of the desired outputs, serving as a surrogate training objective.
• Even for differentiable task objectives (e.g,. MSE or LPIPS), direct optimization is often infeasible due to non-end-to-end architectures commonly used in modern advanced world models. We can only rely on surrogate optimization, too. For example, our transformer-based video world model first tokenizes ground-truth frames and then learns via next-token prediction. For a more in-depth discussion in this aspect, please see our response for Reviewer NuUF (Q1: Can We Directly Optimize Reward Objectives?).
• In contrast, RLVR overcomes limitations by either non-differentiable objectives or architectures. It directly optimizes desired task objectives via policy gradients. This is what we refer to as being more task-aligned.
• A more precise statement of our contribution might be identifying an effective method (RLVR) to enable such direct optimization of task objectives in world models where this is previously infeasible. This contribution has been well recognized by Reviewer NuUF, who subsequently raised the score to 5 ("accept").

Empirically:

• In the context of our discussion, the comparison of SFT and RLVR then becomes a comparison between surrogate and direct optimizations, using the same downstream data. Across all experiments of our paper, this has been done by comparing how much performance gain can be achieved from the same base checkpoint using either SFT or RLVR.
• In the language world model experiments, SFT indeed is applied as a pre-RL step to get a checkpoint with basic task performance. However, let's consider a thought experiment: if we continue SFT on this checkpoint, task performance would stagnate or even overfit (since we already apply early stopping to get the SFT checkpoint in our experiments). However, by switching to RLVR, our experiments show that we can achieve further performance gains. We believe this is already a controlled comparison, which can illustrate the superiority of RLVR over SFT in "aligning models with task objectives."
• In the video world model experiments, the same comparison is also explicitly presented in the original submission. We pre-train our video world model (solely) on the RT-1 dataset with $10^5\sim 10^6$ steps to obtain a base checkpoint (with LPIPS 14.8). Continuing to train on the same dataset with the same standard cross-entropy loss (which is analogous to SFT) yields only slow improvements: As mentioned in Line 235, 150k additional steps can only achieve an LPIPS 14.5. In contrast, switching to RLVR from the base checkpoint (see Figure 3) quickly achieves a significantly better LPIPS 13.4, with just 100 gradient steps (a more than 1000x speedup). This comparison from the same base checkpoint also sufficiently "demonstrates either superior performance or higher sample efficiency of RLVR."
  • [!!! Important] We would like to respectfully correct a factual misunderstanding in the reviewer's comment. Since the community currently does not have a strong enough general-purpose world model, we restrict our entire training pipeline (including pre-training, continual pre-training, and RLVR) solely on the RT-1 dataset. We sincerely apologize for not explicitly stating this in the paper (only Lines 564 and 599 in the appendix contain vague descriptions). Thus, we did NOT compare "pretraining on multiple datasets (SFT) versus fine-tuning on a single task (RLVR)."

We will carefully revise our paper to more clearly present the above comparison. We hope our detailed clarification helps address your concerns and strengthens your evaluation of the value of our work.

Best,

Authors.

We sincerely thank Reviewer NuUF for the thoughtful and insightful comments. We hope our response addresses any concerns or misunderstandings and more clearly conveys the value of our work.

Q1: Can We Directly Optimize Reward Objectives?

We respectfully suppose that there might be a slight misunderstanding regarding the motivation of our work, which we clarify below:

1. Advanced world models (including ours) typically cannot directly optimize task objectives via gradients, due to their non-end-to-end architectures, as discussed in the Introduction.
   • Our autoregressive world models consist of a discrete visual tokenizer and a GPT-style transformer. The standard training pipeline involves encoding ground-truth frames into discrete tokens using a pre-trained tokenizer and training the transformer for next-token prediction via a cross-entropy loss. It is not feasible to directly optimize objectives such as pixel MSE or LPIPS by gradients, because the intermediate steps between transformer and decoder, i.e., sampling predicted tokens and retrieving their corresponding visual embeddings from the tokenizer codebook, are not differentiable.
   • Another popular class of diffusion-based world models also face similar problems: they are trained by step-wise denoising objectives, rather than end-to-end target objectives.
2. These surrogate training paradigms make task-specific prediction objectives in video world models well analogous to rule-based reward functions in LLMs, just as an LLM supervised trained to mimic the chain-of-thought reasoning for math problems does not directly optimize for answer correctness.
3. With this insight, our contribution is not to compete with or enhance direct objective gradient optimization, but to enable direct optimization of task objectives in complicated world model architectures where this is previously infeasible, by leveraging RL. This is why we believe the comparison with direct gradient optimization, as suggested by the reviewer, is not necessary (and actually not feasible) in our context.
4. Clarification on metric-oriented optimization (Figure 4): We would like to clarify that in Figure 4, all metrics are optimized using RLVR (instead of end-to-end gradients which is infeasible here), same as our main results. The purpose of this experiment is to show that RLVR can indeed target optimizing desired objectives, similar in spirit to direct gradient optimization in end-to-end architectures. The difference in test LPIPS between training solely for LPIPS (0.124 in Figure 4) and our main result (0.122 in Figure 3, optimized jointly for MAE and LPIPS) is not statistically significant and is expected, given that both setups emphasize LPIPS.

Q2: Non-Differentiable Rewards

We indeed agree with the reviewer that performing experiments on non-differentiable rewards, which can not even be optimized by gradients in an end-to-end architecture, will strengthen our experiments, which we conduct as follows.

Repetition penalty reward (RPR): In Section 6.3, we show that RLVR can reduce repetition artifacts in multi-step video prediction (for the formal definition of repetition, see Q4). However, the repetition rate remains non-zero. To address this, we introduce an additional reward term, repetition penalty reward (RPR), defined as the negative rate of consecutive identical frames. We note that this term is non-differentiable. After training the model with this extended reward using RL, we observe that it maintains comparable prediction performance while effectively eliminating repetition artifacts.

On physical law: We have discussed the use of physical laws as reward functions in the Discussion section. Unfortunately, we currently lack a comprehensive method to detect whether generated videos adhere to physical laws—a challenging problem that still requires further advancements from the research community.

Q3: Recent World Model UVA

We greatly appreciate the reviewer bringing up the advanced world model, UVA. While the core insight behind RLVR-World is general and not tied to a specific model class, we note that UVA is implemented as a diffusion model, and its associated GRPO algorithm [2] was developed concurrently with our submission. As such, it falls outside the scope of our current work. We are excited to explore this and will add discussions on applying RLVR for diffusion world models in the next revision of our paper.

Q4: Repetition Definition

As our model uses a visual tokenizer to encode pixels into discrete tokens, we define a case of "repetition" when the predicted tokens of the last two frames have exactly the same token IDs, which are also decoded into identical pixel values. Empirically, we find that when the last two predicted frames are the same, it often indicates repetition across many consecutive frames, as showcased in Figure 6.

[1] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. ICML 2025.

[2] DanceGRPO: Unleashing GRPO on Visual Generation. arXiv 2025.05.

We sincerely thank Reviewer ndrJ for the thorough review and valuable questions. We hope our following response can resolve your concerns and strengthen your confidence in the value of our work.

Q1: Comparison with Advanced Video World Models

First of all, we would like to clarify that our work does not aim to introduce a new state-of-the-art world model. Instead, we reveal a new training paradigm (RLVR) that can improve existing world models more effectively than traditional training approaches. This paradigm, along with the underlying insight, is general and applicable to a wide range of models. For demonstration, our paper applies it to the recent state-of-the-art world model, iVideoGPT (NeurIPS 2024).

Nevertheless, in response to Reviewer BbTo's suggestion, we have added comparisons with a number of advanced video world models across three new datasets and two tasks: next-frame prediction and model predictive control.

Please refer to our response to Q2 of Reviewer BbTo for full experimental details and extended results. Key results are presented as follows.

Next-Frame Prediction

We have conducted experiments on three additional datasets (PushT, Rope, Granular), comparing against advanced recurrent latent space models, including DINO-WM (ICML 2025) [1], as well as an action-conditioned diffusion model, AVDC (ICLR 2024) [2]. The results are presented below.

Overall, our model, after applying RLVR, achieves performance comparable to the strongest baseline, DINO-WM, and significantly outperforms all other baselines. Notably, it surpasses DINO-WM by a substantial margin on the most challenging particle-based dataset, Granular.

Model Predictive Control (MPC)

We also conduct MPC experiments, comparing with state-of-the-art world models, including DreamerV3 (Nature 2025), TD-MPC2 (ICLR 2024), IRIS (ICLR 2023), and DINO-WM.

The results show that RLVR also improves the control performance of our base model, achieving the strongest results and outperforming popular world model methods such as DreamerV3 and TD-MPC2.

Q2: Discussion on Text Game State Prediction Accuracy

For the accuracy difference in unchanged and changed cases:

1. The difference in accuracy and RL gains stems from differences in task difficulty. Changed cases are inherently more challenging than Unchanged ones: for Unchanged cases, the model only needs to correctly predict that no objects have changed; for Changed cases, the model must not only identify which objects will change but also accurately predict the outcomes of those changes. Therefore, Changed cases are harder to answer, and improvements are more difficult to achieve.
2. Improvement in both cases indicates the model can handle opposing tasks. Unchanged cases favor more conservative prediction ("no changes" is sufficient), while Changed cases require more aggressive predictions to capture object changes. Our approach avoids collapsing into a single scenario, and instead can accommodate both types of tasks and truly enhance the learned relationship between actions and changes.

In summary, these results show that RLVR is applicable to complex and diverse world model settings, improving performance across multiple task types, with more substantial gains observed in the simpler task.

[1] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. ICML 2025.

[2] Learning to Act from Actionless Videos through Dense Correspondences. ICLR 2024.

Dear ndrJ, we need you to show up and engage in the discussion phase.

The guideline is:

• read the author rebuttal
• engage in discussions (reviewers must talk to authors, and optionally to other reviewers and AC - ask questions, listen to answers, and respond to authors)
• fill in "Final Justification" text box and update “Rating” accordingly (this can be done upon convergence - reviewer must communicate with authors first)

The (new) deadline is Aug 8, 11.59pm AoE.

Thank you to the authors for the detailed and thorough rebuttal. The new experiments comparing RLVR-World with strong recent baselines have fully addressed my primary concern regarding the lack of comparative evaluation. And your explanation for the accuracy difference in the text game state prediction task is clear. I'm rasing my score accordingly.

We sincerely thank Reviewer BbTo for the valuable and comprehensive feedback. We have made our best effort to provide additional clarifications and experimental results to address your concerns.

Q1: Novelty and Insights

We respectfully disagree with the comment that our work lacks significant insight or advancement in this field, and we apologize for not articulating our contributions clearly enough.

As illustrated in Figure 1 and the Introduction, world models are becoming increasingly capable but also more complicated, often involving non-end-to-end architectures and training objectives. To our knowledge, our work is the first to identify the following key insight:

As world models are built with more advanced models, their training methods diverge further from the actual task objective of world modeling.

This fundamental observation has not been discovered or addressed in prior work on autoregressive world models [1,2] or diffusion-based world models.

This necessitates new and more aligned training methods, especially after large-scale pre-training with traditional surrogate objectives. In this context, identifying RLVR as a direct and effective training method to meet the above insight, is a novel and meaningful contribution. Fully leveraging this insight leads to substantial improvements: as shown in Figure 3, the gains achieved in just a few hundred RL steps cannot even be achieved by 1000x more steps of standard next-token prediction training. We also consider this first successful application of RLVR to video world models as a minor but still novel technical contribution.

Although our work may not introduce entirely new techniques, we believe this paradigm-shifting insight is both valuable and impactful. We are encouraged that other reviewers recognized this contribution—for instance, Reviewer ndrj described the idea as "novel and a significant departure from traditional training objectives," while Reviewer d8tC noted that we "broaden the applicability" of RLVR.

Q2: Additional Datasets & Baselines

To provide a more comprehensive evaluation, we include additional datasets and advanced baselines, following the reviewer's suggestion.

Datasets: We adopt three datasets (Push-T, Rope, and Granular), from DINO-WM [3]. We strictly follow the evaluation protocol described in the original paper and official code, including image resolution, training-validation split, number of history frames, and frame skip.

Baselines: We compare our models against all recurrent latent state models provided in [3], which use different latent spaces from pretrained encoders such as R3M, ResNet, and DINO. We also include an action-conditioned diffusion model, AVDC.

Results: The learning curves of our models on three datasets are shown below (Due to NeurIPS policy, we cannot provide figures in rebuttal).

Across all datasets, RLVR consistently improves upon our base model. Notably, when training with next-token prediction on smaller datasets (Rope and Granular, each containing only 1,000 episodes of length 20), the models are prone to overfitting and require early stopping. In contrast, RLVR continues to significantly enhance prediction performance on the validation set, demonstrating greater robustness and generalization.

We then compare the prediction capabilities of different world models. Since DINO-WM only releases model checkpoints for the Push-T dataset among the three, we additionally include our own evaluation results using the publicly available Push-T checkpoint.

Our model after RLVR achieves overall performance comparable to the strongest baseline, DINO-WM, and outperforms it by a significant margin on the most challenging particle dataset, Granular.

Our model underperforms DINO-WM on the Rope dataset. We hypothesize that this is due to our model being prone to overfitting on this relatively small dataset. In contrast, recurrent latent state models that leverage large-scale pretrained visual encoders demonstrate greater robustness. To validate this hypothesis, we explore training a base model jointly on the Rope and Granular datasets (both collected from PyFlex simulators), followed by fine-tuning with RLVR on each individual dataset:

We observe that the LPIPS on Rope improves from 0.0208 to 0.0165. Looking forward, we believe the potential of RLVR can be further unlocked when applied to large-scale pre-trained world models, mirroring the remarkable success of LLMs.

Q3: Scaling Language Models

We appreciate the reviewers' rigor and agree that, in the modern era, the effectiveness of LLM algorithms should ultimately be validated in truly large-scale settings. At the same time, we believe that scientific progress is inherently incremental. The foundational algorithm behind RLVR, GRPO, was initially validated on a relatively small-scale language model, DeepSeekMath 7B. We believe our current results remain valuable and can inspire further exploration within the community.

Nonetheless, we have made our best effort to scale our text game state prediction experiments to 7B-parameter language models. Specifically, we use DeepSeek-R1-Distill-Qwen-7B as our base model.

The results demonstrate that the effectiveness of RLVR-World scales to 7B models, with our final overall accuracy matching that of GPT-4. We hope these new results further strengthen your confidence in the scalability of our method.

Q4: Model Predictive Control

We first note that our original submission already includes experiments on policy evaluation, a critical subtask of model predictive control (MPC). Conducted in the realistic RT-1 domain, this experiment is both challenging and well-suited to demonstrate the benefits of our method for robotic manipulation.

Due to time constraints, it is not feasible for us to set up real robot control experiments. Instead, we conduct additional MPC experiments in the simulated Push-T environment, following the setup used in DINO-WM.

Setup: We strictly follow the same planner configuration as DINO-WM, including goal construction, number of samples, and optimization steps. Since our model predicts discrete tokens and raw pixels without a compact latent space, we use the publicly available DINOv2 encoder to embed predicted frames and compare them to the goal observation during planning. We find that the DINO latent space is more effective for planning than pixel-level MSE or LPIPS, consistent with the findings in DINO-WM. We emphasize that our MPC experiments are intended to compare world models, and the choice of distance metric is orthogonal to our main contributions.

We do not conduct experiments on Rope and Granular due to the lack of MPC configurations and details in the official DINO-WM code, making it difficult to replicate the setup.

Results: The results below show that RLVR also enhances the control performance of our base model, achieving results comparable to DINO-WM.

In Summary

We sincerely thank the reviewer once again, especially for pointing us to the relevant work, DINO-WM. The additional experiments on three new datasets and two task setups, with comparisons against several advanced world models, further strengthen our contributions. We will include all experimental settings, quantitative results, and visualizations of both prediction and planning in the next version of our paper.

[1] Transformers are Sample-Efficient World Models. ICLR 2023.

[2] iVideoGPT: Interactive VideoGPTs are Scalable World Models. NeurIPS 2024.

[3] DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. ICML 2025.

Dear BbTo, we need you to show up and engage in the discussion phase.

The guideline is:

• read the author rebuttal
• engage in discussions (reviewers must talk to authors, and optionally to other reviewers and AC - ask questions, listen to answers, and respond to authors)
• fill in "Final Justification" text box and update “Rating” accordingly (this can be done upon convergence - reviewer must communicate with authors first)

The deadline is Aug 8, 11.59pm AoE.