We sincerely appreciate your comprehensive reviews and suggestions!

Weakness 1. The Nyquist sampling motivation is weak given the lack of a theoretical or empirical method to establish the connection to the theoretical signal optimization needs.

A1. We have added the theoretical analysis of the sample efficiency and effectiveness of our proposed time-aware world model. Due to the word limit, we would like to refer to our answer A1 to reviewer AknR above for the theorem, lemmas, and their high-level idea for theoretical analysis. We would be happy to provide the proof details if the reviewer requests it in the next rebuttal.

Weakness 2. A few points were not clearly addressed, including the benefit of using 4th-order Runge-Kutta instead of Euler and the lack of substantial differences between log-uniform and uniform sampling.

A2. Since this concern overlaps with the reviewer's Q1 and Q3, we would like to refer to our answers A5 and A6 in this rebuttal.

Suggestion 1. Additional references.

A3. We agree that AlphaGo and its subsequent works as well as DayDreamer are indeed relevant to this paper. We have included these works in our references and will be available in the final version.

Suggestion 2. This (Fig. 12) should be Fig. 2 (in the body).

A3. We have updated the main text accordingly to refer to both Figure 2 and Figure 12.

Q1. How much of the performance is due to the use of RK4 instead of Euler's method?

A6. Thank you for raising your concern regarding the choice integration method. We have included additional ablation studies on the RK4 vs Euler integration method in our anonymous website: https://sites.google.com/view/anonymous-site-rebuttal-6714. We also include additional experiments on PDE-Control tasks from the control-gym envs: arxiv.org/abs/2311.18736.

We emphasize that our proposed method’s central contribution is the adaptive time stepping, which is integration-agnostic and can be used with any integration, depending on the nature of task dynamics.

We employed the RK4 integration method because it is generalizable to both systems with simple, linear dynamics and highly complex, nonlinear dynamics and is a standard method for simulations of physical systems. For most robot manipulation tasks (Meta-World), the dynamics are sufficiently simple to be approximated with the Euler integration method. Empirically, our ablation shows that TAWM Euler performed better than RK4-based TAWM in most simple Meta-World tasks, suggesting the underlying dynamics of Meta-World tasks are sufficiently simple to be captured by the Euler integration method. The advantages of RK4 are more apparent in tasks with complex, non-linear dynamics, such as the PDE-Control tasks.

These results reinforce the merit of our core contribution of adaptive time stepping for training world models. The integration method and $\Delta t$-sampling method are two parameters we can adjust to maximize TAWM's performance and efficiency --both of which outperform the baseline.

Q2. What evidence is that log-uniform sampling is superior to uniform scaling? The claims ultimately state that any sampling can be used. If this is the claim being made that should be explicit at the start of the paper.

A7. Thank you for your comments! As explained in A6, the integration method and $\Delta t$-sampling method are two parameters that we can adjust to maximize TAWM's performance and efficiency. Therefore, uniform sampling and/or the Euler method can be better choices depending on the tasks. For example, uniform sampling performs better than log-uniform sampling in tasks with sufficiently slow dynamics to be captured by $\Delta t_{max}$. For tasks demanding fast inference time with sufficiently simple dynamics, Euler method is preferred.

We will be sure to make this point much more clear at the beginning of our paper in the final revision. Thank you for the suggesiton.

Q3. How fast is inference/planning in TAWM?

A8. We provide additional details of the inference time below. Inference time is averaged over 1000 planning steps for each model.

• Baseline: $\mu=$0.027 s; Q1 = 0.026 s; Q3 = 0.027 s
• Euler: $\mu=$0.028 s; Q1 = 0.028 s; Q3 = 0.028 s
• RK4: $\mu=$0.048 s; Q1 = 0.048 s; Q3 = 0.05 s

The tradeoff between Euler and RK4 is well-known in simulation: Euler is faster but can be unstable/less accurate than RK4 depending on the underlying dynamics. As mentioned in our answers A6 and A7, the integration method is one of the adjustable parameters. If underlying dynamics are sufficiently simple and slow to be approximated by the Euler method, the Euler method is preferred. Otherwise, RK4 is a more generalizable method.

We have incorporated your comments and new results into our paper. We thank you again for your comprehensive reviews and valuable suggestions, and we hope our answers sufficiently address your concerns and questions.

Please see additional experimental results here.

Q1. Gap between experimental design and motivation (data collection frequency matches the testing frequency in most manipulation tasks).

A1. We appreciate the reviewer’s concerns but believe this is a misunderstanding of our work. While data collection frequency can match testing frequency in laboratory settings, this is not always feasible or effective in practice.

Prior works have acknowledged such limitations [1]. Consider training a model at $\Delta t=$ 2.5 ms and deploying it on a physical robot with sensors operate at 100 fps ($\Delta t=$10 ms). One could either (1) use the trained model directly or (2) repeat the same action for 4 substeps to compensate for the mismatch. However, as Fig. 2 shows, TAWM outperforms these strategies across various frequencies, making it flexible for any $\Delta t$ encountered in practice w/o additional data or training.

One might argue that we could train the model at 10 ms in simulation to match the real-world frequency. Figure 4 shows that this solution is not robust when $\Delta t$ grows larger, as larger $\Delta t$ introduces instability due to missing important high-frequency dynamics (see Introduction). TAWM overcomes this issue by concurrently sampling multiple frequencies with better performance.

While approaches like DayDreamer [2] can train models for manipulation tasks at low sampling frequencies, their robot motions between time steps are very slow, as shown in their demo. Such slow motions prevent the loss of important dynamics and stabilize training, but result in slow robot actions in practice. In contrast, our TAWM can effectively learn to solve tasks at large $\Delta t$ without limiting the robot's motion speed, effectively learning both fast and slow task dynamics simultaneously.

We further conducted experiments on PDE-control environments, whose dynamics fundamentally differ from manipulation tasks (see A6 of Reviewer 3). Our results indicate that TAWM is effective beyond manipulation tasks and generalizes well across different classes of control problems.

Q2. Lack of novelty and unfair comparison

A2. We respectfully disagree with the reviewer’s comment that the comparison is unfair. Our training strategy—sampling multiple frequencies—is precisely the contribution of our work. The baseline TD-MPC2 model doesn't incorporate the temporal element, so it cannot be trained the same way. A comparison with TD-MPC2 is either (1) use it as-is or (2) manually apply substeps and repeat the same action (see A1). TAWM outperforms both baselines by a considerable margin in as fair a comparison as possible, given the fundamental difference in approaches.

Regarding novelty, one might view our primary modification as simply adding adaptive time intervals to the model input. However, we do not believe this should be dismissed as a lack of novelty for the following reasons:

• Simplicity with Clear Benefits: Often simpler approaches are preferable when they offer clear advantages. Even if the idea may appear 'simple' at first, our extensive experiments demonstrate that it yields substantial performance gains over the baseline without increasing model size or the required training samples. This simplicity also means that other world-model learning methods can readily adopt our approach.

• Integration of Physical Dynamics and Temporal Elements: Prior work has generally neglected explicit temporal modeling and adherence to basic physics principles in world models. Simply adding a time interval input, $z_{t+Δt}​=d(z_t​,a_t​,Δt)$, does not ensure compliance with these principles. In contrast, TAWM incorporates two key, model-agnostic components from physics simulation—time stepping and an integration method—that have been overlooked by previous methods. By integrating these components, we efficiently train our dynamics model by reducing the optimization space (e.g., the state remains unchanged when $\Delta t = 0$). Our main contribution lies in embedding physical dynamics into world models—a concept applicable across various architectural designs. We emphasize that our method is model-agnostic, and our contribution is in the novel consideration of BOTH temporal and physical dynamics in the world model -- a significantly new concept that none of the prior works has explored.

We also support our experimental results by theoretical proofs, as requested by the reviewers.

Q3. Lack of supplementary material.

A3. We already provided an appendix with additional experimental results across all environments. We refer this reviewer to the supplementary material and hope that it addresses any concerns regarding the breadth of our extensive evaluation.

[1] Thodoroff et al., "Benchmarking real-time reinforcement learning."

[2] Wu, Philipp, et al. "Daydreamer: World models for physical robot learning."

Please see additional experimental results here.

Q1. Lack of theoretical claims

A1. To corroborate our empirical results, we offer additional theoretical analysis on the sample efficiency of our proposed time-aware world model. Here we provide only a brief overview of the analysis in this rebuttal; the full analysis will be included in the final version of our paper (we will provide proofs if requested in the next rebuttal).

We start with following definitions:

• $z(t)$: Function that gives state vector of the world at time step $t$. We assume this function is a locally Lipschitz function with a constant $L_1$. That is, for a sufficiently small $\Delta t > 0$, we assume $| z(t + \Delta t) - z(t) | \le L_1 \cdot \Delta t.$ This is a mild assumption, as we are only concerned about physical environments, where every physical property (e.g. position, velocity) changes continuously. Also, note that $L_1$ is proportional to the frequency of the underlying dynamics -- if the environment is almost static, $L_1$ would be near zero.

• $a(t)$: Function that gives an action at time step $t$.

• $f(z(t), a(t), \Delta t)$: Ground truth dynamics function that we want to approximate. This function satisfies the following equation for any pair of $z(t), a(t),$ and $\Delta t$: $z(t + \Delta t) = z(t) + f(z(t), a(t), \Delta t) \cdot \Delta t$.

• $d(z(t), a(t), \Delta t)$: Dynamics function that we optimize to approximate $f$. We denote the predicted next state vector using this dynamics function as $\hat{z}(t + \Delta t) = z(t) + d(z(t), a(t), \Delta t) \cdot \Delta t$.

Then, the following Lemma 1 holds.

Lemma 1. For sufficiently small $0 < \Delta t_1 < \Delta t_2$, $| f(z(t), a(t), \Delta t_1) - f(z(t), a(t), \Delta t_2) \cdot \frac{\Delta t_2}{\Delta t_1} | < \frac{\Delta t_2 - \Delta t_1}{\Delta t_1} \cdot L_1$.

Note that this relationship holds for every training data in our buffer -- therefore, we assume our dynamics function $d$ captures this relationship easily during training.

Assumption 1. For sufficiently small $0 < \Delta t_1 < \Delta t_2$, $| d(z(t), a(t), \Delta t_1) - d(z(t), a(t), \Delta t_2) \cdot \frac{\Delta t_2}{\Delta t_1} | < \frac{\Delta t_2 - \Delta t_1}{\Delta t_1} \cdot L_2$.

We can expect that $L_1$ would converge to $L_2$ during training. Based on these relationships, we can prove the following lemma.

Lemma 2. For sufficiently small $0 < \Delta t_1 < \Delta t_2$, if $| f(z(t), a(t), \Delta t_2) - d(z(t), a(t), \Delta t_2) | = \epsilon$, we can compute the approximation error of the state vector as $| z(t + \Delta t_2) - \hat{z}(t + \Delta t_2) | = \epsilon \cdot \Delta t_2$. Then, for $\Delta t_1$, following holds: $| z(t + \Delta t_1) - \hat{z}(t + \Delta t_1) | \le \epsilon \cdot \Delta t_2 + (\Delta t_2 - \Delta t_1) \cdot (L_1 + L_2)$.

This lemma tells us that when we decrease the approximation error $\epsilon$, it not only reduces the state approximation error at $(t + \Delta t_2)$, but also that of $(t + \Delta t_1)$. That is, when we optimize our model for one time step, it transfers to another time step. Also, it is more effective for the systems whose dynamics has lower frequency, and thus lower $L_1$. Likewise, this lemma shows why our TAWM shows superior, or at least similar sample efficiency than the baseline, even though it has to learn additional temporal element.

Q2. Relationship to the video generative-model based world model (e.g. VideoAgent).

A2. Thank you for highlighting these works, which provide valuable insights that help connect our research with VLM Q&A. Both approaches share a common philosophy regarding the importance of temporal information. While VLM Q&A searches for the most relevant frames from pre-existing data to enhance video understanding, TAWM adaptively samples in time during the data generation process to capture the dynamics of underlying subsystems. In essence, VLM Q&A focuses on identifying key frames, whereas TAWM emphasizes aligning data generation with the system’s temporal dynamics. We will elaborate on this connection in the revised version.

Q3. Possible application of TAWM for reducing sim2real gap.

A3. Although we don't yet have results on TAWM's learning transferability, we conjecture that TAWM can reduce the sim2real gap by adaptively sampling the frequency space to better synchronize temporal effects between simulated and real-world dynamics. This adaptive approach could enhance the robustness of the learning process by mitigating unexpected temporal noise from capturing devices or environmental factors—issues that are often absent in simulated environments. By training our world model across a wide range of frequencies, we expect it to become more resilient against these disturbances and improve its ability to generalize from simulation to real-world applications.