WorldMem: Long-term Consistent World Simulation with Memory

We thank the reviewer for the thoughtful and constructive feedback. We appreciate the detailed analysis and helpful suggestions on memory design, downstream applications, and generalization. We will revise the paper accordingly.

---

W1: The memory bank selector is hard-coded

We agree that our current FoV-overlap-based memory retrieval is heuristic and may not generalize well to complex 3D environments (e.g., multi-level buildings). This limitation is acknowledged in the paper (Line 2-5 in Supp.). Nevertheless, it serves as a proof-of-concept showing that structured spatial priors can support long-term consistent world modeling. Despite its simplicity, the approach works well in controlled settings such as Minecraft and indoor scans.

---

W2: Limited applicability to real-world scenes

Our current experiments focus on synthetic and small-scale real scenes, mainly due to resource constraints. However, our framework is conceptually applicable to real-world scenarios as long as sufficient sensory input and temporal alignment are available. While relying on known camera poses may seem restrictive, such information is available in many practical settings, such as gaming, robotics, and embodied scanning, via built-in APIs or SLAM systems. As pose estimation tools (e.g., Vggt [1]) continue to improve, this constraint becomes increasingly manageable. We also plan to explore retrieval based on learned visual representations to further reduce dependence on explicit pose information.

---

W3: No illustration of downstream decision-making tasks

Thank you for the suggestion. Although our current work focuses on generating quality and consistency, our model is naturally compatible with downstream decision-making frameworks such as Model Predictive Control (MPC). By simulating temporally grounded future observations, our model can be used to evaluate outcomes of candidate action sequences and support action selection. For example, in Minecraft, this allows integration into planning and control pipelines. While we have not demonstrated this in the current work, we consider it an important direction for future research.

---

W4: Missing related work in interactive world simulation

We appreciate the reviewer highlighting this. We will extend our related work section to include recent contributions on interactive and long-context video world models, including but not restricted to [2,3,4,5].

---

W5: Incomplete 3D consistency in generation

We acknowledge that some generation results, especially in long sequences, may not achieve perfect 3D consistency (e.g., terrain deformation). This remains a general challenge in world modeling. Nonetheless, to the best of our knowledge, our method achieves state-of-the-art consistency compared to existing methods, including very recent approaches [6,7]. We are actively exploring ways to further improve consistency.

---

Q1: Modeling dynamic scene changes

This is an important and challenging direction. Our current experiments do not include moving entities (e.g., zombies) primarily due to the computational cost to train a video model that can generate such dynamics with high quality. However, our model architecture is compatible with such dynamics. Since each memory entry is associated with a timestamp, the model can learn temporal evolution, e.g., understanding that a walking zombie at time $t_i$ should appear in a different location at time $t_{i+n}$. This is a key motivation behind our design of state-aware memory, which we plan to explore further with dynamic scenes in future work.

---

Q2: Integration of history guidance

Yes, history guidance [8] techniques are complementary to our method and can improve temporal coherence and visual fidelity. In our RealEstate10K experiments, we have already applied history guidance to stabilize generation (Table 4 in the main paper). These techniques are orthogonal and can be integrated seamlessly to further enhance results.

---

Q3: Generalization beyond pose-based retrieval

Our method leverages the broader concept of state-aware memory, with pose and timestep serving as a specific instantiation of the state. In scenarios where explicit poses are unavailable (e.g., egocentric wrist cameras), the same principle applies: define the state (e.g., the position and the rotation angle of robotic arms, past visual input), and learn associations between current states and stored memory. With sufficient training data, we believe a learned retrieval function can effectively generalize to such unstructured environments.

---

Limitation: Retrieval latency vs. direct generation

Thank you for highlighting this limitation. While retrieval does introduce some latency, it remains minor even at scale. Below is the empirical timing on an H100 GPU for retrieving 8 memory frames from different memory pool sizes:

The cost of generation (20 denoising steps) is ~0.9 seconds per frame (excluding retrieval). Thus, retrieval contributes only ~10–20% of the inference time, even when accessing 1000 candidates.

We believe this is a worthwhile trade-off, as retrieval enhances long-term coherence by anchoring generation in relevant context, which pure autoregressive or diffusion models struggle to maintain. We will move this limitation and analysis from the supplementary to the main paper as suggested.

---

We hope our responses have addressed your concerns. If any points remain unclear, we would be happy to further clarify and discuss.

---

[1] Wang J, Chen M, Karaev N, et al. Vggt: Visual geometry grounded transformer.

[2] Yang M, Du Y, Ghasemipour K, et al. Learning interactive real-world simulators.

[3] Yu J, Qin Y, Che H, et al. A survey of interactive generative video.

[4] Beattie C, Leibo J Z, Teplyashin D, et al. Deepmind lab.

[5] Hafner D, Pasukonis J, Ba J, et al. Mastering diverse domains through world models.

[6] Chen T, Hu X, Ding Z, et al. Learning World Models for Interactive Video Generation.

[7] Po R, Nitzan Y, Zhang R, et al. Long-context state-space video world models.

[8] Song K, Chen B, Simchowitz M, et al. History-guided video diffusion

We sincerely thank the reviewer for the valuable suggestions and sharp observations. Your comments on long-horizon generation, robustness of memory conditioning, and architectural clarity are greatly appreciated. We will revise the paper to include more details.

---

W1: Limited generation length

To further assess long-range consistency, we provide the following PSNR results for longer sequences:

These results indicate that our method maintains strong reconstruction quality up to 500 frames, and although quality degrades at 1000 frames, the generation does not collapse and remains structurally reasonable. This demonstrates that our method is capable of handling relatively long sequences, beyond the typical short-range settings explored in prior work.

---

Q1: On the robustness and design of memory cross-attention

Thank you for the thoughtful question. Our use of cross-attention to incorporate memory is motivated by a balance between robustness and efficiency.

• Robustness to noisy memory: During training, memory is sampled randomly from large video pools, which often include unrelated frames. The model thus learns to selectively attend to informative content while ignoring irrelevant memory. This naturally increases robustness to retrieval imperfections.

Using cross-attention for memory conditioning is not strictly necessary. One could alternatively apply full self-attention over both the current frame and the memory. However, we adopt cross-attention for the following reasons:

• It introduces a clear directional inductive bias, where the current denoising frame attends to the memory without allowing information to flow back.
• It significantly reduces computational and memory cost, particularly when the context or memory length is large.

Thus, the training and inference would both be more effective and efficient.

That said, cross-attention is less standard than full self-attention in typical Transformer architectures. Full self-attention offers greater capacity and flexibility for learning rich interactions between memory and context, which might be suitable for large-scale training.

---

Q2: Clarification on Figure 2(d)

We appreciate your comments. In the "State-aware Memory Attention" module, we use all tokens, including clean frames from the context window, clean frames from the memory bank, and noisy frames, as queries. The clean frames within the memory bank are used as keys and values in the cross-attention operation. We will revise Figure 2(d) to highlight the query-key-value assignments more clearly, including visual markers for which tokens are used in each role.

---

We hope our responses have addressed your concerns. If there are any unclear points or remaining questions, we would be happy to further clarify and discuss.

We thank the reviewer for the thoughtful feedback and helpful suggestions regarding fairness of comparisons, memory scaling, and evaluation clarity. We will revise the paper accordingly.

---

W1: Linear memory growth

We agree that the memory bank grows linearly with video length. However, in practice, we find the cost to be manageable:

The memory bank itself is lightweight. For example, storing 600 visual memory tokens of shape [600, 16, 18, 32] in float32 takes only about 21MB. Furthermore, since memory tokens are detached and not used in backpropagation during inference, the actual GPU memory footprint is low. Retrieval time increases slowly with memory size, making our method scalable in practice even for long sequences.

---

W2: Input length mismatch in comparison

Thank you for pointing this out. While our method uses 16 context frames plus 8 retrieved memory frames, other baselines typically use 16 continuous past frames. We agree this makes direct comparison not entirely fair. However, it is worth noting that the primary role of context frames in our method is to maintain short-term temporal smoothness, rather than to provide rich content information, since we have memory as context.

We provide the following ablation to show how the number of context frames affects performance. The results suggest that as long as the context includes a few relevant frames (e.g., more than 3), the generation quality remains stable:

---

W3: Per-frame noise level

We adopt the standard diffusion forcing setup, where each frame is trained with independently sampled noise levels. During training, we do not perform specific ablations to optimize per-frame-level noise strategies. Instead, we follow the default setting where each frame is independently perturbed with randomly sampled noise levels. This setup simplifies training and encourages the model to generalize across a range of noise conditions.

Our main motivation for using per-frame noise levels is to enable more flexible control during inference. In particular, we apply an autoregressive inference strategy, where frames are denoised sequentially.

---

Q1: Missing ground truth in Figure 6

Thank you for the suggestion. As this is a synthetic rotation trajectory not observed in training data, there is no ground truth to compare with. Instead, the goal of Fig. 6 is to visualize loop closure consistency over one full rotation, where the visual similarity between the first and last frames provides a qualitative indicator of 3D spatial consistency. We will make this clearer in the revised version.

---

Limitations missing in main paper

We appreciate this suggestion. We will lift this section into the main paper in the revised version.

---

We hope our responses have addressed your concerns. If there are any unclear points or remaining questions, we would be happy to further clarify and discuss.

We thank the reviewer for the detailed and insightful comments. Your feedback on memory usage analysis, encoding strategies, and scenario coverage has helped us better identify the limitations of our current design. We will revise the manuscript accordingly.

---

W1 & Q2: Lack of analysis on memory usage, temporal span, and practical constraints

We appreciate the reviewer’s concern regarding scalability and real-world applicability. Below, we provide quantitative analysis covering memory usage, generation duration, training cost, and inference efficiency.

• Memory usage of the memory bank:
  The memory bank is lightweight. Storing 600 visual memory tokens with shape [600, 16, 18, 32] in float32 only takes approximately 21MB.

• Generation duration:
  Our method maintains high-quality reconstruction over long sequences. Below are PSNR values at various lengths:

  Number of Generated Frames	PSNR
  100	23.1
  300	22.8
  500	22.7
  800	21.4
  1000	20.8

  These results indicate that our method maintains strong reconstruction quality up to 500 frames, and while degradation is observed at 1000 frames, the generation remains structurally coherent without collapse. This demonstrates that our method is capable of handling relatively long sequences, beyond the typical short-range settings explored in prior work.

• Retrieval latency (for 8 memory frames), as memory bank scales:

  Number of Memory Candidates	Retrieval Time (s)
  10	0.04
  100	0.06
  600	0.10
  1000	0.16

  The cost of generation (20 denoising steps) is ~0.9 seconds per frame (excluding retrieval). Thus, retrieval contributes only ~10–20% of the inference time, even when accessing 1000 candidates.

• Training setting:

  • ~500K training steps to converge on 4×H100 GPUs
  • Batch size: 4 (the original paper incorrectly reports this and will be corrected)
  • GPU memory usage per GPU: ~51GB
  • Total training time: ~4 days

• Cost Compared with Baseline

  To further address the reviewer’s concern regarding cost and practical feasibility, we include a detailed comparison below between our method and the baseline (without memory), under consistent settings.

  Setup:

  • 8 context frames + 8 memory frames
  • Inference uses 20 denoising steps
  • No acceleration techniques (e.g., timestep distillation, sparse attention)
  • Single H200

  	Training		Inference
  	Memory Usage	Speed (it/s)	Memory Usage	Speed (it/s)
  w/o memory	33 GB	3.19	9 GB	1.03
  with memory	51 GB	1.76	11 GB	0.89

  Introducing memory increases the computational cost during training, with a moderate rise in memory usage and a reduction in training speed. However, the overhead during inference is relatively minor, with only a small increase in memory usage (from 9 GB to 11 GB) and a slight drop in speed (from 1.03 it/s to 0.89 it/s). Overall, the additional cost introduced by memory is acceptable, particularly during inference, where the impact is minimal.

  Moreover, with modern acceleration techniques, such as timestep distillation, early exit, or sparse attention, the inference speed can be further improved to real-time levels (i.e., approximately 10 FPS), making the approach practical for deployment.

---

W2: Conventional encoding of pose and timestamp

We intentionally adopt standard encoding schemes to represent camera pose and timestamp. This choice ensures better generalization across environments and reduces inductive bias, making our framework more adaptable. While these encodings perform well in our current setting, we agree that more expressive alternatives may further improve performance. Also, such complex designs may incur significantly higher training and inference costs. We consider more sophisticated designs a promising direction for future exploration.

---

W3 & Q1: Limited dynamic and real-world scenario evaluation

Our current experiments focus on synthetic environments and small-scale real-world scenes, primarily due to resource limitations. Nonetheless, our framework is conceptually applicable to real-world scenarios, provided that sufficient sensory input and temporal alignment are available.

While our current setup does not include dynamic elements such as moving entities (e.g., zombies), this omission is mainly due to the significant computational cost required to train a video model capable of generating such dynamics with high fidelity. Importantly, our model architecture is inherently compatible with dynamic content. Each memory entry is timestamped, enabling the model to learn temporal evolution. For instance, recognizing that a walking zombie at time $t_i$ should appear at a different location at time $t_{i+n}$​. This capability is a core motivation behind our design of state-aware memory, which we plan to further explore in future work involving dynamic scenes.

We agree that demonstrating results on more diverse environments, including outdoor and highly dynamic scenes, would strengthen our claims. We consider this an important direction for future work and are working toward scaling the framework to support such settings.

---

Q3: Failure cases and limitations

We appreciate the reviewer’s suggestion and will include a dedicated limitations and failure case section in the revised paper. In fact, we have discussed several of these issues in the supplementary and will lift them into the main paper. Below, we summarize key failure modes observed in our current system:

• Retrieval under occlusion: Our current memory retrieval strategy is based on view overlap, assuming that overlapping FoVs imply semantic relevance. However, this assumption breaks in the presence of occluders. For example, if the current viewpoint faces a wall, the retrieved frames with similar FoVs may be located behind the wall and thus contain irrelevant content. Our method does not yet model occlusion-aware visibility, which limits retrieval accuracy in cluttered environments.

• Long-horizon generation degradation: While our method supports relatively long sequences (e.g., up to 1000 frames), it is still subject to common issues in autoregressive generation, such as quality degradation and error accumulation. As the sequence length increases, we observe texture blurring, loss of spatial detail, and occasional color artifacts (e.g., patchy or unnatural regions), especially beyond 1000 frames. These issues are consistent with failure modes observed in other long-range video generation systems.

We consider addressing these limitations in future work, such as incorporating visibility-awareness into memory retrieval and exploring strategies to improve long-horizon consistency.

---

We hope our responses have addressed your concerns. If any points remain unclear, we would be happy to further clarify and discuss.