Video World Models with Long-term Spatial Memory

Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.

---

W1: Wan2.1 is not directly comparable.

Thanks for the feedback. We make slight adaptations to the Wan2.1-I2V so that it could be comparable in the context. Specifically, we use static point cloud rendering to initialize video latents to implement camera control like trajectorycrafter, and add a low level of noise on top. To ensure the motion coherence of dynamic elements, we select the recent frames as the reference frames. Based on the original backbone of Wan2.1-I2V, we perform dual encoding on reference frames through Clip visual encoder and 3D-VAE respectively: the semantic features encoded by Clip-vision are concatenated with text prompts to maintain content consistency, and the spatial features extracted by 3D-VAE are concatenated with video latents to enhance geometric continuity, finally achieving a temporal progressive NVS task similar to ours.

---

W2: Lack of theoretical/insightful analysis.

Thank you for the thoughtful comment. The working memory primarily derives from recent context frames. These clean reference contexts are concatenated with noisy video latents at the frame level, enabling us to adapt the self-attention layers into reference mixing layers that effectively capture cross-view dependencies. This architecture not only enhances 3D consistency but also improves short-term motion coherence; The episodic memory system mainly derives from historical reference key frames. Compared to working memory, these views originate from more distant camera positions, endowing the model with stronger robustness. Particularly when user-input trajectories exhibit "revisit" (overlaps with previously generated regions), our designed historical cross-attention mechanism can effectively retrieve and leverage these stored details. This approach ensures long-term consistency while preventing memory forgetting, as the system maintains access to relevant historical information even during novel view synthesis. We will add the analysis to the final version.

---

W3: More convincing testset setup.

Thank you for the insightful feedback. We are unable to directly adopt the benchmark established in prior work, as the task definitions are not fully aligned and their dataset is incompatible with the requirements of our setting. But in order to mitigate the impact of the domain gap, we construct a new subset of 100 highly dynamic scenes covering both real-world and game-engine domains, including rainy and snowy weather as well as day and night environments from Sekai [1], a large-scale video world exploration dataset. Specifically, for the reconstruction metrics, we far surpass other methods, which reflect our ability to keep 3D consistency and control camera accuracy. In terms of vbench metrics, we show superior video quality, although aesthetic quality and imaging quality lag slightly behind Wan2.1-inpainting, as mentioned in Sec.4.1 of our paper, this is due to their advanced backbone. Overall, our method still shows robust performance in these difficult and OOD cases.

[1] Li, Zhen, et al. "Sekai: A Video Dataset towards World Exploration." arXiv preprint arXiv:2506.15675 (2025).

(Bold is best, italic is the second.)

---

W4: Issues in clarity.

Thank you for pointing this out. We will carefully review the writing and revise the final version to avoid vague or unsupported adjectives.

---

W5: Insight and originality of the work.

Beyond the spatial memory mechanism, another major difference from TrajectoryCrafter and other point cloud-conditioned methods is our ability to handle temporally misaligned inputs and outputs. Specifically, while TrajectoryCrafter performs dynamic novel view synthesis with temporally aligned inputs and outputs, our method predicts future observations based on static conditions and dynamic references stored in working memory. This goes beyond a simple inpainting task. We believe that the problem setup, the construction of corresponding data pairs, and the proposed model design will be inspiring to the community.

---

Q1: Metric clarification.

In Table 1, user study scores range from 1 to 4, while reconstruction metrics (PSNR, SSIM, LPIPS) mainly assess camera accuracy and 3D consistency in static regions. Because our model uses a 3D VAE for video encoding/decoding, these metrics are particularly sensitive—compression-like processing can introduce measurable deviations even when static regions appear nearly identical to human observers. Following your suggestion, we tested repeating static videos and found approximate ranges: PSNR ↑ (0–22), SSIM ↑ (−1 to 0.75), LPIPS ↓ ( 0.15 to +∞).

While the metrics in Tables 2 and 3 are from VBench, a widely used video generation benchmark. According to the VBench paper, all metrics are normalized to 0–100%, where 100% is ideal and lower bounds indicate worst case. Aesthetic Quality (0–1.0) measures artistic merit, Imaging Quality (0–1.0) detects blur/noise, Temporal Flickering (0.6293–1.0) gauges frame consistency, Motion Smoothness (0.7060–0.9975) checks physics-compliant motion, Subject Consistency (0.1462–1.0) tracks foreground stability, and Background Consistency(0.2615–1.0) evaluates scene coherence.

---

Q2: Additional experimental setup.

Following the widely adopted experimental setup in previous video generation benchmarks, we fix a single random seed for all experiments and report the corresponding results. We also observe that running the experiments with different seeds and averaging the results over the full 500-sample test set leads to only negligible variations.

---

Q3: Testset setup.

The 500 test cases are sampled entirely from unseen videos. Specifically, the raw videos have been pre-split into disjoint training and testing sets, ensuring that test clips and training clips never come from the same original video.

---

Q4: User study setup.

In the final version, we will include screenshots of the user study questionnaire and provide detailed user statistics for clarification. Briefly, we invited 13 graduate students working on video generation research to participate in the user study. Each participant was presented with ground truth and generated videos for evaluation. We report the average human ranking scores based on their responses.

---

Q5: Quantitative results of view recall consistency for the ablation.

Thanks for the insightful suggestion. We report the quantitative results of view recall consistency for the ablation as below. Our full model, which uses short-term working and long-term spatial and episodic memory, performs best on reconstruction metrics, demonstrating the effectiveness of the memory mechanism in keeping long-term 3D consistency.

Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.

---

Q1&Q5: Include more related works.

Thank you for the thoughtful suggestions. We will cite the related works [1-4] and include proper discussions in the final version.

---

Q2: Explicit VS Implicit 3D representations.

Thank you for the insightful and valuable feedback. Both explicit and implicit 3D representations have their respective advantages and limitations. Explicit representations (e.g, point clouds) offer high precision, easy editability, and strong compatibility, making them ideal for scenarios requiring strict geometric control, though they come with higher storage and computational costs than implicit representation. In contrast, implicit representations excel in efficient compression and scalability, but face challenges in interpretability and training complexity. We believe the future lies in hybrid representations, for instance, using implicit encoding for global semantics while employing explicit structures for local detail processing. We will continue to explore this direction in our future research.

---

Q3: 3D reconstruction method.

We leverage Mega-SaM during dataset construction and CUT3R during inference for 3D reconstruction. Since their outputs are frame-wise point maps, we further apply TSDF-Fusion to aggregate them into a global point cloud and filter out dynamic elements such as characters and animals. This combination of state-of-the-art 3D reconstruction methods with the traditional TSDF-Fusion approach enables efficient and accurate camera pose estimation and global point cloud reconstruction.

---

Q4: Dynamic humans / other agents in static scenes.

Thank you for the thoughtful comment. Dynamic scenes indeed pose significant challenges. As mentioned in Q3, we chose Mega-SaM and CUT3R in place of VGGT as our main reconstruction tools, as both are specifically designed and trained for dynamic scene reconstruction to ensure accurate camera poses and point maps. However, directly combining per-frame point maps leads to artifacts and misalignment of dynamic elements. To address this, we further incorporate TSDF-Fusion to filter out dynamic content and obtain a clean, static global point cloud.

Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.

---

W1: Comparisons with recent works.

Thank you for the valuable suggestion. We will include Gen3C and more recent works as additional baselines in our final version.

---

W2&Q4: Metric choice biases and to report FVD.

Thank you for the constructive feedback. For evaluating dynamics and actions, we follow prior works by reporting metrics such as motion smoothness from VBench, which is one of the most widely used indicators of motion quality. We would be happy to include additional metrics if the reviewer could kindly provide suggestions. As requested, we also report the FVD results below, where our method demonstrates superior performance compared to the baselines.

---

Q2: Dataset release.

While we are unable to include external links in the response due to the rebuttal policy, the dataset has been fully prepared. We will ensure it is released promptly upon paper acceptance and include the link in the final version.

---

Q3: Did you retrain the baselines on your dataset?

Thank you for the thoughtful question. We have attempted to retrain the baseline models on our dataset, but their performance deteriorates as training progresses. This is because prior works rely on input conditions and outputs that are temporally aligned, whereas our task involves predicting future observations with temporally misaligned inputs and outputs. The data pairs are constructed accordingly, making the problem more challenging. Without tailored architectural modifications, baseline models struggle to adapt. Moreover, since all baselines are pretrained on general video datasets such as OpenVid and MiraData, which share a similar domain with ours, the domain gap is minimal.

---

Q5: Wall-clock time per generated 30-frame video and GPU memory footprint.

Our memory footprint is approximately 25GB, and generation FPS is 1.8 minutes for 30 frames on an Nvidia A800.

---

Q6: Performance degrades when the depth estimator produces noisy maps.

We construct a new subset of 100 highly dynamic scenes covering both real-world and game-engine domains, including rainy and snowy weather as well as day and night environments from Sekai [1], a large-scale video world exploration dataset. Our evaluation reveal depth estimator perform well for night scenes and rainy/snowy environments, though the TSDF fusion stage may not fully filter out dynamic elements, leading to artifacts. However, our video generation backbone leverages static point cloud rendering as conditioning and integrates clean context into the video latent space. This design allows the self-attention layers to effectively capture cross-view dependencies, enabling the system to repair distortions while maintaining 3D consistency. As shown below, our results on this challenging subset remain competitive compared to other methods.

(Bold is best, italic is the second.)

[1] Li, Zhen, et al. "Sekai: A Video Dataset towards World Exploration." arXiv preprint arXiv:2506.15675 (2025).

Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.

---

W1&Q1: Broader baseline set.

Thank you for the thoughtful suggestions. As LinGen only has an unofficial implementation and does not provide a pretrained checkpoint, it is difficult to include in the current rebuttal phase. Instead, we conducted experiments on StreamingT2V and History-Guided Video Diffusion (Dfot, trained on RE10K) using the RE10K testset for comparison. The results are shown below:

We achieve second-best results in terms of aesthetic quality and temporal flickering, while significantly outperforming other methods on subject consistency and background consistency, thanks to the incorporation of geometric memory. These improvements are well aligned with the core motivation behind our model design.

---

W2: Scalability to larger backbones or diverse datasets.

Thank you for the thoughtful feedback. We conducted our experiments on CogVideoX and the current dataset due to limited computational resources, yet the results are already highly encouraging. We would like to clarify that scaling up from the 5B CogVideoX model to larger backbones, such as the 14B Wan2.1 model, is feasible, as they share similar DiT architectures and the parameter gap is not substantial. Furthermore, although our task focuses on camera-guided video generation, the training data pairs we use ultimately fall under the caption-video category, similar to those employed in I2V and T2V tasks. Therefore, scaling up the dataset is entirely reasonable. Due to the limited time for rebuttal, we plan to explore larger backbones and a more diverse dataset than the current version for the final version.

---

W3: Dataset release.

While we are unable to include external links in the response due to the rebuttal policy, the dataset has been fully prepared. We will ensure it is released promptly upon paper acceptance and include the link in the final version.

---

W3&Q2: Dynamic‑scene analysis.

Thanks for the insightful comment.

First, to handle highly dynamic scenes with drastic camera motion, our strategy involves scaling the depth within the current range and reducing the voxel volume size. This adjustment yields a denser, filtered static point cloud, mitigating the issue of incomplete static-dynamic decoupling caused by extreme motion. By refining the static representation, we prevent excessive sparsity in the static spatial memory, thereby improving overall generation quality.

Second, We construct a new subset of 100 highly dynamic scenes covering both real-world and game-engine domains, including rainy and snowy weather as well as day and night environments from Sekai [1], a large-scale video world exploration dataset. Specifically, for the reconstruction metrics, we far surpass other methods, which reflect our ability to keep 3D consistency and control camera accuracy. In terms of vbench metrics, we show superior video quality, although aesthetic quality and imaging quality lag slightly behind Wan2.1-inpainting, as mentioned in Sec.4.1 of our paper, this is due to their advanced backbone. Overall, our method still shows robust performance in these difficult cases.

[1] Li, Zhen, et al. "Sekai: A Video Dataset towards World Exploration." arXiv preprint arXiv:2506.15675 (2025).

(Bold is best, italic is the second.)

---

Q3: Ablation on memory size; Present memory foot‑print and generation FPS.

1. Since we use point cloud renderings as the conditioning input, the resolution of the point cloud does not directly impact the memory footprint. Additionally, the key-frame buffer is relatively small compared to the overall model size, so there is no significant trade-off within the practical range.
2. Our memory footprint is approximately 25GB, and generation FPS is 3.2 minutes for 49 frames on an Nvidia A800. We would like to clarify that the primary focus of our paper is on incorporating long-term and spatial memory into video models, and we do not claim real-time performance. However, we fully agree that bridging academic research and practical deployment is an important direction. In future work, we plan to explore lightweight alternatives and acceleration strategies such as Distribution Matching Distillation to further reduce memory usage and improve inference speed.