WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception

Author Response to Reviewer 6nHu

We sincerely appreciate your recognition and thoughtful feedback, which are instrumental in refining and strengthening our work. We hope the following response adequately addresses your question.

Q1: Clarifying the paper organization.

A1: Thank you for pointing this out. In the revision we will split Section 3.3 into two distinct subsections: (1) Perceptual Memory Bank (design, motivation and ablations), and (2) Group‑wise Noise Scheduling (training/inference alignment, efficiency). This re‑structure cleanly separates the two contributions and improves readability without altering technical content.

Q2: Depth/Flow mis‑alignment mitigation.

A2: Thank you for the question. For depth we apply an affine‑invariant normalization to convert raw depth maps into relative depths in $[-1,1]$, following Marigold[1]:

where $d_{2}$ and $d_{98}$ are the 2nd and 98th percentiles of each frame’s depth distribution. This removes extreme outliers and global scale/shift variations so the model focuses on relative spatial relationships.

For optical flow we divide each displacement vector by the frame diagonal, $\sqrt{H^2 + W^2}$. Since all training clips use the same resolution, this aligns motion magnitudes across the dataset. The normalized flow is then encoded as an RGB video (hue = direction, opacity = magnitude).

Q3: Benefits of internal datasets.

A3: Our task requires full-model fine-tuning rather than freezing the backbone or applying lightweight adaptation methods such as LoRA. However, most large-scale publicly available datasets contain various quality issues—such as embedded subtitles, watermarks, or overly chaotic motion patterns—which significantly affect the training process. Notably, existing models like the pretrained Wan, Magi, and SkyReels-V2 have all undergone a final-stage supervised fine-tuning (SFT) on carefully curated high-quality data. If we were to further fine-tune such models on noisy public data, it would significantly degrade video quality and obscure the true contribution of our modeling framework. In contrast, our internal general-purpose dataset provides relatively higher data quality—with better control over subtitles, watermark removal.

We fully agree that more thorough evaluation across diverse datasets is important. As presented in Table 2 of the main paper, we have already conducted a comparison across multiple methods using the same public robotic manipulation dataset. However, due to the inherent difficulty of reliably cleaning and filtering large-scale public video datasets—e.g., removing subtitles, watermarks, and abrupt transitions—we adopted a controlled setting using our internal general-purpose dataset. Specifically, we re-trained multiple baseline methods on the same internal dataset to ensure fair and consistent evaluation across architectures. This setup allows us to isolate the effect of model design without the confounding influence of data quality variation. We will also include additional results on public general-purpose datasets in the revised version to further demonstrate the robustness and generalization ability of our method.

Q4: Test on in-the-wild inputs.

A4: We thank the reviewer for their valuable feedback regarding the evaluation on in-the-wild inputs. In our experimental design, we utilized text prompts and initial images generated by a third-party tool (ChatGPT). This approach was intentionally chosen to establish a fair and standardized benchmark for all compared methods, thereby eliminating potential biases that could arise from manually curated prompts.

Regarding the model's performance on in-the-wild data, our method exhibits strong robustness. This is because the model itself was trained on large-scale, in-the-wild datasets. Furthermore, our qualitative results on the robotic manipulation dataset are generated from real-world scenarios.

We will incorporate extensive results on in-the-wild videos in the final revision to demonstrate our framework's robustness, as we are limited from showing new results during the rebuttal period.

Q5: Dynamic cases.

A5: We agree with the reviewer on the importance of showcasing more dynamic cases, particularly those with complex motions and semantic transitions, and will incorporate them into the final version. We candidly acknowledge that while our joint training approach improves upon RGB-only models, generating long videos with large motion (like long dance sequences) remains a common challenge for the field. This is largely due to error accumulation in streaming frameworks, where a single degraded frame can impact the entire subsequent sequence.

Our method helps to mitigate this issue. By jointly modeling RGB with stable perceptual cues like depth and flow, it generates a more reliable initial context, thus providing a more robust basis for subsequent frames.

[1] Ke, Bingxin, et al. "Repurposing diffusion-based image generators for monocular depth estimation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

Author Response to Reviewer bGie

We sincerely appreciate your valuable feedback and your positive assessment of our work. Below, we address your comments in detail.

Q1: Clarifying the contributions beyond existing methods.

A1: While our work is partially inspired by VideoJAM, our contributions are distinct in two key aspects.

• First, we systematically analyze different perceptual conditions and find that video depth offers more comprehensive improvements —enhancing both motion quality and temporal consistency—compared to optical flow. Its robustness to visual drift is a key insight that motivates our design.
• Second, we address a fundamental tension in prior methods like Diffusion Forcing: low noise in historical frames leads to drift, while high noise limits useful context. We resolve this by leveraging depth’s stability and proposing a memory-augmented framework that retains clearer historical cues. This allows effective use of long-term context without sacrificing consistency.

Author Response to Reviewer vMBw

We sincerely appreciate your kind words about our work. We believe the response below clarifies your concern.

Q1: Limited novelty.

A1: Thank you for your feedback. While our work is partially inspired by VideoJAM, our contributions are distinct in two key aspects.

• First, we systematically analyze different perceptual conditions and find that video depth offers more comprehensive improvements —enhancing both motion quality and temporal consistency—compared to optical flow. Its robustness to visual drift is a key insight that motivates our design.
• Second, we address a dilemma in prior methods like Diffusion Forcing: low noise in historical frames leads to drift, while high noise limits useful context. We resolve this by leveraging depth’s stability and proposing a memory-augmented framework that retains clearer historical cues. This allows effective use of long-term context without sacrificing consistency.

Q2: Ablation study on general-purpose datasets.

A2: We completely agree with your point that evaluation on general-purpose datasets is essential for validating broader applicability. Due to computational cost constraints, we selected a subset of baselines—CausalVid, Diffusion Forcing, as well as our own framework without perceptual conditioning—and retrained them on the same general-purpose dataset using the Wan-1.3B backbone for direct comparison. These results are consistent with the findings reported in the main paper.

Q3: Definition of Converged Models.

A3: Thank you very much for pointing this out. We used equal computational resources and training configurations for all methods to ensure fairness, except for the total number of training steps, which was intentionally varied to investigate the convergence behavior of different algorithms. In practice, we do not rely solely on loss values to determine convergence. Instead, we monitor both quantitative evaluation metrics and visual results at inference time. Once both have stabilized and no further improvement is observed, we consider the model to be converged and record the corresponding number of training steps. We also continue training beyond the observed convergence point to ensure that no further improvements can be achieved. We will clarify this definition explicitly in the revised version.

Q4: Lack of a fair baseline using same base model.

A4: Thank you for the insightful suggestion. Our method is based on a 1.3B text-to-video (T2V) model, while Wan does not provide a corresponding 1.3B image-to-video model. As an alternative, we use the Wan-14B image-to-video model for evaluation on the same benchmark. It is worth noting that these pretrained model baselines have undergone high-quality supervised fine-tuning (SFT), whereas our training data is not as high-quality as the curated SFT data.

As shown in the table below, we observe that increasing model size does lead to improvements in overall video quality and motion scores. Nevertheless, these larger models still exhibit more pronounced temporal drift. This is largely due to the repeated image-to-video generation strategy, which lacks access to prior context and suffers from a severe training-inference gap.

Despite using a smaller 1.3B model, our method significantly mitigates temporal drift and improves long-horizon video quality through its architectural and training design. Moreover, as presented in Table 2 of the main paper, we compare repeated image-to-video generation with other autoregressive and streaming baselines on the same dataset, and consistently find that autoregressive generation is more suitable for long-horizon video synthesis.

Q5: Clarification.

A5.1 Preliminary: Thank you for the suggestion. We will expand the preliminary section in the revised version to better support a broader audience

A5.2 Complexity of VideoJAM's sampling guidance: VideoJAM’s sampling guidance is considered complex for two main reasons. First, it requires additional forward passes during inference, which increases computational overhead. Second, it introduces multiple guidance scales as additional inference-time parameters. These parameters can affect generation quality and may require manual adjustment depending on the scene or prompt.

A5.3 Typos: We will carefully proofread the paper and correct all typographical and formatting errors in the revised version.

A5.4 Depth estimation robustness: Our depth pipeline is based on state-of-the-art models (Video Depth Anything), which are robust across a wide range of scenes, making the overall depth processing pipeline reliable in practice. Crucially, our method does not rely on precise metric depth. We apply an affine‑invariant normalization to convert raw depth maps into relative depths in $[-1,1]$, following Marigold[1]:

where $d_{2}$ and $d_{98}$ are the 2nd and 98th percentiles of each frame’s depth distribution. This removes extreme outliers and global scale/shift variations so the model focuses on relative spatial relationships. Depth serves as a relative structural prior, providing stable geometric cues that help maintain temporal consistency and stabilize long-horizon generation—which current models are sufficient to support.

[1] Ke, Bingxin, et al. "Repurposing diffusion-based image generators for monocular depth estimation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.

I thank the authors for their detailed response. Most of my concerns have been addressed. I believe the paper is above the acceptance bar despite the concern of novelty. However, the messages conveyed by the paper, e.g., depth > optical flow for long video generation, are useful for the community. Thus, I would like to keep my original rating. Please include these results during the rebuttal phase into the paper.