DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

---

Q1: Heavy Reliance on Specific Foundation Models & Reproducibility and Accessibility Concerns

We thank the reviewer for these critical points. Our pipeline's core design philosophy of modularity directly addresses these concerns. Instead of a monolithic system, each key component (depth, segmentation, VLM) is a "plug-and-play" module. This architecture makes our framework adaptable across domains (e.g., swapping in models for robotics or autonomous driving), future-proof (easily integrating new SOTA models), and accessible (allowing users to substitute heavyweight components with lighter, open-source alternatives). Therefore, our primary technical contribution is the versatile and durable framework itself.

Q2: Specifics of Human-in-the-Loop Review.

Thank you for these important questions. We provide the details of our review process below.

1.Scale and Purpose: Our human review was a quality audit on a ~10K scene subset, focusing on semantic annotations (object, scene, camera captions)

2.Correction Rate & Impact: Within this subset, ~19% of captions required minor corrections (e.g., for specificity). This feedback was used both to fix samples and to iteratively improve our pipeline's prompting strategies.

3.Annotator Team: The review was conducted in-house by authors and graduate students deeply familiar with the project's quality standards.

4.Consistency Assurance: We ensured consistency not via formal IAA metrics, but through a collaborative review process, where discrepancies in overlapping batches were resolved through group discussion with a senior author.

We will add a new section to our appendix detailing this entire human-in-the-loop procedure provide a comprehensive account of our quality assurance procedure.

Q3: DynamicVerse Dataset Validation.

We thank the reviewer for this critical feedback. To directly demonstrate our dataset's utility, we conducted new experiments which provide strong evidence for the quality and effectiveness of the geometric and semantic annotations in DynamicVerse.

1. Geometric Validation: Fine-tuning MonST3R：

To validate the quality and utility of our geometric annotations (i.e., metric-scale point maps and camera parameters), we used a subset of 1000 scenes from DynamicVerse to fine-tune the prominent model, MonST3R [1]. As shown in the table below, fine-tuning on our data leads to a performance improvement on the Sintel benchmark. This directly confirms that the geometry in DynamicVerse is accurate and beneficial for improving existing models.

2. Semantic Validation: Downstream Tasks and Direct Evaluation：

We performed three distinct experiments to validate the high quality of our hierarchical semantic annotations:

(a) Object-Level Semantics via 4D-LangSplat [2]: To demonstrate the effectiveness of our dynamic object annotations, we used them to train a 4D-LangSplat model. Our precise object masks and labels enabled the model to achieve superior results in language-grounded 4D scene understanding. This validates that our object-level annotations are highly effective for complex, multi-modal downstream tasks.

(b) Scene-Level Semantics via G-VEval [3]: We used the established G-VEval benchmark [3] to quantitatively assess our generated scene-level captions. On a random sample of 100 videos from filtered SA-V data [4], our captions achieved a high score, confirming their coherence, accuracy, and richness.

The strong performance across these metrics confirms that our captions are not only factually accurate and relevant to the video content, but also complete in their coverage of events and efficiently concise. This robust, multi-faceted quality makes them highly suitable and reliable for demanding downstream applications.

(c) Camera-Level Semantics via Human Study: We conducted a formal human study on 88 videos from our DAVIS subset to quantitatively analyze our camera motion captions. Following the criteria of Clearness, Conciseness, and Grammar & Fluency from prior work [5], the results were excellent: over 60.22% of captions were rated as both clear and fluent while also receiving high scores for conciseness, confirming the effectiveness of our generation and quality control process.

Q4: Lack of Failure Case Analysis.

We thank the reviewer for this excellent suggestion. We agree that quantifying the frequency and impact of failure cases is crucial for understanding our pipeline's robustness. We have conducted a new analysis to provide these details:

(1) Failure Frequency Analysis

We analyzed the failure rates of our key components on challenging data to quantify their robustness.

A. Moving Object Segmentation: We manually identified challenging subsets within the Sintel training set to analyze our segmentation performance in edge cases. We found failure rates of 28% (3 of 14 scenes) for severe occlusion and 11% (1 of 9 scenes) for low-light conditions.

B. Canonicalization: The primary failure mode is the lack of a detectable horizontal surface (e.g., mid-air scenes). On the DAVIS subset, this process fails in approximately 15.9% of the 88 filtered scenes.

(2) Impact on Downstream Utility of Segmentation

To quantify how segmentation errors affect the final geometric annotations, we performed a controlled experiment on a challenging scene. We compared the final depth and pose metrics when using (1) our pipeline's automated masks vs. (2) manually assited, pseudo ground-truth-level masks. Specifically, we manually identified dynamic objects by placing point prompts on them.

As shown, while segmentation errors do degrade the final geometric accuracy, the results from our automated pipeline remain reasonable. This indicates a degree of resilience in our downstream bundle adjustment process to minor imperfections in segmentation.

Q5: Design Choice on Model Selection.

While the powerful SA2VA model can perform both segmentation and captioning, we use the specialist Describe Anything Model (DAM) for captioning to achieve superior quality and flexibility. We chose DAM for two advantages over SA2VA: (1) Superior Detail: DAM is optimized for generating richer, more localized descriptions. (2) Mask-Conditioned Input: Crucially, DAM can take GT masks (e.g., from 2D segmentation datasets) as input for precise, targeted captioning, a feature SA2VA lacks but was a requirement for our pipeline.

Q6: Details of Camera Motion Captioning.

Thank you for these detailed questions. We generate camera motion captions using a specialist VLM from CameraBench [5], which was fine-tuned on ~3K expert-annotated videos. To ensure high fidelity and mitigate hallucinations, we employ a two-stage quality control process: first, our upstream data filtering removes videos with erratic motion, and second, a final human review corrects any remaining inaccuracies. The effectiveness of this entire process was confirmed through a human study on 88 DAVIS videos, where our captions demonstrated high quality across the criteria of Clearness, Conciseness, and Fluency [5] (detailed results are in the table for Q3).

Q7: Limitation & Typo.

We are grateful for the reviewer's guidance and careful reading. We agree that the limitations section can be significantly improved. In the final version, we will expand it to explicitly discuss the pipeline's failure modes and our reliance on large-scale foundation models, as you helpfully pointed out.

We would like to sincerely thank you once again for your constructive comments throughout the review process.

---

Referenecs:

[1] MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion. ICLR 2025.

[2] 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models. CVPR 2025.

[3] G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o. AAAI 2025.

[4] SAM 2: Segment Anything in Images and Videos. ICLR 2025.

[5] CameraBench: Towards Understanding Camera Motions in Any Video. Arxiv 2025.

Dear Reviewer uDiP,

Thanks again for your constructive review. To make sure we fully address your concerns on Q3: DynamicVerse Dataset Validation, we wish to supplement our quantitative experiments with the following qualitative evidence. We strongly encourage you to examine the compelling visual proof within our supplementary material:

• Demo Video (suppl. - webpage/index.html) showcases the reconstructed moving objects, dynamic scenes, and metric-scale camera trajectories. Specifically, these demos include challenging cases such as significant occlusion (e.g., the "Public Bus" scene) and fast motion (e.g., the "Public Bus" and "Walking Woman" scenes). This video evidence directly demonstrates the high quality of our dataset annotations in these very challenging scenarios.
• Figures 2-4 (suppl.) showcase our metric-scale 4D reconstruction results on several challenging in-the-wild videos. These qualitative examples highlight our method's ability to produce cleaner static backgrounds and more coherent shapes for dynamic objects, with significantly fewer artifacts like "ghosting" or "streaking" compared to baselines. These results serve as compelling evidence that our full pipeline is robust and produces high-fidelity geometric annotations even from complex, unconstrained video inputs.
• Figure 5 (suppl.) provides a direct comparison of our dynamic object segmentation against other methods. We urge the reviewer to note how our approach maintains sharper boundaries and greater temporal consistency, especially for fast-moving objects (e.g., "Sprinting" and "Car Racing Competition" scenes). Crucially, it demonstrates superior robustness to partial occlusions (e.g., "Figure skating" scene), correctly segmenting objects even when parts are momentarily hidden. This precise segmentation is the cornerstone of robust downstream 4D reconstruction.

Together, this qualitative evidence visually corroborates the strong quantitative results, confirming that our DynamicGen pipeline produces the high-fidelity geometric and semantic annotations that constitute our DynamicVerse dataset, proving its suitability for demanding downstream tasks.

Best,

DynamicVerse's authors.

Dear Reviewer uDiP,

We sincerely thank you for your positive assessment and for recommending our paper for acceptance.

We are particularly grateful for your recognition of our "solid pipeline," the "thoughtful engineering" of its components, and the value of DynamicVerse as a "valuable resource" for the community. Your supportive and constructive review is greatly appreciated.

Best,

DynamicVerse's authors.

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

---

Q1: Lack of Video Results / Insufficient Visualization / Robustness in Challenging Scenarios.

We completely agree that video demonstrations are the most effective way to intuitively analyze the quality, realism, and robustness of our generated 4D data. While the NeurIPS policy prohibits providing images, videos, or other rich media during the rebuttal period, we are fully committed to significantly expanding our supplementary material.

• Demo Video (suppl. - webpage/index.html) showcases the reconstructed moving objects, dynamic scenes, and metric-scale camera trajectories. Specifically, these demos include challenging cases such as significant occlusion (e.g., the "Public Bus" scene) and fast motion (e.g., the "Public Bus" and "Walking Woman" scenes). This video evidence directly demonstrates the robustness of our pipeline on the very types of challenging scenarios the reviewer highlighted.
• Figures 2-4 (suppl.) showcase our metric-scale 4D reconstruction results on several challenging in-the-wild videos. These qualitative examples highlight our method's ability to produce cleaner static backgrounds and more coherent shapes for dynamic objects, with significantly fewer artifacts like "ghosting" or "streaking" compared to baselines. These results serve as compelling evidence that our full pipeline is robust and produces high-fidelity reconstructions even from complex, unconstrained video inputs.
• Figure 5 (suppl.) provides a direct comparison of our dynamic object segmentation against other methods. We urge the reviewer to note how our approach maintains sharper boundaries and greater temporal consistency, especially for fast-moving objects (e.g., "Sprinting" and "Car Racing Competition" scenes). Crucially, it demonstrates superior robustness to partial occlusions (e.g., "Figure skating" scene), correctly segmenting objects even when parts are momentarily hidden. This precise segmentation is the cornerstone of robust downstream 4D reconstruction.

Q2: Missing Loss Weights for Eq. (1).

Thank you for highlighting this. Eq. (1) is a conceptual overview for our multi-stage pipeline, not a single weighted loss. We optimize different subsets of these terms sequentially in distinct stages for optimal performance. Therefore, weights are defined per-stage, as detailed below:

• Stage 3 (Static BA): We set weights $\lambda_\text{BA}=\text{1.0}$ and $\lambda_\text{cam}=\text{1.0}$.
• Stage 4 (Non-rigid BA): We set weights $\lambda_\text{NR}=\text{1.0}$, $\lambda_\text{NR}=\text{100}$ and $\lambda_\text{motion}=\text{10}$.

We will add this detailed breakdown to the revised paper.

Q3: More Evaluation to Prove Technical & Data Contribution.

We thank the reviewer for this excellent suggestion. To demonstrate our work's contribution, we provide new experiments validating our dataset's utility and clarifying our pipeline's technical novelty.

1. Data Contribution: Validating DynamicVerse Annotations

To prove the value of our dataset, we conducted new experiments validating both its semantic and geometric annotations, which are the core components of DynamicVerse.

A. Validating Semantic Annotations on a Demanding Cross-Modal Task: To directly address the reviewer's suggestion, we validated our semantic annotations on a challenging cross-modal task that exemplifies the "linguistic 4D Gaussian Splatting" concept from our introduction (lines 40-41): language-grounded 4D scene understanding, using the established 4D-LangSplat [1] model.

To isolate the impact of our annotations, we replaced the sparse labels in the 4D-LangSplat with our pipeline's richer descriptions. The resulting performance boost (shown below) provides direct quantitative proof of the value of the high-quality annotations that comprise our DynamicVerse dataset.

B. Validating Geometric Annotations via Fine-tuning: To demonstrate the quality and utility of our geometric data, we fine-tuned the prominent model MonST3R [2] on a subset (~1000 scene) of DynamicVerse. The model, now enhanced with our data, shows a performance improvement on the standard Sintel benchmark. This confirms that our generated geometry is not only accurate but also provides valuable diversity that improves the robustness and performance of existing models.

2.Technical Contribution: Novelty Beyond Integration

The creation of such high-quality data was only possible due to our pipeline's significant technical contributions, which go far beyond combining existing models.

A. Geometric Novelty

Our dynamic BA pipeline introduces three key components absent in prior work: (a) Epi-Mask Filtering, (b) VLM-Based Dynamics Analysis, and (c) Sliding Window Global Refinement. The ablation study below quantifies the significant impact of each component.

B. Semantic Novelty

Our pipeline introduces a sophisticated, multi-level captioning system that moves far beyond the sparse object labels of prior work. This is achieved through several novel techniques:(a) Semantic-Aware Keyframe Extraction (SAKFE), (b) Hierarchical Prompting (HP), (c) Automated Caption Rephrasing and (d) Chain-of-Thought (CoT) Prompting.

In summary, by demonstrating significant improvements on both a novel cross-modal task and an established fine-tuning task, and by detailing the novel components of our pipeline, we have provided concrete new evidence for both the data contribution of DynamicVerse and the technical contribution of DynamicGen. We thank the reviewer again for prompting this clarification.

Q4: More Dataset Statistics.

We have performed a detailed characterization of the dynamic content within a significant subset of 1,268 videos (filtered from the original 3,471 in YouTube-VIS). We will include the full statistics in our appendix, and a summary of our findings on the distribution of moving objects and actions is provided below.

(1) Moving Object Distribution: This subset contains 2,158 unique moving object instances across 355 distinct categories. The distribution features a high frequency of common objects followed by a long tail of more specific categories, ensuring a mix of both common and rare examples. The top 5 most frequent categories are: person (16.2%), panda (4.4%), rabbit (2.6%), bird (2.5%), and giraffe (2.2%), showcasing a rich variety of both human and animal subjects.

(2) Action and Motion Distribution: To characterize the "body actions," we analyzed the verbs from our generated captions. The distribution highlights a strong focus on dynamic interactions and object manipulation. The top 5 most frequent actions are: moving (276 instances), holding (178), wearing (144), walking (80), and standing (78). The full distribution covers a wide array of fine-grained actions.

---

Reference:

[1] 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models. CVPR 2025.

[2] MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion. ICLR 2025.

[3] G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o. AAAI 2025.

We sincerely thank the reviewer for their positive and encouraging feedback. We are delighted that the reviewer found our paper to be a high-impact contribution with thorough experimental validation and a clear presentation. We address the remaining points below.

---

Q1: Questions on Generalizability Beyond Internet Videos.

We thank the reviewer for this insightful question regarding generalizability. The robustness of our pipeline on challenging, in-the-wild data is a key prerequisite for its applicability to demanding domains like egocentric robotics and professional film. We demonstrate how our pipeline's qualitative superiority on core challenges translates to value in these areas.

Our pipeline's ability to achieve more robust moving object segmentation and global refinement directly translates to higher-quality results on complex scenes. The full extent of this robustness is best appreciated through video visualizations. While the NeurIPS policy prohibits providing images, videos, or other rich media during the rebuttal period, we are fully committed to significantly expanding our supplementary material.

• Demo Video (suppl. - webpage/index.html): This showcases our full 4D reconstruction. Specifically, these demos include challenging cases such as significant occlusion (e.g., the "Public Bus" scene) and fast motion (e.g., the "Public Bus" and "Walking Woman" scenes). Handling such dynamic events is a fundamental challenge in both robotic perception, where tracking objects in cluttered environments is key, and in analyzing cinematic action sequences in film.
• Figures 2-4 (suppl.): These figures highlight our ability to produce clean static backgrounds and coherent shapes for dynamic objects, with minimal "ghosting" or "streaking." This level of fidelity is critical for applications ranging from creating accurate environment models for robotic navigation to generating clean plates for VFX and compositing in film.
• Figure 5 (suppl.): This figure demonstrates our superior dynamic object segmentation, showing sharper boundaries and greater temporal consistency. Such precise, temporally-stable segmentation is vital for reliable object tracking in robotics and for high-quality rotoscoping and character interaction in film production.

In summary, by demonstrating robustness on these core 4D challenges, we provide strong evidence that our DynamicGen pipeline and the resulting DynamicVerse dataset offer a powerful foundation for both general-purpose 4D understanding and specialized applications in robotics and film.

Q2: Insufficient Dataset Characterization and Documentation.

We sincerely thank the reviewer for these important suggestions on improving the dataset's characterization and documentation. We agree that providing these details is essential for community adoption and responsible research. We have prepared the following analyses and statements, which will be added to the appendix and our project website in the final version.

1. Detailed Dataset Characterization

To provide a concrete statistical analysis as requested, we have characterized a significant subset of our dataset: the 1,268 high-quality videos sourced and filtered from the original 3,471 videos in the YouTube-VIS dataset. Our analysis reveals a rich diversity of objects and actions:

• Object Distribution: This subset contains 2,158 unique moving object instances across 355 distinct categories. The distribution is characterized by a high frequency of common objects followed by a long tail of more specific categories, ensuring a mix of common and rare instances. The top 5 most frequent categories are: person (16.2%), panda (4.4%), rabbit (2.6%), bird (2.5%), and giraffe (2.2%). This demonstrates a rich variety of both human and animal subjects.
• Action & Motion Distribution: To characterize the "motion types," we analyzed the verbs from our generated captions. The distribution highlights a strong focus on dynamic interactions. The top 5 most frequent actions are: moving (276 instances), holding (178), wearing (144), walking (80), and standing (78). The full distribution covers a wide array of fine-grained actions.

2. Data Format and Accessibility Tools: To ensure our dataset is transparent and easy to use, we will provide comprehensive documentation detailing our data structure. Each scene will be organized in a self-contained directory with annotations saved in standard, easy-to-parse formats: all camera parameters (intrinsics and poses) will be stored in a single anno.npz file for efficient loading; per-frame depth maps will be saved as high-precision 16-bit .png files; object segmentation masks will be provided as a sequence of .png images; and all hierarchical text captions will be organized in a captions.json file. To further improve accessibility, we will release a suite of Python scripts for easy data parsing and visualization. Furthermore, the complete DynamicVerse dataset will be publicly hosted on the Hugging Face Hub to ensure broad access for the research community.

3. Ethical Considerations and Data Bias: We have also drafted a detailed datasheet and ethics statement that will accompany the dataset release. All videos are sourced from public platforms under permissive, non-commercial licenses (e.g., Creative Commons). We acknowledge that any dataset sourced from the internet may reflect the geographical and cultural biases of its source platforms, and we will encourage users to be aware of this. DynamicVerse is intended solely for academic and research purposes and not for surveillance or any application that may infringe on personal privacy.

We are confident that these additions will make DynamicVerse a more transparent, accessible, and responsible resource for the research community.

Q3: Unreported Inference Speed and Computational Cost.

We thank the reviewer for highlighting this important point. A clear analysis of the computational cost, including both processing time and memory requirements, is indeed crucial for assessing the practical scalability of our DynamicGen pipeline.

To provide concrete and reproducible figures, we processed all 23 videos from the Sintel training set on a single NVIDIA H20 GPU. The table below presents the average processing time and peak VRAM usage for each major module in our pipeline.

---

We believe this detailed, per-module cost analysis provides the necessary information for the community to assess the practical viability and scalability of our pipeline. We will add this breakdown to our supplementary material for full transparency.

Dear Reviewer 6prH,

Thank you for your very positive and encouraging review, and for championing our paper for acceptance.

We are particularly delighted that you found our work to be a "valuable and high-impact contribution" that "addresses a significant problem" in the community. We also truly appreciate your kind words on our "thorough experimental validation" and the paper's "clear presentation."

Thank you again for your thoughtful and supportive assessment.

DynamicVerse's authors.

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

---

Q1: Lack of DynamicVerse Dataset Validation.

We thank the reviewer for this critical feedback. To directly demonstrate our dataset's utility, we conducted new experiments which provide strong evidence for the quality and effectiveness of the geometric and semantic annotations in DynamicVerse.

1. Geometric Validation: Fine-tuning MonST3R：

To validate the quality and utility of our geometric annotations (i.e., metric-scale point maps and camera parameters), we used a subset of 1000 scenes from DynamicVerse to fine-tune the prominent model, MonST3R [1]. As shown in the table below, fine-tuning on our data leads to a performance improvement on the Sintel benchmark. This directly confirms that the geometry in DynamicVerse is accurate and beneficial for improving existing models.

2. Semantic Validation: Downstream Tasks and Direct Evaluation：

We performed three distinct experiments to validate the high quality of our hierarchical semantic annotations:

(a) Object-Level Semantics via 4D-LangSplat [2]: To demonstrate the effectiveness of our dynamic object annotations, we used them to train a 4D-LangSplat model. Our precise object masks and labels enabled the model to achieve superior results in language-grounded 4D scene understanding. This validates that our object-level annotations are highly effective for complex, multi-modal downstream tasks.

(b) Scene-Level Semantics via G-VEval [3]: We used the established G-VEval benchmark [3] to quantitatively assess our generated scene-level captions. On a random sample of 100 videos from filtered SA-V data [4], our captions achieved a high score, confirming their coherence, accuracy, and richness.

The strong performance across these metrics confirms that our captions are not only factually accurate and relevant to the video content, but also complete in their coverage of events and efficiently concise. This robust, multi-faceted quality makes them highly suitable and reliable for demanding downstream applications.

(c) Camera-Level Semantics via Human Study: We conducted a formal human study on 88 videos from our DAVIS subset to quantitatively analyze our camera motion captions. Following the criteria of Clearness, Conciseness, and Grammar & Fluency from prior work [5], the results were excellent: over 60.22% of captions were rated as both clear and fluent while also receiving high scores for conciseness, confirming the effectiveness of our generation and quality control process.

Q2: Limited Novelty, High Similarity to Uni4D.

We thank the reviewer for the comparison to Uni4D. While both works tackle 4D reconstruction, our motivation and scope are fundamentally different. Uni4D's contribution is a geometric reconstruction framework; our goal is to build a scalable pipeline for a rich, multi-modal dataset with comprehensive annotations (geometry, masks, captions, etc.). This required significant new developments.

1. Core Geometric Contributions Beyond Uni4D Our BA pipeline introduces three novel components absent in Uni4D for more robust reconstruction:

• (a) Epi-Mask-Based Dynamics Filtering: For cleaner static/dynamic separation before BA, leading to more stable pose estimation.
• (b) VLM-Based Semantic Dynamics Analysis: Uses a VLM for semantic motion understanding, enabling smarter keyframe extraction and more robust object masks.
• (c) Optical Flow-Based Sliding Window Global Refinement: A sliding-window global refinement to enforce temporal consistency and correct drift.

The following ablation study quantifies the significant, measurable impact of each component.

2. A Major Contribution Beyond Geometry: Rich Semantic Annotation Beyond geometry, our primary novelty is a sophisticated semantic annotation pipeline that sets us far apart from Uni4D. Unlike Uni4D's sparse labels, we generate three levels of rich, hierarchical captions (for objects, scenes, and camera motion). This is achieved with novel techniques like (a) Semantic-Aware Keyframe Extraction (SAKFE), (b) Hierarchical Prompting (HP), (c) Caption Rephrasing and (d) Chain-of-Thought (CoT) Prompting. Our ablation study below validates the effectiveness of each technique in improving the final caption quality.

In summary, these validated contributions in both geometry and semantics result in a more robust and powerful pipeline for creating rich 4D multi-modal data, representing a significant step forward from prior work.

Q3: Potentially Oversimplified Data due to Filtering.

Thank you for this question. Our filtering process is a necessary quality control step, not a means of simplification. Our primary goal with filtering is to ensure annotation quality by removing videos that are intractable for current SOTA foundation models (e.g., those with chaotic non-rigid motion or extreme motion blur). This prevents noisy, unusable annotations from contaminating the dataset.

Crucially, our quantitative analysis in the paper (Table 1) empirically proves that DynamicVerse retains the highest degree of dynamism compared to existing datasets, directly addressing the concern of oversimplification. Furthermore, our pipeline's modularity is a key strength; as better foundation models emerge in the future, researchers can seamlessly integrate them into our pipeline to process the very types of complex videos that are currently intractable, making our framework future-proof.

Q4: Choice of Feature Extractor.

We thank the reviewer for the question and clarify that there is a misunderstanding. VGGT is not used for our final, high-fidelity camera parameter estimation.

Its role is strictly limited to an early, fast pre-filtering step to get a rough initial estimate of intrinsics. For the final, accurate results, we use our sophisticated multi-stage dynamic Bundle Adjustment (BA) pipeline. We deliberately do not use models like VGGT for the final, critical step due to their inherent limitations, such as a lack of metric scale and the requirement for input image cropping, which discards valuable information.

---

Reference:

[1] MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion. ICLR 2025.

[2] 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models. CVPR 2025.

[3] G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o. AAAI 2025.

[4] SAM 2: Segment Anything in Images and Videos. ICLR 2025.

[5] CameraBench: Towards Understanding Camera Motions in Any Video. Arxiv 2025.

Dear Reviewer 6NBV,

Thanks again for your constructive review. To make sure we fully address your concerns on Q1: Lack of DynamicVerse Dataset Validation, we wish to supplement our quantitative experiments with the following qualitative evidence. We strongly encourage you to examine the compelling visual proof within our supplementary material:

• Demo Video (suppl. - webpage/index.html) showcases the reconstructed moving objects, dynamic scenes, and metric-scale camera trajectories. Specifically, these demos include challenging cases such as significant occlusion (e.g., the "Public Bus" scene) and fast motion (e.g., the "Public Bus" and "Walking Woman" scenes). This video evidence directly demonstrates the high quality of our dataset annotations in these very challenging scenarios.
• Figures 2-4 (suppl.) showcase our metric-scale 4D reconstruction results on several challenging in-the-wild videos. These qualitative examples highlight our method's ability to produce cleaner static backgrounds and more coherent shapes for dynamic objects, with significantly fewer artifacts like "ghosting" or "streaking" compared to baselines. These results serve as compelling evidence that our full pipeline is robust and produces high-fidelity geometric annotations even from complex, unconstrained video inputs.
• Figure 5 (suppl.) provides a direct comparison of our dynamic object segmentation against other methods. We urge the reviewer to note how our approach maintains sharper boundaries and greater temporal consistency, especially for fast-moving objects (e.g., "Sprinting" and "Car Racing Competition" scenes). Crucially, it demonstrates superior robustness to partial occlusions (e.g., "Figure skating" scene), correctly segmenting objects even when parts are momentarily hidden. This precise segmentation is the cornerstone of robust downstream 4D reconstruction.

Together, this qualitative evidence visually corroborates the strong quantitative results, confirming that our DynamicGen pipeline produces the high-fidelity geometric and semantic annotations that constitute our DynamicVerse dataset, proving its suitability for demanding downstream tasks.

Best,

DynamicVerse's authors.