GeoVideo: Introducing Geometric Regularization into Video Generation Model

Thank you very much for your time and valuable insights. We truly appreciate your recognition of certain aspects of our work—it is highly encouraging. Please find our responses to your comments below.

1.Details about the test videos. Due to the lack of high-resolution dynamic scenes in most public video datasets, we collected video data from websites such as Pexels for our experiments. It is difficult for us to list the exact source of each individual video, but we can roughly provide the distribution of the test videos as follows.

2. More qualitative results. While you may have already noticed this, we would still like to kindly remind you that, in addition to the figures shown in the main paper, we have also provided additional video results via an anonymous link in the supplementary material. Due to NeurIPS's rebuttal policy, we are not allowed to provide further visualizations at this stage, but the results in the supplementary material do include some complex scene cases.

3. Complete VBench metrics. Since many metrics in VBench are not directly related to geometric properties, we primarily showcase those that are geometry-related in the main paper. Here, we provide the complete set of metrics for reference. If necessary, we will include these metrics in the revised version of the paper.

4. Cases where geometric regularization performs worse. Such cases may exist, for example, as mentioned in our limitation section, for phenomena like fire and smoke. These objects can somehow be estimated with depth, and are therefore modeled with corresponding depth in our generation process. However, the addition of the geometric loss term $L_{geo}$ might reduce the dynamic characteristics of such videos compared to ours w/o $L_{\text{geo}}$. These non-rigid, volumetric phenomena may require more sophisticated mechanisms, which are currently beyond the scope of this study. Due to current constraints, we are unable to provide visualizations and can only describe the results verbally. We appreciate your understanding. Nevertheless, based on the average quantitative metrics across most scenarios, our geometric regularization still shows improvement for the majority of video generation scenarios.

   In most cases, ours w/o $L_{\text{geo}}$ already outperforms baseline methods. This is because it introduces additional channels with minimal interference to the original pretrained model. Even when depth is not semantically essential to the video content, modeling depth as auxiliary information can still enhance temporal consistency, as depth labels exhibit strong temporal coherence.

5. More comprehensive limitations. While our method incorporates geometric reasoning to improve the consistency of video generation, it still exhibits some limitations. First, the concept of "video" spans a wide range of visual content, and our approach struggles in scenarios where geometry is ill-defined or not meaningful—such as smoke, fire, reflections, or 2D animations—which often leads to suboptimal outputs. Second, although the geometric loss term helps enforce spatial coherence, it may also impose rigid constraints that suppress plausible appearance changes or motion patterns inconsistent with the assumed geometry. This can reduce the diversity or realism of generated results in cases that fall outside standard 3D assumptions. Lastly, the model may not generalize well to out-of-distribution inputs, particularly those involving visual phenomena that cannot be effectively captured by depth-based representations. Addressing these challenges may require more flexible or content-aware mechanisms that can selectively apply geometric priors based on scene characteristics.

Thank you very much for your time and valuable insights. We truly appreciate your recognition of certain aspects of our work—it is highly encouraging. Please find our responses to your comments below.

1. Difference from WVD. WVD simply models additional XYZ channels jointly, without introducing any specific regularization on those channels. Moreover, their approach is limited to static scenes, cannot generate dynamic videos, and does not support prompt-based video generation. And WVD does not provide open-source code, making a comprehensive comparison difficult. In our paper, we have cited this work in the Related Work section and highlighted the differences. The main contribution of our method lies in introducing geometric regularization specifically on the additional modeling channels to enhance consistency in video generation.

2. Comparison with AC3D and ViewCrafter. Both AC3D and ViewCrafter focus primarily on improving the spatial controllability of video generation. In contrast, our goal is to enable the video model itself to generate geometry-grounded videos, which is fundamentally different. Nevertheless, we can still compare the video quality of our method against these approaches, as shown below. (See the next question — we use WorldScore uniformly for comparison here.) Since these methods introduce explicit camera pose as conditional input, their models exhibit better camera controllability.

3. WorldScore Metrics. WorldScore is a recently proposed benchmark. In fact, some of the metrics we compute—such as MVCS—share the same underlying principles as certain metrics (3D Consist etc.) from WorldScore. Here, we provide a supplementary evaluation of our method on the WorldScore benchmark. If necessary, we will include these metrics in the revised version of the paper.

   Models	WorldScore-Static	WorldScore-Dynamic	Camera Ctrl	Object Ctrl	Content Align	3D Consist	Photo Consist	Style Consist	Subjective Qual	Motion Acc	Motion Mag	Motion Smooth
   ViewCrafter	45.01	11.01	88.26	48.52	58.45	78.10	69.13	66.52	59.93	2.38	1.08	17.00
   AC3D-5B	51.14	45.12	91.01	48.14	65.08	65.48	60.61	38.61	41.17	22.34	43.74	32.41
   DimensionX	59.58	47.63	93.27	39.52	36.33	82.04	82.44	81.08	61.33	63.63	25.59	54.76
   CogVideoX-T2V	54.18	48.79	40.22	51.05	68.12	68.81	64.20	42.19	44.67	25.00	47.31	36.28
   Ours-T2V	57.38	52.03	42.84	55.49	70.61	90.88	73.64	49.71	48.88	28.66	50.63	39.33
   CogVideoX-I2V	62.15	59.12	38.27	40.07	36.73	86.21	88.12	83.22	62.44	69.56	26.42	60.15
   Ours-I2V	64.57	62.03	41.02	42.41	38.45	92.11	90.15	86.10	64.38	72.02	29.10	62.85

Thank you very much for your time and valuable insights. We truly appreciate your recognition of certain aspects of our work—it is highly encouraging. Please find our responses to your comments below.

1. How geometric regularization is incorporated. At any diffusion step, we can compute the corresponding clean latent representation from the predicted noise using the predefined noise schedule. We apply our geometric regularization directly on computed clean depth latent. Because applying regularization to the clean depth latent is directly effective, and only the clean depth latent can be reliably decoded by our depth decoder into a meaningful depth map. We will provide a more detailed explanation of this mechanism in the revised version of the paper.

2. Comparison with DimensionX and ViewCrafter. Both DimensionX and ViewCrafter focus primarily on improving the spatial controllability of video generation. In contrast, our goal is to enable the video model itself to generate geometry-grounded videos, which is fundamentally different. Nevertheless, we can still compare the video quality of our method against these approaches, as shown below. (We use WorldScore [1] uniformly for comparison here.) Since these methods introduce explicit camera pose as conditional input, their models exhibit better camera controllability. If necessary, we will include these comparisions in the revised version of the paper.

   Models	WorldScore-Static	WorldScore-Dynamic	Camera Ctrl	Object Ctrl	Content Align	3D Consist	Photo Consist	Style Consist	Subjective Qual	Motion Acc	Motion Mag	Motion Smooth
   ViewCrafter	45.01	11.01	88.26	48.52	58.45	78.10	69.13	66.52	59.93	2.38	1.08	17.00
   AC3D-5B	51.14	45.12	91.01	48.14	65.08	65.48	60.61	38.61	41.17	22.34	43.74	32.41
   DimensionX	59.58	47.63	93.27	39.52	36.33	82.04	82.44	81.08	61.33	63.63	25.59	54.76
   CogVideoX-T2V	54.18	48.79	40.22	51.05	68.12	68.81	64.20	42.19	44.67	25.00	47.31	36.28
   Ours-T2V	57.38	52.03	42.84	55.49	70.61	90.88	73.64	49.71	48.88	28.66	50.63	39.33
   CogVideoX-I2V	62.15	59.12	38.27	40.07	36.73	86.21	88.12	83.22	62.44	69.56	26.42	60.15
   Ours-I2V	64.57	62.03	41.02	42.41	38.45	92.11	90.15	86.10	64.38	72.02	29.10	62.85

3. Quantitative evaluation of different prediction heads.

   3.1 Camera pose head. We report the results of our camera head, which predicts camera poses from video latents, on two datasets—CO3Dv2 and RealEstate10K—using the standard AUC@30 metric, which combines Relative Rotation Accuracy (RRA) and Relative Translation Accuracy (RTA). We also compare our results with methods such as VGGT. Since our camera head is distilled from VGGT, it achieves comparable accuracy to the original VGGT estimator.

   Methods	Re10K (unseen) AUC@30 ↑	CO3Dv2 AUC@30 ↑
   Colmap+SPSG	45.2	25.3
   DUST3R	67.7	76.7
   VGGT (Feed-Forward)	85.3	88.2
   Ours (Feed-Forward)	84.9	88.9

   3.2 Motion probability map head. Since ground truth motion probability maps are not available, it is difficult to directly assess the correctness of the predicted maps. As our motion probability map design is inspired by MegaSAM and the corresponding head is also distilled from it, we report the MSE difference between the outputs of our motion prediction head and those of MegaSAM’s corresponding head on the DyCheck and Sintel benchmarks, to demonstrate that our predictions are qualitatively similar.

   Datasets	MSE
   DyCheck	0.042
   MPI Sintel	0.056

   3.3 Depth head. Our depth head learns a mapping from depth latent to actual depthmaps. As the generated videos do not have ground truth depth, we evaluate the difference between the depth decoded by our depth head and the depth labels predicted by VideoDepthAnything on the DL3DV dataset, as VideoDepthAnything predicted labels are used as training supervision in our method.

   Heads	MSE
   Our depth head	0.0014
   Video VAE Decoder	0.0011

4. Writing errors.

   Regarding the writing issues, we will carefully revise the manuscript from beginning to end to correct such errors. We sincerely thank you for your careful attention and feedback.

[1] Duan H, Yu H X, Chen S, et al. Worldscore: A unified evaluation benchmark for world generation[J]. arXiv preprint arXiv:2504.00983, 2025.

Thank you very much for your time and valuable insights. We truly appreciate your recognition of certain aspects of our work—it is highly encouraging. Please find our responses to your comments below.

1. Robustness of geometric supervision. Yes, our approach can be influenced by the accuracy of the various prediction heads. Although components like our camera head already yield reasonably accurate predictions (see our response to Question 3), the outputs from different heads are inherently challenging to predict with perfect precision. To improve the robustness of our model, we apply geometric regularization only at locations where the difference between the reprojected depth and the generated depth is sufficiently small. We can include an ablation showing that removing this tolerance threshold leads to a degradation in generation quality, mainly due to noise from pose errors and other sources. Introducing a simple tolerance threshold proves to be an effective strategy that reveals the strength of our method.

   Method	CLIPScore ↑	FVD ↓	SC ↑	BC ↑	MS ↑	SR ↑	VISC ↑
   CogVideoX-5B (T2V)	32.30	145.3	93.8	95.1	93.2	79.4	-
   Ours w/o tolerance (T2V)	32.41	141.6	94.6	95.0	95.4	86.6	-
   Ours (T2V)	34.25	122.7	97.2	97.8	98.1	90.3	-
   CogVideoX-5B (I2V)	33.42	139.8	94.6	96.4	95.9	80.5	95.2
   Ours w/o tolerance (I2V)	33.18	136.5	95.2	96.0	95.6	87.6	95.4
   Ours (I2V)	35.02	120.5	98.1	98.5	98.6	91.1	97.6

2. Effects of motion probability maps on complex dynamic scenes. Due to NeurIPS rebuttal policy, we are not allowed to provide visualizations of the motion probability maps during the rebuttal period. As our motion probability head is distilled from MegaSAM, the predictions closely resemble those of MegaSAM. Their motion probability maps have been shown to improve pose accuracy during bundle adjustment in complex dynamic scenes. In our setup, incorporating such motion predictions also helps improve geometric consistency in generated results for complex dynamic scenarios. We believe the dynamic scenes used in our experiments already present a sufficient level of complexity to demonstrate the effectiveness of our method. Quantitative comparison results for dynamic scenes are presented in the main paper, and additional videos illustrating complex dynamic scenes are included in the supplementary material via an anonymous link.

3. Using external pose/depth priors. Yes, in fact, we have already used pretrained estimators! Our camera pose decoder is distilled from pretrained estimators (VGGT) and learns to predict poses from generated latents. Here, we add a comparison between our predicted poses from video latents and those estimated by VGGT on two datasets—CO3Dv2 and RealEstate10K—using the standard AUC@30 metric, which combines Relative Rotation Accuracy (RRA) and Relative Translation Accuracy (RTA). Since our camera head is distilled from VGGT, it achieves comparable accuracy to the original VGGT estimator. Our depth head and motion probability head are also distilled from pretrained estimators—VideoDepthAnything and MegaSAM, respectively—and their predictions achieve accuracy comparable to the original estimators. As for COLMAP, it is too time-consuming to be run jointly with video generation, making it impractical for our pipeline. Moreover, its accuracy is lower than that of the pretrained estimator VGGT we use for distillation.

   Methods	Re10K (unseen) AUC@30 ↑	CO3Dv2 AUC@30 ↑
   Colmap+SPSG	45.2	25.3
   DUST3R	67.7	76.7
   VGGT (Feed-Forward)	85.3	88.2
   Ours (Feed-Forward)	84.9	88.9

4. Comparison with WVD and UniGeo. WVD does not provide open-source code, making a comprehensive comparison difficult. Their method only performs joint modeling of additional XYZ channels, without applying any explicit regularization on them. Moreover, their approach is limited to static scenes, cannot generate dynamic videos, and does not support prompt-based video generation. We have cited their work in the Related Work section and discussed these differences. Here, we report a comparison on depth estimation metrics as presented in their paper for the NYU, BONN, and ScanNet datasets.

   UniGeo, on the other hand, is a fundamentally different type of method. It focuses on estimating geometry from existing video inputs, while our work is centered on video generation. Although UniGeo has a GitHub repository, they do not provide their model code or pretrained weights, making direct comparison difficult. We instead refer to the depth estimation metrics reported in their paper on the 7Scenes dataset. It is also worth noting that the UniGeo paper was released on arXiv on May 30, 2025—after the NeurIPS submission deadline.

   To enable a fair comparison with the above methods in terms of depth estimation, we made a specific modification to our framework: we concatenate a real video as conditional input with the noise, and let our model reconstruct both the video and the corresponding depth. In this way, our model can function to some extent as a depth estimator. However, it is important to emphasize that our method was not originally designed for geometry estimation, but rather for geometry-aware video generation. Nevertheless, as shown in the comparison tables below, our method also demonstrates a advantage in depth estimation. That said, with the help of explicit supervision, our model can also learn to predict depth from video input effectively.

   Methods	NYU-v2		BONN		ScanNet++
   	Rel ↓	δ₁.₂₅ ↑	Rel ↓	δ₁.₂₅ ↑	Rel ↓	δ₁.₀₃ ↑
   WVD	9.7	90.8	7.0	96.4	5.0	57.2
   Ours	6.1	97.2	6.8	96.7	3.2	80.6

   Methods	7Scenes
   	AbsRel ↓	δ1 ↑
   UniGeo	20.69	62.2
   Ours	18.42	71.1