Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos

We appreciate your comment that our proposed method for dynamic novel-view synthesis is "simple" and the "test-time finetuning is important". We also show "results on various types of scenes (real, synthetic, with/without scene dynamics)". We address your concerns below.

Q1: Consistent multi-view generation

Every time CogNVS is run with a new camera trajectory, it would result in a different plausible output. Having said that, we observe that test-time finetuning does increase the consistency of the inpainted regions with respect to the content of the visible scene. If 3D multi-view consistency across samples is desired, we suggest the following: CogNVS should be used iteratively by inferring and fusing the point clouds from all previous novel-view generations. This is a fairly straightforward and common strategy adopted by many generative models [R1, R2, R3]. We have verified this on our end by generating a panorama trajectory in chunks for an outdoor scene which looks spatiotemporally consistent. Unfortunately, visuals cannot be uploaded as part of this rebuttal.

[R1] Text2room: Extracting textured 3d meshes from 2d text-to-image models, ICCV23.
[R2] Wonderjourney: Going from anywhere to everywhere, CVPR24.
[R3] 4DiM: Controlling space and time with diffusion models, ICLR25.

Q2: Performance on DyCheck before test-time finetuning

We include the performance on DyCheck before test-time finetuning below. For completeness, we also include a comparison on Kubric 4D and ParallelDomain 4D. Across all datasets, the takeaway is that test-time finetuning benefits all metrics by adapting the weights of CogNVS to suit the test scene’s appearance, texture, lighting and 3D cues. This is one of our primary contributions in this work.

Table A. Test-time finetuning improves performances on DyCheck.

Table B. Test-time finetuning improves performances on Kubric-4D.

Table C. Test-time finetuning improves performances on ParallelDomain-4D.

Q3: Duration of test-time finetuning

CogNVS takes 70/140 mins for test-time finetuning on short/long videos and another 5 mins for final rendering. A detailed analysis of average using 8 A6000 GPUs for our method and all baselines on DyCheck evaluation set with 400-frame sequences is included below. In general, our optimization is faster than some test-time optimization approaches with the additional benefit of being able to inpaint/hallucinate unknown regions in a spatiotemporally consistent manner. Additionally, our inference is on par with other feed-forward methods.

Table D. Run time analysis of ours and baselines.

Note that per the graph in Fig. 8 (left) in supplement, our method achieves 96% of its performance in just half the number of finetuning steps. This duration can always be traded-off with the required compute. This has recently been referred to as “inference-time scaling” for diffusion models [R4].

[R4] Inference-time scaling for diffusion models beyond scaling denoising steps, arXiv 2501.09732.

Q4: Effect of occlusion / defocus in novel-views on mask

During mask creation, we do not explicitly hallucinate new occluders in the novel view to reject more points from the point cloud during reprojection. However, this is not a bottleneck for training, or for inference, as the random nature of our training mask creation, in expectation, covers cases where an occluder would be visible in the novel view. Even if this was not the case, the inference from CogNVS would be unaffected as all that CogNVS does is pixel-level inpainting given a set of masked pixels. The resulting inpainted region can hallucinate new objects which act as occluders in the novel view. Note that all occlusion scenarios except “containment” are covered in the generated training data. As you notice correctly, for the scope of the paper, we only model a pinhole camera as the methods we use for preprocessing (e.g., MegaSAM) are also trained for a pinhole camera assumption. We will gladly add this discussion to the main paper.

Q5: Dependent on quality of structure-from-motion

This is correct. Since our method takes as input a partial rendering of the input scene from the novel-view, its performance will increase with better structure-from-motion approaches which can give us better depth maps, camera poses and intrinsics. As you notice correctly, our generations are better when groundtruth depth and cameras are available, e.g., in Kubric 4D.

Q6: Improved writing

Thanks for your suggestion. g_src only contains information about the scene from the source view visible in V_src (i.e., not full 3D). Line 122 in fact does not have a typo and it should be V_src because we use the source video as groundtruth during finetuning. Nevertheless, we will make the writing more clear as suggested.

Q7: Contrast is reduced after finetuning

This can happen in some cases when the scene is out-of-distribution (e.g. the textured ground plane in this case) and finetuning is not enough for adaptation. We will highlight this as a limitation of our method more clearly in the main text.

Q8: Cameras inside objects

Our current setup does not allow generating cameras inside objects because the training datasets we consider do not cover these cases.

We appreciate your comment that we propose a "simple pipeline" yielding "good results". We address your concerns below.

Q1: Claiming inpainting as a contribution

Thank you for your comment. We want to highlight that we do not claim "warping then inpainting" as a contribution (L52-55). In fact, we sufficiently cite other works which employ a similar inpainting strategy (c.f. MegaScenes, NVS-Solver, Gen3C, TrajectoryCrafter). We apologize for the oversight on Text2Room. We will gladly cite it and clarify these details further in the manuscript.

Q2: Multimodal generations lack 3D consistency

It is correct that every time CogNVS is run with a new camera trajectory, it would result in a different plausible output. Having said that, we observe that test-time finetuning does increase the consistency of the inpainted regions with respect to the content of the visible scene. If 3D multi-view consistency across samples is desired, we suggest the following: CogNVS should be used iteratively by inferring and fusing the point clouds from all previous novel-view generations. This is a fairly straightforward and common strategy adopted by many generative models [R1, R2, R3]. We have verified this on our end by generating a panorama trajectory in chunks for an outdoor scene which looks spatiotemporally consistent. Unfortunately, visuals cannot be uploaded as part of this rebuttal.

[R1] Text2room: Extracting textured 3d meshes from 2d text-to-image models, ICCV23.
[R2] Wonderjourney: Going from anywhere to everywhere, CVPR24.
[R3] 4DiM: Controlling space and time with diffusion models, ICLR25.

Q3: Limited technical contribution

As elucidated in the main paper, we would like to reiterate that our main technical contributions are, (1) training a dynamic novel-view synthesis method purely on 2D videos, and (2) self-supervised finetuning with input videos at test time using our data generation strategy. These help our method achieve state-of-the-art performance on dynamic novel-view synthesis. Reviewer N7JL in their review regards these contributions as "key differences" that are methodologically novel as compared to prior works, and to the best of our knowledge, these contributions have not appeared in the literature before.

TrajectoryCrafter is one similar and concurrent work (to be published at ICCV in October). While both TrajectoryCrafter and CogNVS finetune a video generation model to an inpainting model, TrajectoryCrafter’s contribution is a “multiview” model that requires the input view as a conditioning. Our architecture is simpler as it simply inpaints in. Because of this we are able to finetune CogNVS at test-time, and it is impossible to do so with TrajectoryCrafter as you cannot have access to the groundtruth novel view at test-time. Empirically, this test-time finetuning enabled by our architecture design manifests in improved performance metrics. See below for quantitative comparison.

Q4: Primary source of performance improvement

Please note that as stated above, our primary source of improvement comes from the last stage of training – self-supervised finetuning of CogNVS at test time on a test video. For example, on Kubric 4D, TrajectoryCrafter’s PSNR↑/FID↓ is 20.9/130.2. Our pretrained CogNVS without test-time finetuning (with a similar CogVideoX backbone, warping strategy and groundtruth point cloud as TrajectoryCrafter) achieves a PSNR↑/FID↓ of 20.1/114.0, which is on par with the performance of TrajectoryCrafter out of the box. Upon test-time finetuning, CogNVS achieves a PSNR↑/FID↓ of 22.6/102.5, which surpasses the performance of TrajectoryCrafter. As we cover in the main text, test-time finetuning helps CogNVS adjust to the target scene of interest, and adapt the weights to suit the scene’s appearance, texture, lighting and 3D cues.

Q5: Mask creation design

The underlying motivation for the warp-back strategy is indeed the same as TrajectoryCrafter and [1]. On another note, two distinctions of our work with respect to TrajectoryCrafter (as touched upon in Q3) are that, (1) we train CogNVS purely on 2D videos, and (2) use a self-supervised test-time finetuning phase to adapt CogNVS to the test video, which is not possible to do with TrajectoryCrafter's architecture design. Finally, we want to point out that TrajectoryCrafter will be published at ICCV in October and therefore, should be regarded as concurrent work.

Q6: Would finetuning cause overfitting?

Yes, overfitting to the target scene is the intention with test-time finetuning. This is the same concept by which NeRFs or Gaussian Splats are overfit to a single scene at test-time. In practice, we do not find that this overfitting is detrimental to novel-view synthesis in any way. Regardless, the duration of test-time finetuning can always be adjusted to favor generating a more diverse scene at the cost of lower adherence to the target scene.

We appreciate your comment that our data generation strategy is "well-designed", our proposed self-supervised finetuning of test videos is a "key difference methodologically" and that our method’s "results look very good qualitatively, and better than the previous state-of-the-art". We address your concerns below.

Q1: Test-time finetuning with TrajectoryCrafter

We understand the similarity between the data generation strategy of our method and TrajectoryCrafter, but we want to clarify that unlike TrajectoryCrafter we train our model with only 2D videos. While both methods finetune a video generation model to an inpainting model, TrajectoryCrafter’s contribution is a “multiview” model that requires the input view as a conditioning. Our architecture is simpler as it simply inpaints in. Because of this we are able to finetune CogNVS at test-time, and it is impossible to do so with TrajectoryCrafter as you cannot have access to the groundtruth novel view at test-time. Finally, we want to point out that TrajectoryCrafter will be published at ICCV in October and therefore, should be regarded as concurrent work.

Q2: Runtime analysis with baselines

Your analysis of the runtime is correct – CogNVS takes 70/140 mins for test-time finetuning on short/long videos. However, per the graph in Fig. 8 (left) in supplement, note that our method achieves 96% of its performance in just half the number of finetuning steps. This duration can always be traded-off with the required compute. This has recently been referred to as “inference-time scaling” for diffusion models [R4].

Nevertheless, we provide a more detailed analysis of optimization and rendering/inference runtime in comparison to baselines below. These are average runtimes for the DyCheck evaluation set. We run finetuning on 8 A6000 GPUs, and for a 400-frame sequence, our method takes ~2 hrs for test-time finetuning and another 5 mins for final rendering. In general, our optimization is faster than some test-time optimization approaches with the additional benefit of being able to inpaint/hallucinate unknown regions in a spatiotemporally consistent manner. Additionally, our inference is on par with other feed-forward methods.

Table A. Run time analysis of ours and baselines.

[R4] Inference-time scaling for diffusion models beyond scaling denoising steps, arXiv 2501.09732.

Q3: VAE encoding of missing pixels

Yes, we simply treat missing regions as zeros in the image-space. While there are other ways to encode missing pixels, e.g. providing explicit masks as you suggest (which TrajectoryCrafter also uses), it is not clear what advantage this brings. For instance, we find that TrajectoryCrafter is unable to inpaint unknown regions in some Kubric 4D examples even though they have been explicitly marked as unknown in their input masks. We instead take a data-driven approach – we rely on the model to infer where and how to inpaint missing regions based on context.

The latent space of the VAE can capture semantic information, including the context provided by neighboring pixels. Therefore, black pixels or missing pixels (even though both are represented by 0s) should have different latent embeddings. Similarly, 8x8 patches with few or many missing pixels should have different latent embeddings. In practice, we empirically observe that such patches are handled well by CogNVS without explicit masking.

We appreciate your comment that we solve an "important" task of dynamic novel-view synthesis by "avoiding costly 3D geometric data", building a pipeline that is "easier to scale up" and "improves temporal consistency and 3D plausibility over the baselines". We address your concerns below.

Q1: Distorted foreground objects

This is correct. The distortion that is visible in in-the-wild examples (e.g. left leg of a man hiking) is because of the distortion in the MegaSAM outputs. We will include visualization of renders from MegaSAM that are input to CogNVS to make this clear. Additionally, we note that there is no such distortion in the Kubric 4D objects as ground truth metric depth is available in this case.

Q2: Maximum rotation angle

As we show in Kubric 4D evaluation and some in-the-wild examples, CogNVS can plausibly generate novel views with up to 90 degrees of variation (up, down, left, or right). We also attempted to generate panorama views of scenes (outward facing 360 degree view) by dividing the camera trajectory into 49-frame chunks and progressively generating the new views conditioned on the past generations – a fairly straightforward and common strategy adopted by many generative models [R1, R2, R3]. Qualitative analysis suggests that CogNVS can still generate 3D plausible and temporally coherent novel-views even in such extreme cases. Unfortunately, visuals cannot be uploaded as part of this rebuttal but we have verified this on our end.

Additionally, we want to point out that our method is not able to generate table top views of object-centric scenes (inward facing 360 degree view) beyond 90 degrees of camera deviation, likely because such training data was not seen by the model during pretraining.

[R1] Text2room: Extracting textured 3d meshes from 2d text-to-image models, ICCV23.
[R2] Wonderjourney: Going from anywhere to everywhere, CVPR24.
[R3] 4DiM: Controlling space and time with diffusion models, ICLR25.

Thanks for the detailed response that addresses my concerns. Therefore I raise my score. Please add the promised visualizations and discussions in the final version.