Training-free Camera Control for Video Generation

Thank you for your meticulous review and insightful comments of our work.

We start with some explainations about the weakness:

W1: Camera motion should keep the scene intact and should not change the arrangement.

A1: Yes, scenes should be kept intact when camera moves. Simultaneously, the other goal we want to achieve is to have more dynamics in the generated videos, just like what we record through our phones in the real world. And along with the camera movements, the arrangment of scenes can be changed. In this setting, the whole scene can be reconstructed from video by some 4D reconstruction algorithms.

W2: Inaccurate camera motion.

A2: We follow the terminology in cinematography to describe different camera movements, where the definition of "pan" refers to "rotating the camera horizontally from a fixed position", and the motion that moves horizontally is described as "truck". The definitions of camera motion are presented in Line 178 and Line 765 of the paper.

W3: Missing computational analysis. The approach seems very expensive.

A3: We add the computational analysis into the revised version of our paper in Table. 2 and Line 375-412 (the updated parts are highlighed in blue). We test the GPU memory and time requires in inference process for different methods, the results are shown as below:

Table2. Computational Analysis

The evaluations are conducted under unified settings for all approaches. As a training-free method, our method requires no extra GPU memory during inference process compared with base model. This makes it save 10-20GB memory than other finetuned methods under the same circumstances, and makes it applicable running on a single RTX 3090 GPU. The major consumption of our method comes from the time rendering multi-view images. Even so, the time is acceptable as the other two methods also need considerable time.

W4: Missing quantitative ablations.

A4: We add two more quantitative ablations in the revised version of paper: 1) Ablations to vanilla SVD. We evaluate the quantitative performance of vanilla SVD to demonstrate the effectiveness of our approach in camera motion control. 2) Ablations on t_0. We conduct quantitative experiments with different t_0 to show how it affect output videos in motion accuracy.

W5: Writing quality in methodology part.

A5: Sorry for the confusion. When writing the paper, we also felt some parts are difficult to describe, especially the illustration about latents' casuality and error-resilience towards generated outputs. We will use some grammar tools to refine them.

W6: Unclear motivation of generalizability.

A6: The generalizability refers to camera control ability of videos from other domains. We add comparison results of generalizability in Fig. 4 and Line 366 - 373 in the revised version of paper. Since both MotionCtrl and CameraCtrl are finetuned on limited scenes (i.e., real estate videos) with static content, they can hardly generalize to other scenes, such as videos in non-realistic style or videos that include people. They fail to offer accurate camera control, nor can they generate content with dynamic movements. In comparison, CamTrol preserves most of the prior knowledge from video base models, enabling it to handle general scenes as well as generate dynamic content. We put these results in the updated supplementary materials, in the end of the qualitative comparison file named "rebuttal_1_qualitative_comparisons_new".

Secondly, we will answer the questions as follows:

Q1: Why does the scene often change with camera movement? Why is the camera often not that accurate even for simple movements?

A1: The reason for this two issues is because we use smaller t_0. If we set t_0 to a larger value, the camera motion will be more accurate and the scene will be more consistent, but it will lead to loss of video dynamics. Setting smaller t_0 will allow the generated videos with more dynamics, while sacrificing some accuracy in motion control. However, these motions remain close to the original trajectory without significant deviation.

Methods that are finetuned on camera-annotated datasets have higher accuracy in motion control, but they cannot handle scenes outside the training domain (like MotionCtrl and CameraCtrl can only handle scenes of real estates), neither can they deal with dynamic contents. When users input a camera motion, such as "Zoom In" into our method, we assume they prefer a well-generated, dynamic, realistic video, even the generated motion has little diviation from given one, it still looks natural. Instead, a video that follows exactly the input motion, but with limited scenes or static objects may not be preferred.

Q2: This question refers to the same issue with W3.

We thank reviewer 14Er for checking the results in the supplementary materials and for the response.

Please allow us to explain further about the concerns:

1. The issue of scene consistency, and also the problem of motion degradation compared with base models.

For most video foundation models, their training datasets (e.g., Webvid, UCF-101) don't contain videos with such quick, severe camera movements like what our method produces. Referring to our anonymous demo page, we'll find that many videos have completely different scenes in merely 16 frames with camera moving. Some of the results in 3D part even have movements as drastic as SV3D can produce, and SV3D is a 21-frame model finetuned on 3D data. We assume this is the reason that degradations can occur, as we want to let base models produce quick, severe camera moving videos, where they originally only produce videos with moderate camera motion. This problem could be solved if we had a base model that can originally produce drastic moves, or had a annotated dataset that contains generic, diverse scenes to finetune base model. Yet we have neither of these two in our community.

2. The dynamic control of finetuned methods using limited datasets (MotionCtrl and CameraCtrl) and of CamTrol.

We consider the domain collapse issue of MotionCtrl and CameraCtrl not come from the operation of "finetune". If we have the proper datasets, finetuning generally can give the best performance compared with other heuristic methods. The problem is that current camera-annotated datasets only contain very limited scenes, i.e., real estate videos with static content. As generative models are basically mapping functions from one distribution to another, the original output distribution of video base model will shrink to a smaller, contrained distrbution after finetuning on these datasets. For our approach, it does not change the output distribution of video base models (as it only uses the inference process of them), but only gives them a guidance of how output videos should be formed (i.e., in certain layouts). This allows it to perserve more prior knowledge inherited from foundation models (such as the generalization of scenes and dynamics of generated content) compared with methods finetuned on mentioned datasets.

We had some examples in Fig.4 of the revised paper to show the performance of MotionCtrl and CameraCtrl dealing with generic and dynamic scenes. We also updated some extra comparisons in the supplementary materials, in a new html file named "second_rebuttal_dynamic_comparisons.html". We need to clarify that the open-sourced SVD model itself does not perform very well in generating dynamic content (it exhibits mostly static videos, which could be referred to through: https://huggingface.co/stabilityai/stable-video-diffusion-img2vid) and has limited capacity generating videos with drastic camera move, thus all four methods based on SVD (including SVD itself) struggle to produce consistent, perfect results, but our method performs better than others in generating dynamic content and generalizing to more common scenes. Applying to a more powerful, dynamic base model, it will show more dynamics in generated videos. We provide more dynamic videos produced by CamTrol in the newly added file named "second_rebuttal_more_dynamics". There are also high-dynamic results presented in our anonymous demo web page. These files contain many camera-moving videos with quite high dynamics in their generated content. We believe that no approach, to our knowledge, can generate such high-dynamic video for generic scenes (e.g., include human, animals, natural scenes, even 3D objects from 3D datasets) with camera control.

We thank reviewer 14Er for raising the score.

We arranged the results the same way that reviewer 14Er recommended in our anonymous demo page, i.e., camera is shared across many prompts to show that one camera works across prompts for different dynamics, especially in the "hybrid and complex trajectories" and "3D rotation-like generation" parts, and these results are all complex combinations of rotations and translations. There are 40+ videos presented in these two parts on the anonymous demo page, all with complex (combinations of rotations and translations, which reviewer 14Er recommended), dynamic moves, arranged in the mentioned way (one camera works across prompts for different dynamics).

For the benefit over previous works, we recommend reviewer 14Er to check the demos produced by these methods, MotionCtrl: https://wzhouxiff.github.io/projects/MotionCtrl/ and CameraCtrl: https://hehao13.github.io/projects-CameraCtrl/, to judge whether our method produce more dynamic results than them. Most videos produced by MotionCtrl and CameraCtrl are very static, as can be seen from the exhibited results on their websites. If one claims that demos on webpage do not fully show the performance of methods, we also include comparions under the same circumstances in the supplementary material, to demonstrate their weakness in handling generic, dynamic scenes. We strongly recommend reviewer 14Er to check previous methods' websites for a judgement that whether our method has benefit over them, in both training cost and generating performance.

Best regards,

Authors

We thank reviewer 14Er for the response.

Reviewer 14Er might expect a bigger step in the motion quality problem to justify this as a contribution, however, particularly dynamic video generation is not what our method aims for. The purpose of our method is to provide a training-free (don't need finetuning to run) , low-cost (low in GPU memory) tool that can be directly adept to video diffusion models (it can work on most video diffusion models) to control video's camera motion. The generalizability and dynamics, in which aspects our method has better performance than others, are not initial purpose our method is proposed, but side-advantages of how our method utilizes the prior knowledge in video models. We also disagree with the saying that this task has been investigated already, methods like MotionCtrl and CameraCtrl have limited capacity which reviewer 14Er also agreed, what they mainly did was adding some layers to inject camera trajectory, then used camera-annotated datasets to finetune video model. Our method is the first one to achieve camera control on video diffusion models that does not require any finetuning and rely on video model's extensive knowlegde rather than finetuning for generating camera-moving videos.

Reviewer 14Er also said it's not clear what existing issue has been significantly improved compared to MotionCtrl and CameraCtrl. We recommend reviewer 14Er to think about what our method can achieve while MotionCtrl and CameraCtrl can't, for which we can claim many: training-free and do not require finetuning, directly work on almost any video diffusion model, have more generalizability in generic scenes, low-gpu costs... . We also have some 3D generating results on the anonymous demo page, in the "3d rotation-like generation", where we generate camera-moving videos on 3D data from Omniobject3D, unsupervisedly. We believe previous methods like MotionCtrl and CameraCtrl cannot produce these kinds of results due to their limited capacity, and even 3D generative models like SV3D need finetuning on 3D dataset to produce these results. These are all benefits our method has over previous methods.

Dear Reviewer 14Er,

We have some new results updated in this comment, which may be helpful to the argument of visual quality:

I do not see a big improvement in the visual results.

About the visual quality, we did conduct a user study among different methods, we did not put it in the paper. We feel necessary to put it here:

(Details of this user study: We collect 7 text prompt for 8 basic camera trajectories (zoom in/out, pedestal up/down, truck left/right, roll clockwise/anticlockwise), yield 56 samples for each method to generate. Users are asked to sort the four videos displayed randomly generated by four methods, and rate them from the highest score 3 to the lowest 1 (the higher the better). We calculated the average score attained by each method, forming the table up above. We have 17 people took part in the user study.)

The results of user study prove that the visual quality of our method is acknowlegded by most users. If reviewer 14Er still feel not convinced by the results given by user study, there are also other reviewers who have positively recognized the generated quality of our work: Reviewer syyx stated that "the experiments and demo show good results", reviewer 4R7b commented that our method "outperforms competing methods in both perceptual quality and motion alignment, particularly in complex camera movements", and reviewer WcRk said our results "addresses the concerns of dynamics".

I agree that MotionCtrl and CameraCtrl are very static due to training on RealEstate10K but the scenes shown are also rather static.

We kindly ask reviewer 14Er to provide some concrete examples for this, i.e., the results of our method that are rather static compared with MotionCtrl and CameraCtrl.

The results look somewhat similar to previous works from one year/half a year ago.

On their websites, we did not see MotionCtrl and CameraCtrl produce results with

1. such diverse scenes with dynamic content, including human with drastic moves (e.g., playing drums, walking, playing guitar), dynamic generic scenes (e.g., street with racing cars, burning flames), moving animals (e.g. walking, jumping). We haven't seen previous works like MotionCtrl and CameraCtrl generating these kinds of results.

2. rotation videos for 3D data, which are exhibited in the "3d rotation-like generation" part of our anonymous webpage, here we generate 3D rotation videos unsupervisedly using a single rendering image of object from OmniObject3D. In fact, we believe no method before has achieved unsupervised, training-free 3D generation merely from video models.

We hope these updated results as well as the upper claims could help address the issue of visual quality as well as the novelty compared with previous works, and we look forward to further reply from reviewer 14Er.

Best regards,

Authors

We appreciate reviewer for the meticulous review and insightful questions.

We will answer the questions as follows:

Q1: Can you clarify the details in quantitative evaluations? It seems there doesn't have a uniform experimental setup in camera control video genration.

A1: Yes, there doesn't have a uniform experimental setting in this domain. In our experiments, we randomly sampled 500 videos from RealEstate10k dataset as reference, and use their prompt-trajectory pairs for generation. In the previous version of paper, we used the default video base model for each method, i.e., VideoCrafter for MotionCtrl and AnimateDiff for CameraCtrl. We've changed this setting by using SVD as base model for every method.

SVD is originally trained on resolution of 576 × 1024, but SVD-based CameraCtrl only support 320 × 576. Since changing the original resolution leads to suboptimal generation quality, we use 576 × 1024 for MotionCtrl and CamTrol, then resize their outputs into 320 × 576 for calculating metrics. For all methods, the number of frames and the decoding size of SVD are set to 14. We use 25 steps for both inversion and generation processes. We updated the detailed setting for quanlitative comparisons in the revised versin of our paper, starting from Line 304 (the revised parts are highlighted in blue). We also updated the newest quantitative results in Table.1. We also provide the video comparisons in the supplementary materials, in an html named "rebuttal_1_qualitative_comparisons_new".

Q2: Not any qualitative comparisons with AnimateDiff.

A2: In the previous version of our paper, we have qualitative results of AnimateDiff in Fig.3. Since we unified the video base models and conduct re-evaluations after, this figure is also altered. But we kept AnimateDiff's comparison in the supplementary, which we uploaded along with the paper submission. For better comparison, we made it into an html file named "2. comparisons_old". The qualitative results of AnimateDiff are shown here.

We exclude the comparisons with AnimateDiff in the revised paper because: 1) AnimateDiff itself is a video base model, and the quality of base model affect heavily the results of camera-control videos, we can't compare with it fairly when other methods applying SVD as base model; 2) AnimateDiff cannot handle complex trajectories, and the scale of its basic trajectory is unknown. Thus, even evaluated under basic trajectories, the motion accuracy cannot be calculated as there is no ground truth.

Q3: Comparisons with Camco and Vd3d.

A4: Both Camco and Vd3d are not open-sourced, I'm afraid we can't conduct comparisons with these methods.

Q4: Generation results under superior video foundation models.

A4: Yes. During the experiments, we find that combining with better foundation models, CamTrol can generate videos with more consistent frames and more rational dynamics. We provide results of CogVideoX-5b+CamTrol in the updated supplementary materials (In the paper we used CogVideoX-2b), with an html file named "rebuttal_3_CogVideoX_results". CogVideoX-5b is trained at 8 fps, 720x480 resolution with 49 frames. Note that we exhibit the raw outputs of the model, without any post-processing algorithms like super-resolution or frame interpolation. These results, compared with SVD, are more alike to real-world videos with camera movements.

We've also tried the latest mochi model by genmo, but it seems its basic outputs are not ideal even without any camera control.

Secondly, please let us explain about the weakness:

W1: Different stages of pipeline may cause inconsistent, and ultimately compromise the output quality.

E1: We presented in Fig. 6 that the failure of inpainting and point cloud reconstruction possesses little damage to the output quality. For diffusion models, the operation of adding and removing noise have a effect of purifying imperfect inputs, as both perfect and imperfect inputs share the nearly-gaussian latent distribution. Similar ideas are wildly used in previous papers (SDEdit [1], Ilvr[2]). Thus, the performance in previous stages (depth estimation, inpainting, point cloud reconstruction) are of highly tolerance in our method, since they do not affect ultimate generations but only serve as layout guidance which indicates perspective changes.

W2: Experimental part was poor. Qualitative comparisons are not shown on demo page.

E2: We updated the revised version of our paper with unified and detailed experimental settings. Please check them for more detailed experimental settings and analysis.

The qualitative comparisons with state-of-the-art methods are attached in the supplementary materials, in parallel with the demo web page, we kept it in the revised supplementaries for further review.

[1] Sdedit: Guided image synthesis and editing with stochastic differential equations

[2] Ilvr: Conditioning method for denoising diffusion probabilistic models

We thank reviewer syyx for improving the score, we really appreciate it.

Best regards,

Authors

Thanks for your meticulous review, and for your positive comments on our work.

We'll answer the questions above:

Q1: Base models in Table.1.

A1: We used the default model of each method, i.e., VideoCrafter for MotionCtrl and AnimateDiff for CameraCtrl. We've corrected this unsuitable setting and re-evaluate all methods based on the same SVD models. The results are updated in Table.1 in the revised version of our paper (the revised parts are highlighted in blue). We also provide video comparisons in the updated supplementary materials, in an html file named "rebuttal_1_qualitative_comparisons_new".

Table. 1 Quantitative Comparisons

Q2: Can we get some videos with similar appearance but with different trajectories? Comparison with CVD.

A2: Yes! This is a great application of our method to generate multi-trajectory videos for one scene. We've added this into the revised version of our paper, in Fig.9 and Line 517-521. We also provide some video results for multi-trajectory generation in the supplementary materials. For better visualization, we made it into an html file named "rebuttal_2_CVD-like_results". You can click it to see some CVD-like results.

However, as CVD is not open-sourced yet, I'm afraid we cannot conduct comparisons with it.

Q3: How find the best trade-off of t_0?

A3: In previous works (SdEdit [1], DiffusionCLIP [2]), similar early-stopped inversion steps are often manually set. Since inference of image/video generation could be done within minutes, it is fast and convenient to produce several outcomes and compare them. We offered some recommended choices of t_0 for different kinds of moves in Line 269, they are empirical findings, but there’s one rule that can be referred: Motions with drastic moves prefer larger t_0, as larger t_0 leads to less noise thus more relevance between adjacent frames of latents. For moderate moves, latents are relevant to each other in large spatial regions, which is easier for video model to recognize, thus t_0 of moderate camera motion doesn’t have to be very large.

Secondly, please let us explain about the weakness:

W1: This weakness refers to the same issue with Q1.

W2: Inpainting models cannot handle some situations well, leading to inconsistency or unreasonable objects.

E2: Inpainting models can fail in some situations, but these failure doesn't have severe impact on the generated videos. We present this point in Fig. 6, to show that the imperfections of inpainting possess little damage to the output. For diffusion models, the operation of adding and removing noise have a effect of purifying imperfect inputs, as both perfect and imperfect inputs share the nearly-gaussian latent distribution. This makes our method highly tolerant of the inpainting model's performance.

W3: CamTrol may have some difficulties in properly using pretrained video models to generate consistent and reasonable videos.

E3: Utilizing pretrained video models to generate camera-control videos relies heavily on the performance of pretrained models. For most open-sourced video models, their ability for generating consistent and reasonable scenes is not sufficiently enough, even without any camera control (e.g. SVD). During the experiment, we find that applying CamTrol on better video generation models lead to more consistent and reasonable videos. We put the results of combining CamTrol with CogVideoX-5b in the updated supplementary materials, in an html file named "rebuttal_3_CogVideoX_results". Using these models, CamTrol can generate more videos with more reasonable content, consistent scenes and dynamic movements.

[1] Sdedit: Guided image synthesis and editing with stochastic differential equations

[2] Diffusionclip: Text-guided diffusion models for robust image manipulation

Thanks for your review and for your questions, we will answer them one by one:

Q1: Lack of novelty.

A1: Infinite Nature uses camera-annotated dataset to train a network for generating novel views, and its dataset only contains images of aerial coastlines.

Our method doesn't need any additional network, and doesn't need training on limited datasets that have camera pose labels. More importantly, methods trained on such datasets, e.g., MotionCtrl, CameraCtrl and Infinite Nature, can only handle camera control of certain scenes. For Infinite Nature, it can only deal with images depicting aerial coastlines. Nor can they handle the generation of dynamic content, as these datasets only contain static scenes.

Q2: Optimization is time-comsuming, preventing it from practical applications.

A2: We tested the time consumption of our method. The optimzation process takes about 0.42s for one frame at 576x320. For a video with 14 frames, the optimization process only takes about 6s. We offer more detailed computational analysis in the revised version of our paper in Table 2 (the revised parts are highlighted in blue). Our method saves about 10-20 GB GPU memory than other finetuned method under the same circumstances, and can be applied on a single RTX 3090 GPU.

Table2. Computational Analysis

Q4: No design for encouraging dynamics in video.

A4: In Line 261 - 269, we discussed how to encourage generated videos with more dynamics or more alignment with input trajectories. In Fig.7, we illustrated how the choiice of t_0 effect the output in dynamics and motion alignment. We also updated our paper with a more detailed quantitative evaluation of different t_0s.

Q5: t_0 is challenging to choose.

A5: Similar early-stopped inversion steps are often manually set in previous works (SdEdit [1], DiffusionCLIP [2]). Unique value of t_0 may not be easily defined, but multiple choices provide users with more flexibility to decide whether they want videos with more dynamics or align better with given trajectories. Inference of video generation can be conducted within minutes, and it is fast to compare results with different t_0. We offered some empirical choices for t_0 in Line 269.

Q6: How it tackle text-to-video tasks?

A6: In Line 163, we explained that image could be created by image generators like Stable Diffusion. In Line 751, we clarified the image generation models we use for handling text input.

We considered adding this into Fig.1, but excluded it to keep the diagram’s layout intact.

[1] Sdedit: Guided image synthesis and editing with stochastic differential equations

[2] Diffusionclip: Text-guided diffusion models for robust image manipulation

We thank reviewer WcRk for the clarification and for the response.

Yes, for generating dynamic results, we 1) added some comparisons in Fig.4 of the revised version of paper, to demonstrate our method's ability in tackling more generic, dynamic scenes, while other finetuned methods fail to do so; 2) added more video examples in supplementary materials, html file named "second_rebuttal_dynamic_comparisons", to exhibit how different methods perform on generating videos that contain dynamic content, basd on SVD. Note that the open-source SVD itself does not perform very well in generating dynamic content (most of its examples are videos with static scenes, which could be referred to through https://huggingface.co/stabilityai/stable-video-diffusion-img2vid), thus all four methods based on SVD (including SVD itself) struggle to produce highly dynamic results, but still our method performs better than others in generating videos with dynamic content. 3) We added more generating samples of our method based on SOTA video generative model CogVideo-X in the supplementary materials, in the html file named "rebuttal_3_CogVideoX_results", where the results have more dynamics than SVD while align well to the input camera motion. 4) We provide even more dynamic videos generated by CamTrol in the html file "second_rebuttal_more_dynamics", supplementary materials, which have considerable movements of the generated content. There are also high-dynamic videos presented in the anonymous demo page (which can also be referred to in the supplementary materials, file "1. anonymous_demo_page"). These files contain many camera-moving videos with quite high dynamics in their generated content, and can stand as proofs that our method is capable of handling dynamic video generation with camera control.

Thanks for your careful review and meticulous comments for our work.

We will answer the questions mentioned above:

Q1: Evaluation Details.

A1: Yes, the evaluation dataset in Table.1 is RealEstate10k. We used text prompt from MotionCtrl for video generation, these prompts are not extracted from RealEstate10k and might cause domain gap. We assume this is the reason why our method surpasses them even they've been finetuned on RealEstate10k.

Q2: Base model of MotionCtrl and CameraCtrl in Table.1

A2: We used their default setting of base model, i.e., VideoCrafter for MotionCtrl and AnimateDiff for CameraCtrl. We've corrected this setting and unified the video base model to SVD for all methods. Following your advice, we also include the video quality of vanilla SVD as a reference. The results are shown in the table below, we've also updated it into the revised version of our paper (the revised parts are highlighted in blue). We also provide video comparisons in the updated supplementary materials, in an html file named "rebuttal_1_qualitative_comparisons_new".

Table. 1 Quantitative Comparisons

Q3: As ParticleSFM may fail, motion metrics shoule be provided with lower bounds.

A3: Yes, we should include the scores evaluated on the ground truth videos as lower bounds of these motion metrics. We've integrated these results into the table above, and also in the revised paper.

Q4: Same trajectory may cause different moves, and can author provide quantitative comparisons on complex trajectories?

A4: The results in Table.1 are tested under complex trajectories. We randomly sampled 500 trajectories from RealEstate10k. The reason that same trajectories may cause different moves is we used different t_0. If we set t_0 to bigger value, all results will have the same movements that align to the input trajectory, but this will also lead to loss of video dynamics. Setting smaller t_0 will allow the generated videos with more dynamic, while generally follow the input camera motions. As MotionCtrl and CameraCtrl are finetuned on camera-annotated dataset, they have less output inconsistency for the same trajectory, but this only works on static, real estate-like scenes with non-dynamic content.

Secondly, please let us explain shortly about the wearkness:

W1: Each of the current step will have a bad effect on the next step.

E1: The operation of adding and removing noise make diffusion model capable to deal with imperfect inputs, as both perfect and imperfect inputs share the nearly-gaussian latent distribution. Similar ideas are wildly used in previous papers (SDEdit [1], Ilvr [2]). Thus, the performance of depth estimation and inpainting are of highly tolerance in our method, since they only serve as layout guidance which indicates perspective changes. We presented some examples in Fig.5 to show that their inferior performance possesses little damage to the generation.

W2: Different videos may have different t_0.

E1: Yes, similar early-stopped inversion steps are often manually set in previous works (SdEdit [1], DiffusionCLIP [3]). We offer some empirical choices for t_0 in Line 269 of the paper.

W3: Inpainting may lead to inconsistency.

E3: We carefully designed the inpainting process and depth optimization algorithm to improve consistency between adjacent views, we present this part in Line 184 – Line 190 of the paper.

W4: Latents can carry unfavorable semantics for video generation, making this method unsuitable for generating long videos.

E4: Semantic and layout information are attached, it is the layout of semantic information in noisy latents that help diffusion model to identify what to represent in certain positions and how these content moves. We majorly rely on inpainting model instead of video model to complete scenes and generate new contents. After inversion, the semantic information of new content will be encoded into noisy latents. This process can be repeatedly performed regardless of number of frames.

[1] Sdedit: Guided image synthesis and editing with stochastic differential equations

[2] Ilvr: Conditioning method for denoising diffusion probabilistic models

[3] Diffusionclip: Text-guided diffusion models for robust image manipulation

The author's rebuttal has addressed my main concerns; however, I still find the proposed pipeline is complex and lacking scalability. I tend to recommend acceptance, but with a borderline rating.

We thank reviewer 4R7b for the positive review of our work.

For complexity, the proposed pipeline doesn't require extra carefully-designed network; for scalability, the proposed pipeline can be generalized to most video diffusion models without finetuning, and can inherit scalability from base models (e.g., scene diversity, generation capability). We assume the proposed pipeline, in some cases, has advantages in these two terms, compared with finetuned methods.

Best regards,

Authors