Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion

Dear Reviewer,

Thank you for taking the time to provide your valuable feedback on our work. We appreciate your insights.

In response to your comments, we have made the following updates and would like to provide the following clarifications:

1. We have added video demos generated by our model, accessible at this link. Also, we included a 19-page appendix in our supplementary materials, which also features 11 pages of video samples embedded in the PDF. Due to the submission size limit, it was not feasible to integrate it into the main document.
2. Regarding the model's generalizability, we would like to clarify that our primary focus is achieving precise motion control in video generation, which is particularly important for applications such as autonomous vehicle (AV). Although our approach utilizes an off-the-shelf general-purpose model, which suggests potential adaptability to other domains with sufficient fine-tuning data, our specific interest lies within the AV domain for this particular work.
3. Regarding the comment on the missing comparisons, to the best of our knowledge, this is the only study that conducts bounding box prediction directly in the pixel space, which is one of the novel aspects of our model. To clarify, Table 2 specifically evaluates the quality of bounding box sequence predictions, which are not the generated videos. Therefore, it wouldn’t be meaningful to compare our results with trajectory-conditioned video generation methods, as the metrics in Table 2 are tailored to bounding box prediction performance.
4. Regarding the comment on the unclear metric scores, it is true that the mAP evaluations are not directly comparable, as they are conducted on different datasets and with different evaluation methods. Due to the way bounding boxes are tracked and utilized in our work, the calculation of average precision differs. The details of these evaluation differences are provided in lines 448-456. However, since they can still serve as a reference for comparison, we chose to include these numbers. Additionally, we have provided detailed explanations of these mAP evaluations in the appendix of the supplementary material (Appendix B.3 and Appendix B.4).
5. Regarding the comment on no insight provided, we acknowledge that our core idea may seem straightforward, but its novelty lies in its departure from conventional approaches. Specifically, prior work processes bounding box information as strings and extracts encodings using sequential or transformer-based models (e.g., Boximator). Moreover, existing methods overlook bounding-box trajectory predictions, they either:
   • Provide the bounding-box trajectory as input (e.g., TrackDiffusion)
   • Only focus on objects’ locations in the first and last frame; ignore intermediate frame bounding boxes (e.g., Boximator) In contrast, our approach introduces a distinct innovation: leveraging a diffusion model to predict bounding box trajectories directly in the pixel space. By bridging bounding box encoding and pixel space, our method unlocks two key advantages:
   • Scalability: It enables conditional generation of an unlimited number of bounding boxes.
   • Improved bounding box prediction: As demonstrated in Table 2, predicting bounding box trajectories in pixel space outperforms sequential methods such as Trajenglish.

     To our knowledge, this is the first work to explore bounding box trajectory prediction with diffusion models in pixel space, making our contribution novel and noteworthy.

6. Regarding the comment on our work lacking an ablation study, we would like to point out that it includes several ablations:
   • Table 1: We compare the generation quality of our model against a fine-tuned SVD model that does not condition on bounding boxes.
   • Table 2: We ablate the Box2Video model by isolating the BBox-Generator to compare its performance with Trajenglish.
   • Table 3: We evaluate the Box2Video model using teacher-forced bounding box trajectories to assess its ability to generate videos that adhere to the provided bounding box trajectories.
7. Regarding the comment on transferring to out-of-distribution prompts outside the scope of bounding boxes,, we have not tested this capability of our model, as this paper focuses solely on using bounding boxes to control object motion.

I hope these responses have addressed all your concerns. We are also ready to discuss any issues that remain unclear. If we have cleared up the confusion, we hope you will consider boosting your scores for our work.

Best regards.

The requested videos (comparisons of the fine-tuned SVD baseline vs. Ctrl-V) have been added to our anonymous repository at the bottom.

The bounding box conditioning inputs are from validation sets specified in Appendix B.3 Dataset Configuration.

Dear Reviewer,

Thank you for your valuable feedback. We appreciate your input and will revise our manuscript accordingly. Specifically, we will add figure references in the text and modify the notation in Table 1 to “Ctrl-V: BBox Generator + Box2Video”.

To further illustrate our model’s performance, we have provided video demos generated by our model, available at link. These demos offer additional insights into our model’s capabilities. And there is supplementary material with details of our method’s implementation.

Given that your concerns appear to be primarily formatting-related, we would appreciate clarification on two points:

• Are there any substantive or technical issues we should address beyond formatting?
• If we resolve all formatting issues, would this potentially influence your evaluation scores?

Thank you again for your time and expertise. Best regards.

Dear Reviewer,

Thank you for taking the time and effort to review our manuscript. We have carefully considered your feedback and revised the manuscript accordingly. Below, we provide detailed responses to address your concerns and clarify the points you highlighted.

1. Regarding the comment on video demos, we have provided a new link containing several sample clips generated by our model.

2. Regarding the notation and reference for Figure 2(Pg 5), we have updated the figure and its caption. Additionally, we have included references to the figure in the main text.

3. Regarding the missing equation in Section 3.3 (Pg 6), we have resolved the issue.

4. Regarding the definition of the scaling equations for EDM in Section 3.1 (Pg 4), we have included them in Appendix E (Pg 32).

5. Regarding the experimental setup in Section 4.2 (Pg 7), our generation pipeline consists of two models: the BBox generator and the Box2Video model. The BBox generator creates plausible bounding-box trajectories based on the provided conditioning bounding-box frames. This trajectory is then used as input to the Box2Video model to condition the video generation. In the "Teacher-Forced Box2Video Generation" scenario, instead of using the BBox generator to create bounding box trajectories, we use the ground-truth bounding-box trajectories provided by the dataset.

6. Regarding the comment on the motivation for input conditioning, our goal is to achieve precise control of generated objects by using a sequence of bounding boxes. While users can provide entire bounding box sequences, they can be hard to create. We sought to simplify this process by creating the BBox Generator model which only requires users to provide the beginning and ending frames. The numbers of initial frames used are chosen to test how our model responds with and without knowledge of the bounding boxes’ momenta (i.e. single frame bounding boxes only contain position data, using 3 initial frames allows the model to deduce speed and acceleration information etc.). As suggested by reviewer EsMH, we also included experiments where the last-frame bounding boxes are absent (Appendix C.8, Pg 28). We believe these experimental setups are crucial for testing our model’s ability to predict and control vehicle motions in AV videos.

   In response to the comment on the types of conditioning (e.g., 2D/3D bounding box frames), it is worth noting that some of the datasets we use contain 3D bounding boxes, while others do not. We utilized whichever bounding box data was available in each dataset. Additionally, to the best of our knowledge, this is the first work to explore 3D bounding box conditioning in video generation. Conditioning on 3D bounding boxes in pixel space is very straightforward, which makes this approach a novel contribution.

Dear Reviewer,

Thank you for your insightful feedback. We greatly appreciate the time you invested in our work.

In response to your comments, we:

• Rewrote Section 3 in its entirety
• Rewrote portions of Section 4
• Reworked Figure 2
• Highlighted all rebuttal changes in red
• Created a global post summarizing revisions

We kindly request your review of the revised manuscript. Please share any further suggestions.

Thank you for your detailed feedback and for taking the time to review our revision. We appreciate your acknowledgment of the improved clarity in the method section.

We understand your concerns regarding the organization of the paper and would like to assure you that we have carefully addressed all the points you raised in this revision:

1. Problem Formulation and Conditioning on Bounding Boxes: We have clearly defined the problem formulation several times in our paper:

   • Abstract (Ln 16-19):

     To achieve this, we 1) use a separate, specialized model to predict object bounding box trajectories given the past and optionally future locations of bounding boxes, and 2) generate video conditioned on these high quality trajectory predictions

   • Introduction (Ln 85-89):

     “Bounding box Motion Generations with Diffusion: We devise a novel diffusion based approach for generating 2D/3D bounding box trajectories at the pixel-level (as 2D videos) based on their initial and final states, and the first frame.“ “2D-bounding box and 3D-bounding box Conditioning: We condition on 2D or 3D Bounding boxes in order to provide a fine-grained control over the generated videos.”

   • The entire Section 3.2 (Ln 208-220):

     “1. First, we generate bounding box frames using our diffusion based bounding box predictor network, the BBox Generator, which is shown on the left side of Figure 2. These frames contain only bounding boxes. They make up a video of moving (or stationary) bounding boxes and it serve as the “skeleton” for the generated video.” “2. Then, we generate a video using our video generator network, Box2Video, shown on the right side of Figure 2, where the bounding boxes frames act as the control signal. The bounding boxes in each frame determine the objects generated in the corresponding frames of the video.

2. Bounding Box Representation Contribution: While the bounding box representation is an important aspect of our work, it is not our sole contribution. We also introduce the use of a diffusion model for predicting object trajectories, which is thoroughly discussed in the abstract, introduction, experiments, and conclusion sections.

   Additionally, we have provided a detailed explanation of the bounding box representation in the text:

   "We encode information such as track ID, object type, and orientation for each bounding box using a combination of visual attributes, including border color, fill color, and markings. Specifically, the track ID represents a unique identifier for each tracked object across frames, the object type specifies the category of the object (e.g., car, pedestrian), and the orientation indicates the direction the object is facing."

   Details about how these bounding box frames are rendered can be found in Appendix B.1. We chose to include this information in the appendix because space in the main paper is limited, and these details, while helpful, are non-essential to our main contributions. Moreover, we provide multiple visualizations throughout the main paper to ensure clarity (Figures 1, 2, 3).

3. Baseline Model Explanation: The Stable Video Diffusion (SVD) baseline model was first introduced in detail in the preliminary Section 3.1 (Ln 175-204):

   "In this section, we provide an overview of the Stable Video Diffusion (SVD) (Blattmann et al., 2023a;b) model, due to its importance in our approach. SVD is an image-to-video (I2V) model that employs video diffusion… "

   We have added a new footnote about the baseline model on Line 375-377.

4. We have fixed the typos in Section 3.1.

We understand your view that significant revisions were needed; however, we believe these changes and clarifications have thoroughly addressed your concerns. We hope the revisions meet your expectations.

Dear Reviewer,

Thank you for reviewing our paper and providing constructive feedback. We appreciate your time and effort. We have revised our manuscript. Below, we'd like to address your concerns and clarify the points you've raised.

1. Regarding the suggestions for an ablation study, we conduct additional experiments (Appendix C.8, Pg 28) to investigate the effect of conditioning various bounding-box frames on Ctrl-V's video generation capabilities. Notably, we assess the Ctrl-V performance with:

   • No final bounding-box frame
   • Replacement of the final bounding-box frame with a trajectory frame, where central points represent bounding boxes

   Key findings from our ablation study include: a) Complete ground-truth bounding-box trajectory conditioning yields the best generation quality; b) Even minimal conditioning with the final frame’s bounding boxes’ central points substantially enhances quality over the no-final-frame baseline. Additional details, findings, and illustrations supporting this study are available in the supplementary material (Pg 28-30).

2. Regarding the concern on the visualization of occluding objects and new object appearance, we supplement our analysis with Figure 25 in Section E.2 (Pg 33). Figure 25 illustrates the model's robustness in handling:

   • New object entry: car and pedestrian correctly assigned to bounding boxes
   • Occlusion: cars temporarily obscured during traffic crossover
   • Reappearance: The car re-emerges with preserved appearance after occlusion during crossover

   Moreover, in Section C.6 (Pg 25-26), we examine the model's performance in turning scenarios, marked by abrupt changes in viewpoint, resulting in many instances of new object emergence.

3. Regarding the suggestion of the discussion of the failure cases, we present a new Section E.1 (Pg 32), which delves into the limitations of our model in capturing fine visual details and the challenges of resolution degradation.

4. Regarding the concern of the computation overhead, our pipeline was trained and evaluated on a single A100-80GB GPU. A detailed account of the model's architecture and parameter count is provided in Section A.5 (Pg 16).

5. Regarding the concerns about handling additional control information, our current work focuses specifically on controlling object motion in video generation via bounding boxes. However, our foundation model, an SVD network, is inherently flexible and can accommodate text and other types of conditioning. Future integration of other conditioning is feasible but falls outside the scope of this study.

Best regards.