LLM-grounded Video Diffusion Models

We really appreciate your encouraging review and suggestions! Here we address each point of your comments:

Even though the proposed method is training free, it takes longer time during inference to generate videos due to the optimization steps needed for the energy function.

Thanks for the great point! All previous methods based on backward guidance for training-free guidance (e.g., LMD, BoxDiff, Layout Guidance, Self-Guidance, and our method) take more inference time. The reason is that such methods need to find a way to leverage the knowledge inside the pretrained model during inference, without supervision from any external box-video pairs.

However, training-free methods not only are free of training costs (time and compute requirements) but also have no requirements on external data or human annotations (e.g., box annotations on frames). For example, our method can be applied off-the-shelf on video diffusion models such as ModelScope without external annotations. Moreover, training-free allows exploring a wide range of formulations without training them each time.

Furthermore, even though inference time is usually not the main focus of this line of work (e.g., LMD, LayoutGPT, BoxDiff, etc.), if low inference time is needed for the production environment, our method can be distilled into a conditioned generation model so that the inference-time optimization of energy function is no longer required. Specifically, the generation of our method can potentially be distilled into a model with additional input tokens (e.g., ReCo) or attention adapters (e.g., GLIGEN), thus achieving fast generation without external datasets or additional human annotations. This way, we get the advantages of being both free of external annotated data and having fast inference, enjoying the benefits of existing training-free and training-based methods. We leave the distillation to future work as this is a general enhancement applicable to many training-free guidance methods.

Training-free layout control already exists in previous literatures [1, 2]. Therefore, the design of the energy function and backward guidance is not that novel.

The energy function proposed in our pre-rebuttal manuscript, which applies spatial control per frame, is indeed a little similar to the energy function used in previous works. However, as explained in the General Rebuttal $Q1$ , we proposed a temporal energy term that provides guidance on the positions and velocities of center-of-mass of each object, which significantly improves our generation accuracy, especially on the temporal tasks. Since previous layout-to-image methods only focus on generating one image (i.e., do not take temporal consistency into accounts), we believe our introduction of temporal center-of-mass energy term sets our method further apart from previous image layout conditioning methods mentioned by the reviewer. We will update the manuscript accordingly. Thanks for the good point!

Ablation study of the model design is not given (e.g., number of DSL guidance steps, energy function design).

Thank you for the suggestion. We have conducted ablation studies on the CoM energy function design as well as the number of DSL guidance steps in LVD. Please refer to the General Rebuttal [Question 4] for more details.

Could the authors provide some reasoning why they report video-text similarity metric in Bain et al., 2021? It would be nice to also report CLIPScore, since its widely reported in other text-to-video generation baseilnes.

Good question! CLIPSIM, the video-text alignment score obtained from CLIP, is computed per-frame and then averaged across the feames (reference). This leads to its temporal ambiguity. For example, it could not distinguish a barrel floating from the left to the right vs from the right to the left. Instead, we use Frozen-In-Time, a CLIP-like model that takes in the whole video sequence and outputs a video-text alignment, which does not treat a video as a bag of frames and can potentially model object motion in the alignment score.

Per your request, we also evaluate our model and report the CLIPSIM metric compared to the baseline, following a similar setting as Tab. 4 (Tab. 5 in the updated PDF) except we change the alignment model to a per-frame CLIP.

Our method gives better CLIPSIM compared to baseline ModelScope, indicating better text-to-video alignment.

However, we would like to mention that previous benchmarks, such as FVD or similarity-based metrics (e.g., metrics based on CLIP or Bain et al., 2021), do not perform detailed checks for correctness in aspects such as spatial dynamics and generative numeracy. While these methods mainly focus on the semantic alignment and generation quality, our proposed benchmark in Tab. 2 pioneers in quantitatively assessing the correctness in prompt understanding in video generation, opening up a direction to explore future video generation research.

The examples provided in the paper are with non-overlapping bounding boxes. Will the proposed method work well with overlapping layouts?

Yes. We have added an example with overlapping layouts in Figure 8(b) in the paper appendix (Section A.1 in our updated PDF). Generally we find that the LVD also works with overlapping boxes. Our energy function, with its topk operator, simply selects the most appropriate place to encourage object appearance, and two layout boxes can select different places due to the different topk selection. Therefore, two or more overlapping boxes will not conflict with each other.

If there are multiple objects, is the final energy function summing over the energy function corresponding to each object?

You are right! The final energy function is the sum over the one for each object. We will make it clear in the manuscript.

It seems that the background areas of the generated images with proposed method are quite static (Fig1, 7, 8, 9). Is this because the model encourages static background, or becuase the visualized images happens to have relatively static background?

Good point! Empirically we found that our base model, ModelScope, often tends to generate videos with small background variations, potentially because it uses a lot of static images in training. To ensure that our model retains the ability to generate non-static background, we let it generate a bear taking a shower in a lake, lightning strikes as a simple example, which we show in Figure 8(c) in the paper appendix (Sec A.1 in the updated PDF). Although the background motion is not extreme, the model indeed generates rain dropping on the water and the lightning, showing that our method does not impede the ability to generate non-static background.

Based on my understanding, another concurrent work, VideoDirectorGPT [1], is also for text-to-video generation with layout guidance. Even though the technical routes are different from this paper, it would be nice to have some discussions and comparison in the related work section.

Thanks for the suggestion! We have included a reference to [1] and updated a comparison in the related work section in the paper. Since this work was released on arXiv only a few days before the submission deadline, [1] is a concurrent work that should not be used to devalue our work.

Our work has a different focus compared to [1]:

1. Our work is training-free and focuses on inference-time algorithms, whereas [1] trains adapters with human annotated images.
2. Our work focuses more on in-depth analysis on the capabilities of LLMs to generalize to unseen properties and to use knowledge in the weights rather than mimick in-context examples.

In light of the completion of several requested experiments and comparisons, we would like to kindly ask if there are any other concerns or additional questions that we can respond to or if you are willing to consider increasing your score!

Really appreciate for your detailed explanations to address my concern about CLIPSIM and overlaping bounding boxes!

But I still have concern about the reasoning of static background you mentioned above. ModelScope is also trained on webvid dataset, and a lot of videos in webvid have obvious camera motions. So I'm not very convinced by the statement "ModelScope often tends to generate videos with small background variations".

In the additional example you generated (Fig 8(c)), it seems that the background is still very static, even though there are lightings. But the lightings can be regarded as "foreground" objects rather than the background. Therefore, it would be nice if the authors can provide some examples with obvious background motion.

And another minor question is about the resolution of videos generated from ModelScopeT2V. As described in their technical report, their model is trained and evaluated on resolution of 256*256. In LVD, the videos are of resolution 512*512. Do the authors observe some negative visual quality change after increasing the resolution?

Dear reviewer 34cZ,

We appreciate your positive and insightful feedback on our work! We would like to check again to see if you've had the opportunity to review our responses to your additional questions. We've conducted additional experiments to generate videos with background motion that arises from camera motion according to your request.

In light of our updates to our work addressing your concerns, completion of many requested experiments, and discussions about related work mentioned in the review, we would like to ask if you are willing to reconsider your score, and also if there are any additional questions that we can respond to during the last day of the author-reviewer discussion period!

Thank you so much!

Best Regards,

Authors

From Table 3, we can see that LVD improves the video quality. What causes the improvement?

Our method indeed improves on the FVD scores on both UCF-101 and MSR-VTT datasets over the base video diffusion models without additional training, as shown in Table 3.

FVD measures the alignment between the generated videos and reference videos, which in turn reflects both video-text alignment and video quality. Two factors together contributed to our improvements in the FVD score:

1. Our method, with an LLM processing input text prompts to generate dynamic scene layouts, has improved ability to understand the input text prompt. This is reflected in the improved video-text alignment.
2. Although improving video quality is not our main goal, our method also improves the quality of generated videos by using the intermediate dynamic scene layouts for planning, which in turn leads to better attribute binding and spatial layouts that are more reasonable.

For example, in the first row of Fig. 1 in our paper, baseline ModelScope confuses the semantics of the bear and the pikachu mentioned in the prompt and generates an object that looks neither like a bear or like a pikachu. By giving spatial-temporal guidance to the video diffusion model, our method can successfully generate a bear and a pikachu with high visual quality, as required by the prompt. The ability for high-quality video generation in cases like this leads to our improved FVD score.

Is LLM able to generate reliable layouts using text without direction information, such as “a walking dog”.

That is a great point! As your requested, we ask the LLM to generate a walking dog without specifying the direction information, as shown in Figure 8(c) in the appendix of the updated paper PDF. We observe that the LLM is not only able to generate dynamic scene layouts for a walking dog but also provide assumptions that it makes about the walking directions in the reasoning statement. This shows that our LLM is indeed able to make reasonable guesses for ambiguous user prompts.

Reasoning: A dog walking would have a smooth, continuous motion across the frame. **Assuming it is walking from left to right**, its x-coordinate should gradually increase. The dog's y-coordinate should remain relatively consistent, with minor fluctuations to represent the natural motion of walking. The size of the bounding box should remain constant since the dog's size doesn't change.

If the assumption does not meet the user's expectation (e.g., the user wants a dog walking from the right instead), the user can reply to the LLM, and the LLM can incorporate additional information to satisfy the user's needs.

Given the successful generation of the dynamic scene layouts, our layout-conditioned video generation method also successfully generates a video of a walking dog. We have added this example in the appendix of our updated manuscript. Please see Sec A.1 and Fig. 8 (a) in our updated PDF for the video generation. Thanks again for the suggestion!

In light of this, we would like to kindly ask if there is anything to explain to make you consider raising your score. Additionally, we are open to addressing any additional concerns or answering any questions you may have!

Thank you for your encouraging review! Here we address your comments individually:

The idea of using LLM to generate layout is already explored in LayoutGPT (Feng et al., 2023) and LMD (Lian et al., 2023).

The first layout generation part is similar to LMD, and the second part is a straightforward application of other training-free layout-to-image methods in the video domain.

Thanks for the comment. However, we would like to point out a few differences in terms of findings and the proposed methods between our work and LMD.

As for the LLM-grounded dynamic scene layout generation stage, **there are two noteworthy distinctions between our work and previous works, as explained in detail in the General Rebuttal $Q2$ **.

1. Most of text-to-image-layout generation tasks introduced in previous works (LayoutGPT and LMD) can be completed without in-depth understanding of the properties of the objects in the prompt. In contrast, even a simple generation of dynamic scene layouts of "a falling object" requires understanding several world properties, object properties, and camera properties, which is highly non-trivial for an LLM and not tested by previous works.
2. More importantly, previous works either only evaluate properties that are already seen in in-context examples (e.g., the 4 main tasks that LMD showcases are already demonstrated in the in-context examples in Table 10 of the LMD paper), or retrieve related layouts from a layout database as in-context examples (LayoutGPT). Neither work demonstrates whether LLMs have the knowledge for generalization without relevant in-context examples. In contrast, in Section 3 (Discoveries), we show that LLMs can generalize to physical properties or object properties that are unseen in the in-context examples (e.g., inferring the property of a paper airplane, while similar concepts are never mentioned in the in-context examples). This implies LLMs have an intrinsic understanding of the physical world instead of just emulating from in-context examples.

As for the dynamic scene layout to video generation stage, as explained in the General Rebuttal $Q1$ , we propose a temporal energy term that provides guidance on the positions and velocities of center-of-mass of each object, which significantly improves our generation accuracy, especially on the temporal tasks. Since previous layout-to-image methods only focus on generating one image (i.e., do not take temporal consistency into accounts), we believe our introduction of temporal center-of-mass energy term sets our method further apart from previous image layout conditioning methods that you mentioned. We have updated our paper PDF to include the descriptions to this energy term.

We hope the reviewer can agree with us on these significant distinctions that are either not applicable to previous LLM-grounded text-to-image works or are not demonstrated by them. We will update the manuscript accordingly to express these distinctions more clearly.

Dear ZTWf,

We appreciate your positive feedback on our work! We’re reaching out to see if you've had the opportunity to review our responses to your concerns about the novelty and contribution of our work, improvements in video quality, and clarification of ambiguous prompts. We've conducted additional experiments and generated new videos based on your suggestions.

Could you kindly let us know if our responses have adequately addressed your comments? If there is any additional information or experiments that you believe would be beneficial, please don't hesitate to let us know. We're more than willing to provide further details during the author-reviewer discussion.

Thank you so much!

Authors

Thanks for your detailed review and highly-valuable suggestions! Accordingly, we improved our approach with a temporal energy function, added additional studies on a benchmark of a much larger scale, conducted key ablation studies to analyze our method in response to your comments. We address each of your comments in detail:

Since they are dealing with video generation using layouts, it would have been nice to see the authors' contribution in extending to the temporal axis, but this is not evident, which is disappointing.

Thanks for the suggestion! Indeed, although our method already greatly outperformed the baseline with the current energy function, innovation on temporal axis should further boost the temporal consistency of the generated objects. Therefore, **as introduced in the General Rebuttal $Q1$ **, we added a temporal center-of-mass term, which significantly improves our generation accuracy by 10%, especially on the temporal dynamics and sequential movements, while reducing the mismatches between the generated objects and the dynamic scene layout input. Our introduction of center-of-mass energy term sets our method further apart from previous image layout conditioning methods that do not deal with the temporal axis. We hope this novel energy term meets your expectation for novelty related to the temporal axis! Thanks for your suggestion again!

I am concerned that the scale of the sample size for the proposed DSL benchmark may be too small to conduct a sufficiently robust evaluation.

This point is totally valid! Taking your suggestions, we propose an enlarged version of our benchmark that is 10x larger compared to the previous version. Specifically, each task now has 5x more prompts in a similar structure as the previous version, with 500 prompts in total. For each prompt, we generate two videos from dynamic scene layout, with 1000 videos per benchmark run in total.

We run both the baseline and our method twice with the enlarged benchmark and we observe evaluation results with small overall variation across the runs.

Furthermore, our enlarged benchmark still results in very similar results compared to the previous smaller benchmark, which also validates the findings obtained with the previous benchmark. We hope the robustness in our benchmark results can address your concerns!

Finally, we would like to note that while previous benchmarks (e.g., CLIPSim and FVD) focus on generation quality and semantics, our benchmark is a pioneer for correctness in spatial-temporal alignment in the literature, which is of great significance for future research, especially given that the results justify the robustness of our benchmark.

For the final version of the paper, we plan to run all our ablation studies on this enlarged benchmark and include the results in our work.

The paper's contribution appears to lack novelty. There is existing work in text-to-image generation that has already established the capability of LLMs to create layouts, and this research seems to merely extend that to assess whether the same capability applies to temporal understanding. I didn't perceive any novel ideas stemming from the temporal aspect of the problem that would distinguish this work significantly from its predecessors.

Thanks for the comment. There may be some misconceptions on the capabilities justified by previous works that use LLMs for image layout generation. **There are two noteworthy distinctions between our work and previous works, which we briefly explain here, and we provide more detailed clarifications in the General Rebuttal $Q2$ **.

1. Most of text-to-image-layout generation tasks introduced in previous works (LayoutGPT and LMD) can be completed without understanding the properties of the objects in the prompt in depth. In contrast, even a simple generation of dynamic scene layouts of "a falling ball" requires understanding of complex properties of the physical world, such as physical property (e.g., gravity which decides the trajectory of the ball), object interaction (e.g., ball hitting the ground), object properties (e.g., elasticity of ball which decides how much the ball will bounce), and camera properties (e.g., perspective geometry which decides if the size of the ball will change based on its distance to the camera), which are all highly non-trivial for an LLM and have not been assessed, demonstrated, or evaluated by previous works. In contrast, we show that LLMs demonstrate the understanding of these properties, as illustrated in Figure 4 in the paper.
2. Previous works either only evaluate properties that are already seen in in-context examples (e.g., the 4 main tasks that LMD showcases are already demonstrated in the in-context examples in Table 6 of the LMD paper), or retrieve related layouts from a layout database as in-context examples (LayoutGPT). Neither work demonstrates whether LLMs have the knowledge for generalization without relevant in-context examples. In contrast, in Section 3 (Discoveries part), we show that LLMs can generalize to physical properties or object properties that are unseen in the in-context examples (e.g., as shown in Figure 4 (c) in the paper, when generating layouts of throwing a paper airplane, it can figure out the airplane will slide in the air due to its weight and aerodynamics, while similar concepts are never mentioned in the in-context examples). This implies LLMs have an intrinsic understanding of the physical world instead of just emulating from in-context examples.

We will clarify these two points further in our manuscript. Thanks again for the valuable feedback!

The paper seems to lack a detailed analysis or ablation studies concerning the prompts given to the LLM for generating Dynamic Scene Layouts (DSLs). Such investigations are crucial to understand how different prompts affect the LLM's output and the subsequent quality of the video generation. Further exploration in this area could significantly bolster the robustness of the presented approach.

Thank you for the suggestion! We have added ablation studies on how the number, content, and format of prompted in-context examples will affect the generation quality. Please refer to the General Rebuttal $Q3$ , in which we show that 3 in-context examples already result in high-quality dynamic scene layout generation and LLM is robust to different in-context examples in both content and format.

The paper's current framework could indeed benefit from additional ablation studies or analytical experiments to demonstrate the effectiveness of using DSLs for training-free guidance of text-to-video diffusion models.

Thank you for the comment. To analyze different design choices of the DSL-to-video stage in LVD, we conduct ablation studies on the CoM energy function design as well as the number of guidance steps in the DSL-grounded video generation pipieline. Please refer to the General Rebuttal $Q4$ for more details.

Can the authors elaborate on how the model performs with ambiguous or complex text prompts that might yield multiple valid interpretations in terms of spatial and temporal dynamics?

That is a good point! The potential ambiguity in the user-provided prompts is indeed worth exploring. We find that LLMs can oftentimes perform reasonable guesses if the spatial-temporal information is missing in the user-given prompt.

For example, with a simple prompt as a walking dog, the LLM gives the following reasoning statement, with the assumption clearly written in it:

Reasoning: A dog walking would have a smooth, continuous motion across the frame. **Assuming it is walking from left to right**, its x-coordinate should gradually increase. The dog's y-coordinate should remain relatively consistent, with minor fluctuations to represent the natural motion of walking. The size of the bounding box should remain constant since the dog's size doesn't change.

This allows the subsequent layout generation to be consistent with respect to the assumption of walking from the left to the right. We show the generated video of this example in Figure 8(a) in appendix of the updated paper PDF.

Additionally, if the assumption does not meet the user's expectation (e.g., the user wants a dog walking from the right instead), the user can reply to the LLM, and the LLM will incorporate additional information to satisfy the user's needs.

Could the authors discuss any observed limitations or failure modes of the LVD approach, particularly in cases where the LLM might generate less accurate DSLs?

Thank you for the suggestion. We show two failure cases in the DSL-to-video stage in Figure 10 and Figure 11 of the updated paper PDF. For the text-to-DSL stage, although the LLM-generated DSLs have a high accuracy in following the prompt (98% accuracy in Table 1 in the paper), it still has some failure cases. The failures mainly come from generating complex dynamics such as sequential movements. One failure case, which we added in the updated paper PDF, is shown as below and also demonstrated in Figure 9 in Section A.6 of the paper appendix:

prompt: A realistic lively video of a top-down viewed scene in which a ball initially on the lower left of the scene. It first moves to the upper left of the scene and then moves to the upper right of the scene, outdoor background.

generated boxes (visualized in Figure 9 of the updated paper PDF):

Frame 0: (0.097, 0.878, 0.195, 0.976) 
Frame 1: (0.097, 0.683, 0.195, 0.781) 
Frame 2: (0.097, 0.488, 0.195, 0.585)
Frame 3: (0.292, 0.488, 0.390, 0.585) 
Frame 4: (0.488, 0.488, 0.585, 0.585) 
Frame 5: (0.683, 0.488, 0.781, 0.585)

Note that for each frame, the box is represented by 4 float numbers, where the first two are the x and y coordinates of the upper left corner of the box, and the last two are the x and y coordinates of the lower right corner of the box.

In this case, although the prompt indicates the ball moves to the upper left corner of the scene, the LLM does not generate the box in the upper left corner, but instead a box slightly above the middle of the whole scene. The failure is probably because the sequential movements contain multiple stages and LLM has a higher probability to oversee or misunderstand one of the stages. We will add these discussions on the failure cases to the paper.

(Minor point) Thank you for pointing out this typographical error in Section 4, second paragraph. The sentence in question should indeed conclude with "feature map" instead of "feature ma." A revision is recommended for accuracy and clarity.

Thanks for the good point! We have fixed this in our manuscript.

In light of the additional improvements to our methods and benchmarks we've provided, as well as the completion of numerous requested experiments, we would like to kindly ask if you might be open to increasing your assessment score, and if there are any new concerns or further questions that we can address for you!

$Q1$ Differences with previous layout-conditioned image generation methods / Adding guidance to the temporal axis

One common concern (Reviewer 6SgA, 34cZ, ZTWf) is the lack of differences with previous layout-conditioned image generation methods such as Layout Guidance (Chen et al. 2023a) and BoxDiff (Xie et al., 2023) since the pre-rebuttal version of our work only leverages a spatial energy term applied to each frame. To resolve this concern, we added an additional temporal energy function to specifically provide guidance across the temporal axis (i.e., across the frames).

To extend the current LVD energy function to the temporal axis and utilize the temporal information, we propose an additional CoM (Center of Mass) energy term which promotes the position and motion consistency between temporal dynamics of generated objects and layouts, which leads to benefits especially for tasks that require strong temporal understanding abilities.

Specifically, the proposed energy function first calculates the CoM of the cross-attention map as well as the bounding box mask, and then penalizes the difference between the position and velocity of two CoMs. In this way, the generated object can better follow the spatial position and temporal dynamics of the generated boxes.

In the table below, we report the performance of LVD with CoM energy term, and compare to vanilla LVD without CoM energy term. To further benchmark the alignment between the generated objects and the input scene layout, we also add an additional metric which measures the average distance between the center of each generated object and its corresponding input bounding box, termed "Generation Mismatch".

We can see that the CoM energy term, as a temporal term, significantly improves LVD's performance, especially for object dynamics and sequential movements that are at the core of text-to-video generation. It also generates objects that better follow the layouts, as measured by box distance. The CoM energy function improves the average accuracy by 10% over the vanilla LVD and greatly reduces the generation mismatch. We have updated our paper PDF to include this new energy term in our method section.

The intuition behind this is that the topk function used by both our original energy term and previous methods mostly cares about whether an object is generated somewhere in the box. However, the exact position is loosely constrained.

This is usually not a problem for image generation, since a slight deviation for object positions within an image is usually tolerable. However, in a video generation setting, the topk function may ask the model to generate an object on different areas of the input box in different frames, especially during the first few denoising steps when the cross-attention maps are noisy and are only loosely-correlated across the temporal axis. This could mislead the video diffusion model to believe the two objects in different frames are disjoint, thus applying the wrong temporal prior, making the energy function difficult to optimize, and finally causing misalignment with the input text prompt. In contrast, our center-of-mass energy term applies a precise control of both object positions and velocities, which is vital for the video diffusion model to correctly apply its temporal prior to generate videos that align with the layout boxes, thereby improving the alignment between the input condition and the actual generation.

Finally, we implement the energy functions from previous text-to-image work Layout Guidance [1] and BoxDiff [2] applied on the same video diffusion model for fair comparison. We observe that our formulations, especially with the CoM energy term, greatly outperform previous works.

Our center-of-mass energy term deals with the guidance across the temporal axis and significantly improves the generation alignment, which adds novelty to our work and further distinguishes our work from previous layout-conditioned text-to-image methods that do not deal with the temporal dimension.

$Q3$ Ablation studies on LLMs' capability of generating DSLs from text

Reviewers (6SgA, 34cZ) are concerned that ablation studies on LLM's capability of generating DSLs from text are lacking. Especially, Reviewer 6SgA points out that an analysis of the prompts and in-context examples given to LLM is needed. Here we conduct ablation studies on how the number, the content, and the format of in-context examples can affect LLM's capability of generating DSLs. We will add the results in the paper.

We first test the effect of different numbers of in-context examples. We test with 1, 3, and 5 in-context examples, respectively, and report the performance of each on our proposed benchmark. Results are shown below. We can see that using only 1 in-context prompts can degrade the average accuracy by 13%. This is likely because the diversity of in-context examples is not enough for sufficiently informing the LLM about the task formulation. However, we find that the performance of 3 and 5 examples are approximately the same. This indicates that 3 examples is enough to enable LLMs to generalize to many unseen test cases, implying that the capability of understanding the spatial and temporal properties of the physical world does not come from exhaustive in-context examples, but comes from the pretrained LLM itself. Therefore, by default we use 3 in-context examples in the paper. We will add this ablation to the paper to strengthen our results.

We also test if LVD is robust to different in-context examples. We have a set of 5 in-context examples and randomly sample 3 examples to prompt LLM. We repeat this test for 3 times, and report the performance of each test. The results are shown below. We can see that the variation across different tests is about 2%, which indicates the robustness of our method to different in-context examples. We will update the results in the paper.

Finally, we test whether LVD is robust to different prompting formats for the in-context examples and outputs. Specifically, rather than the using mostly natural language for representing the examples and outputs (in Table 8 of our work), we let the LLM process in-context examples in a JSON format and directly output a JSON response to represent the dynamic scene layouts. We run this ablation on our benchmark and show negligible variations of the generation accuracy. This indicates that our use of LLM for dynamic scene layout generation does not depend on a particular form of layout representation format in prompting.

$Q4$ Ablation studies on the algorithm for generating videos from the DSLs

Reviewers (6SgA, 34cZ) are concerned that ablation studies on DSL-grounded video generation algorithm are lacking. To analyze the effect of different design choices in the algorithm, we conduct ablation studies on the energy function as well as the number of DSL guidance steps, respectively. We will update the results in the paper.

First, to analyze the effect of the proposed CoM energy function, we compare video generation results with or without this function. Detailed results are shown in General Rebuttal $Q1$ . We find that the proposed energy function greatly boosts the performance of video generation, especially in temporal tasks, validating this design choice.

Second, we ablate on the number of DSL guidance steps between two denoising steps in LVD. We test 1, 3, 5, and 7 steps of DSL guidance, respectively. The results are shown below. We can see that the average accuracy improves when adding more DSL guidance steps from 1 to 5 steps, and the performance plateaus when we use more than 5 steps. This means 5 steps is enough and probably the best choice considering the tradeoff between performance and efficiency. We will also add this result to our paper.