Real-Time Video Generation with Pyramid Attention Broadcast

We sincerely thank the reviewer 4GeG for the valuable questions and comments. For the concerns and questions, here are our responses:

Q1: In Table 1, the PAB range is different for different models. Besides, the diffusion timesteps in the Appendix are different. Does this mean that these hyperparameters need to be determined experimentally?

A1: Thanks for the comment. Indeed the hyperparameters need to be determined experimentally, but it's very simple and easy.

There are only two hyperparameters in PAB: 1) broadcast range. 2) the beginning and end steps of the broadcast.

1. Broadcast range

Experiment settings: We conduct the following experiment to find out how to determine the best broadcast for all methods, where PAB$_{\alpha\beta\gamma}$ denotes broadcast ranges of spatial ($\alpha$), temporal ($\beta$), and cross ($\gamma$) attentions. The italicized results mean significant performance degradation.

Analysis: As shown in the table, we can discover the following findings:

• When we increase broadcast range from PAB$\_{222}$ to PAB$\_{246}$ or PAB$\_{235}$, there is not much performance degradation, as the redundancy degree of temporal and cross attention is larger. So broadcast range should follow: cross range > temporal range > spatial range.
• When extending to PAB$\_{257}$ or PAB$\_{333}$, there will be severe performance degradation.
• Although different models have different schedulers, timesteps, and training strategies, their best broadcast ranges are similar. The optimal range is either PAB$\_{246}$ or PAB$\_{235}$ considering both effect and efficiency.

2. The beginning and end steps of broadcast

Experiment settings: We conduct experiments to find out how to determine the best start and end steps for all methods, where $[\cdot, \cdot]$ denotes the end and start diffusion step of broadcast. The more steps it covers, the more speedup PAB can achieve. The italicized results mean significant performance degradation.

Analysis: As shown in the table, we can discover the following findings:

• Reducing broadcast steps to $[150,750]$ or $[200,700]$ does not increase worthy performance, but leading to less speedup. However, increasing broadcast steps beyond $[50,950]$ leads to significant degradation.
• Similarly, although different models have different schedulers, timesteps and training strategies, their best strategy is similar. The optimal strategy is either $[100,800]$ or $[100,850]$.

Conclusion: PAB is easy to find optimal hyperparameters. It can consistently achieve good results for all three methods by setting:

• Broadcast range to PAB$\_{246}$ or PAB$\_{235}$.
• Broadcast beginning and end steps to $[100,800]$ or $[100,850]$.

Q4: If it's possible to dynamically select layer-wise broadcasting in conjunction with distillation-based methods to facilitate real-time video generation.

A4: Indeed, the integration of layer-wise broadcasting with distillation-based methods is technically feasible and potentially advantageous. The approach is particularly promising because distillation methods typically constrain generation to a fixed number of steps (e.g., 1 or 4), while our proposed method can be adapted to dynamically optimize redundancy reduction across different layers.

Given layers and data exhibit varying degrees of redundancy and distinct computational characteristics, this combination could offer complementary benefits - distillation providing speed-up through step reduction, and PAB enabling fine-grained optimization of computation allocation.

For example, a simple idea is to visualize the redundancy of attentions for each layer like in Figure 4c (Line 162), then apply approppriate broadcast strategy for them.

However, due to the current absence of distillation-based video generation models, we cannot conduct experimental validation of this integration. We believe this represents a promising direction for future research that could further advance real-time video generation capabilities.

We are particularly happy to continue the discussion in the discussion phase!

We sincerely thank the reviewer 6TRe for the valuable questions and comments. For the concerns and questions, here are our responses:

Q1: Why opt for the Mean Squared Error (MSE) metric to assess redundancy, and does the MSE metric accurately reflect the nuances of redundancy? If there might be a more suitable metric for assessing redundancy?

A1: Thanks for the comment.

We choose Mean Squared Error to access redundancy because:

• Outliner sensitivity: More sensitive to outliers than other losses like L1 loss (which means less redundancy), and tend to reduce the impact of really small changes (which means redundancy). Thus more sensitive to the change rate of latents, helping us to identify the redundant region.
• Theoretical basis: Aligns with Gaussian noise in the diffusion process, matching the likelihood function of Gaussian distributions.

There are also some metrics that can reflect redundancy, including MSE, L1, cosine similarity and mutual information, as shown in Figure 17 (Line 1188).

We find all of the metrics indicates the similar redundancy region (the middle 70% region) as MSE. If you have any other suggestions for the metric, welcome to propose!

Q2: It appears that utilizing broadcasting may lead to an increase in communication time. How can we quantify this additional overhead?

A2: Simply applying broadcast will not increase communication cost, because all outputs is stored and broadcast on the local machine.

But using multiple GPUs will increase communication cost, because it will need to change the sequence layout among GPUs for temporal attention. PAB can help to reduce the cost of such communication as shown in Figure 6 (Line 232), because PAB can skip the whole temporal attention module, thus eliminating the need for communication.

To demonstrate how PAB contributes in multiple GPUs case. Here we demonstrate the breakdown the contribution of PAB.

Experiments:

Experiment settings: DSP is the sequence parallel method we use. We follow the settings of Table 4, using Open-Sora to generate 8s of 480p videos. The speed is slightly different from the results in Table 1 because we are using different machines and environments.

Analysis: As shown in the experiment, we evaluate the independent contribution of PAB from computation and communication.

• With PAB's computation speedup (save computation by attention broadcast), the latency is further reduced by 24% compared with DSP only.
• With PAB's communication speedup (can save all communication costs when temporal attention is skipped), the latency can be further reduced by 5.0% compared with computation speedup only.
• Since we only evaluate based on PAB$_{246}$ (better quality but less speedup), PAB is able to achieve more speedup if using more aggressive strategies.

Conclusion: PAB can help to reduce both computation and communication costs.

Q3: Is it feasible to apply PAB to T2I (Text-to-Image) models, such as FLUX?

A3: Yeah, although T2I models only have spatial attention and cross attention, PAB can also generalize well to these methods. We adapt PAB to FLUX by applying spatial and temporal attention broadcast only. PAB$_{\alpha\gamma}$ denotes broadcast ranges of spatial ($\alpha$) and cross ($\gamma$) attentions.

Speedup: PAB can achieve 1.77x speedup for FLUX with minor quality loss using PAB$_{55}$ . We choose it because it offers best balance between speedup and image quality.

Note that this is only ran on single GPU. If you are interested in multiple performance with real-time performance, we are willing to implement!

Qualitative results: In appendix section B.8 (Line 1216), we update the qualitative results for FLUX with PAB$_{55}$. As shown in Figure 18 (Line 1221), the image quality is basically comparable with the original methods, which proves the generalization and effectiveness for PAB.

Quantitative results: We are still runing experiments for quantitative results with more PAB settings, but it will take some time because the model is big. We will provide update as soon as we get the final results.

Improvement plan: In the revision, we will update the quanlitative and quantitative results in the appendix B.8 (Line 1216). And the source code of PAB implemention for FLUX will be uploaded as well.

We sincerely thank the reviewer PKxB for the valuable questions and comments. For the concerns and questions, here are our responses:

Q1: Is the visualization in Fig 4 based on Latte? Whether the attention feature difference plot is derived from an entire dataset analysis or a single video generation process.

A1: Thanks for the corrrection. The visualization is indeed based on Latte, sorry for the mistake in the paper. We select Latte because its DDIM scheduler samples uniformly, therefore more suitable and clear for visualization of difference.

While the visualization is demonstrated using Latte, we note that our observations generalize well to other models as the results show in our experiments.

Regarding the attention feature difference plot, we conduct our analysis using the complete Vbench dataset to ensure comprehensive and statistically robust results.

Revise: In the revision, we correct the caption of Figure 4 (Line 175) from Visualization of attention differences in Open-Sora. to Visualization of attention differences in Latte.

We will further carefully check all details in the revision. Thanks for pointing out this.

Q2: Key implementation details like PAB$_{246}$ should be presented in the main paper.

A2: Thanks for this valuable feedback regarding the presentation of PAB$_{246}$.

We introduce PAB$_{\alpha\beta\gamma}$ in the caption of Table 1 (Line 288), where it's first mentioned. For the convenience, we provide the original caption as follows:

"PAB$_{\alpha\beta\gamma}$ denotes broadcast ranges of spatial ($\alpha$), temporal ($\beta$), and cross ($\gamma$) attentions."

Following the advice, we plan to make the explanations more clear.

Revise: In the revision, we clarify this term again in the Table 1 analysis paragraph (Line 266): "PAB$_{\alpha\beta\gamma}$ denotes broadcast ranges of spatial ($\alpha$), temporal ($\beta$), and cross ($\gamma$) attentions."

We sincerely thank the reviewer R3bt for the valuable questions and comments. For the concerns and questions, here are our responses:

Q1: How to determine the optimal broadcast ranges?

A1: Thanks for the comment. There are only two hyperparameters in PAB: 1) broadcast range. 2) the beginning and end steps of the broadcast. And both of them are easy to find the best hyper parameters.

1. Broadcast range

Experiment settings: We conduct the following experiment to find out how to determine the best broadcast for all methods, where PAB$_{\alpha\beta\gamma}$ denotes broadcast ranges of spatial ($\alpha$), temporal ($\beta$), and cross ($\gamma$) attentions. The italicized results mean significant performance degradation.

Analysis: As shown in the table, we can discover the following findings:

• When we increase broadcast range from PAB$\_{222}$ to PAB$\_{246}$ or PAB$\_{235}$ , there is not much performance degradation, as the redundancy degree of temporal and cross attention is larger. So broadcast range should follow: cross range > temporal range > spatial range.
• When extending to PAB$\_{257}$ or PAB$\_{333}$, there will be severe performance degradation.
• Although different models have different schedulers, timesteps, and training strategies, their best broadcast ranges are similar. The optimal range is either PAB$\_{246}$ or PAB$\_{235}$ considering both effect and efficiency.

2. The beginning and end steps of broadcast

Experiment settings: We conduct experiments to find out how to determine the best start and end steps for all methods, where $[\cdot, \cdot]$ denotes the end and start diffusion step of broadcast. The more steps it covers, the more speedup PAB can achieve. The italicized results mean significant performance degradation.

Analysis: As shown in the table, we can discover the following findings:

• Reducing broadcast steps to $[150,750]$ or $[200,700]$ does not increase worthy performance, but leading to less speedup. However, increasing broadcast steps beyond $[50,950]$ leads to significant degradation.
• Similarly, although different models have different schedulers, timesteps and training strategies, their best strategy is similar. The optimal strategy is either $[100,800]$ or $[100,850]$.

Conclusion: PAB is easy to find optimal hyperparameters. It can consistently achieve good results for all three methods by setting:

• Broadcast range to PAB$\_{246}$ or PAB$\_{235}$.
• Broadcast beginning and end steps to $[100,800]$ or $[100,850]$.

The authors have done additional experiments to answer my comments, which are interesting and insightful, such as the "broadcast the attention outputs vs. attention scores", and an efficient video generation model.

I am standing on my previous rating.