AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising

Q1: Can the similarity of the hidden states be quantitatively measured? For instance, does a low MSE between hidden states indicate that the two states are “similar”?

Thanks for the valuable comment. We provide quantitative analysis and visualization of the hidden state similarities in Figure 2 and Figure 3 of the attached PDF FILE.

Figure 2 presents the cosine similarity and MSE of each block's hidden state between adjacent steps, providing a quantitative measure of similarity. The results show that hidden states are highly similar between most steps, except at the initial stage. This high similarity allows for using features from the previous step to parallelize calculations. Figure 3 visually illustrates this phenomenon by displaying the hidden states of adjacent steps during the diffusion process. Both quantitative and qualitative analyses support our key insight. The initial instability in the curve explains why slightly increasing the warm-up steps can make the AsyncDiff output more consistent with the original output, as shown in Table 2 of our submission.

Q2: Is this phenomenon specific to the U-Net architecture, or is it agnostic to the backbone of the denoising model (Diffusion Transformer)?

Thanks for the comment. As a universal acceleration method, AsyncDiff is agnostic to the backbone of the denoising model and is compatible with DiT-based diffusion models. As shown in Table 1, our method significantly speeds up the DiT-Based Stable-Diffusion-3-medium model while maintaining high-quality outputs. The qualitative results in Figure 1 (b) of the attached PDF file demonstrate that AsyncDiff preserves the excellent image quality and accurate text generation of SD 3.

Table 1: Quantitative evaluations of AsyncDiff on DiT-Based SD 3

Note: We compare the generative quality when using the DDIM sampler and AsyncDiff to achieve the same speedup. The SD 3 using a default 28-step DDIM sampler is the baseline.

Q3: Is it only applicable to DDIM sampling? Does the phenomenon also hold for other fast ODE solvers, such as the DPM solver?

Thanks for the valuable feedback. AsyncDiff is a universal method that can be used with various samplers, including the DPM-Solver. Table 2 below provides the quantitative evaluation of AsyncDiff on the SD 2.1 with DPM-Solver sampler. With the same speedup ratio, AsyncDiff significantly improves generation quality compared to the baseline. We also provide the qualitative results in Figure 1 (a) of our attached PDF File. Our method achieves significant acceleration while maintaining higher consistency with the original output.

Table 2: Quantitative evaluations of AsyncDiff using DPM-Solver sampler

Note: We compared the generative quality when using the DPM-Solver sampler and AsyncDiff to achieve the same speedup. The SD 2.1 using a 25-step DPM-Solve sampler is the baseline.

Q4: A comparison with the original model using a number of DDIM steps that achieve a similar speedup would strengthen the argument for AsyncDiff.

Thanks for the valuable suggestions. In Table 3, we present a more comprehensive quantitative evaluation of AsyncDiff. Our method achieves significantly better generation quality at similar speeds, and the advantage increases with higher speedup.

Table 3: Additional Quantitative Evaluations of AsyncDiff

Note: We compared the generative quality when using the DDIM sampler and AsyncDiff to achieve the same speedup. The default 50-step SD 2.1 is the baseline.

Q5: Would a similar asynchronous denoising approach be applicable in a conditional generation setup as well?

Thanks for the valuable comments. AsyncDiff is a universal method that performs well with conditional generation setups. In Figure 1 (c) of the attached PDF file, we show how AsyncDiff accelerates ControlNet+SDXL. Our method reduces latency from 19.90s to 14.30s using two A5000 GPUs while nearly mirroring the original output.

Q1: Some concepts, such as the “dependency chain” in ABS, are not well explained. It would be beneficial if the author could provide a minimal explanation for these concepts

Thank you for the valuable feedback. This is indeed a question that requires further explanation. In the diffusion process, there are two "dependency chains": one between time steps and the other between model blocks.

Dependency between time steps: The denoising process is sequential, meaning that denoising at timestep t depends on the completion of denoising at timestep t-1. This sequential dependency causes latency accumulation and limits the ability to perform parallel inference.

Dependency between blocks: The large denoising model is split into N components, each with similar computational loads. The input to the nth block depends on the output of the n-1 th block, which poses challenges for running the entire denoising model in parallel.

Our AsyncDiff seeks to disrupt both the dependency chain between time steps and the dependency chain between blocks by leveraging the hidden state's similarity between adjacent steps, allowing for the parallel execution of the denoising process while closely approximating the results of the sequential process.

Q2: Could the author provide a more detailed discussion about the key differences between AsyncDiff and Distrifusion? What is the main advantage of AsyncDiff?

Thank you for the comment. As you mentioned, the main difference between AsyncDiff and Distrifusion lies in their core approaches. AsyncDiff focuses on model parallelism, enabling the denoising model to run in parallel regardless of the input data. In contrast, Distrifusion emphasizes data parallelism, parallelizing the computation across different patches of the image. Based on these differences, the advantages of AsyncDiff can be summarized as follows:

1. Wider Applicability:
AsyncDiff is a more universal acceleration framework because it focuses on model parallelism, independent of the type of input data or the task of the original model. This makes it suitable for various diffusion-based generation tasks, including text-to-image, text-to-video, image-to-video, super-resolution, ControlNet, speech generation, etc. It can also be used with both Unet-based and DiT-based diffusion models.

2. Lower Memory Requirements:
Distrifusion requires caching activation maps for each layer, significantly increasing GPU memory demands and complicating practical applications. In contrast, AsyncDiff only needs to cache a few hidden states on each device, keeping memory requirements nearly the same as the original model.

3. Higher GPU Utilization:
When the generated image resolution is not very high, Distrifusion can lead to lower GPU utilization, resulting in underutilization of computing resources. AsyncDiff mitigates this issue by not altering the size of the input data, thus maintaining efficient GPU usage. The experimental results in Table 3 of our submission show that AsyncDiff can achieve the same speedup with four GPUs as Distrifusion does with eight GPUs, while also delivering better generation quality.

Table 1: Quantitative comparison Distrifusion on SD 2.1

To ensure a fair comparison with Distrifusion, we increased the warm-up steps in our method to match the speedup ratio of Distrifusion, allowing us to fairly compare generation quality and resource costs.

Q3: Are there any failure cases for the proposed AsyncDiff?

As an acceleration method, AsyncDiff depends on the quality of the original base model. If the base model's generation quality is not good, AsyncDiff will also produce unsatisfactory results. While our method can significantly speed up the process and maintain the original generation quality, it typically cannot enhance the base model's original generation quality.

Q4:Is this method helpful for large batch sizes?

Thanks for the comment. As shown in the following Table 2, AsyncDiff will still be helpful for large batch sizes.

Table 2: Acceleration and Latency under Different Batchsizes

We evaluate the running speed of AsyncDiff on SD 2.1. The latency is evaluated using Nvidia RTX A5000 GPUs, CUDA 12.0, and PyTorch 2.1.

Q5:Is this method applicable to more GPUs?

Thanks for the comment. Asyncdiff can be applicable to more GPUs. However, we do not recommend splitting the model into more than four components, as the speed gains are minimal. In practice, combining AsyncDiff with other data parallel methods is more effective. Asyncdiff is compatible with any other data-parallel solution (batch splitting, patch splitting). Taking advantage of model parallelism and data parallelism at the same time will achieve a more extreme speedup ratio while minimizing the sacrifice of generation quality.

Q1: Is the communication cost huge compared to the inference cost? Will there be issues if this is extended to 8 or 16 GPUs? Is this method still competitive when the number of devices exceeds 8, where cross-node communication becomes substantial?

Thanks for the valuable comment, this is indeed a question that needs further explanation

As demonstrated in Table 1, the communication cost is minimal compared to the inference cost.

Applying to 8 or 16 GPUs can lead to increased communication costs due to GPU architecture. In a common 8-GPU node, GPUs 0, 1, 2, and 3 form one group, while GPUs 4, 5, 6, and 7 form another. Cross-group communication is about 40% more costly than within a group, the cross-node cost is even higher. This is a shared challenge for all distributed inference frameworks, both in the Diffusion and LLM areas.

Our proposed stride denoising technique effectively alleviates this issue by reducing communication frequency. As shown in Table 2, during a 50-step diffusion process, AsyncDiff requires only 25 communications. This significant reduction means our method is less affected by increased cross-group or cross-node communication costs, keeping it competitive. In the future, we will explore more innovative technologies to further address this general challenge.

Table 1: Time cost comparisons on SD 2.1

Note: 'Overall' represents the overall latency, 'Running' represents the time cost of model running, 'Comm.' represents the communication time cost. 'Ratio' in this table represents the proportion of communication cost to overall latency. All measurements were conducted on NVIDIA A5000 GPUs.

Table 2: Effect of stride denoising in the 50-step Diffusion Process

Note: Stride denoising significantly lowers communication costs by decreasing the communication frequency

Q2: It would be beneficial if the authors could provide more analytical results about the method, such as the similarity of hidden states.

Thank you for your valuable comment. We provide quantitative analysis and visualization of the hidden state similarities in Figure 2 and Figure 3 of the attached PDF FILE.

Figure 2 shows the cosine similarity and MSE for each block's hidden state between adjacent steps. As illustrated, the hidden states are highly similar between most steps, except at the initial stage. This high similarity enables us to use features from the previous step to parallelize calculations. Figure 3 further visualizes this phenomenon by displaying the hidden states of adjacent steps in the diffusion process. The initial instability demonstrates why slightly increasing the warm-up step can make the AsyncDiff output image pixel-wise consistent with the original output, which is also shown in Table 2 in our submission.

Q3: Figure 5: “Quantitative” should be “Qualitative”.

Thanks for the careful review. We will correct this error in the next version.

Q4: What happens if the total number of timesteps is not divisible by the number of GPUs? Is the proposed method still applicable in this scenario?

Thank you for the valuable comment. In this case, AsyncDiff will not face any issues. Although we split the denoising model into N parts, we still obtain one timestep of noise after each parallel batch. However, the runtime for this step in the denoising model is reduced to 1/N.

This is supported by the experimental results in Table 1 of our submission. When accelerating a 50-step diffusion model, AsyncDiff performs well, even when using 3 or 4 GPUs.

Q5: Is the proposed method still effective on low-end GPUs or edge devices?

Thank you for the valuable comment. This is a very practical question. We conducted additional experiments to show the acceleration ratio of AsyncDiff on consumer-grade graphics cards (NVIDIA RTX 2080 Ti GPUs). As shown in Table 3, our method still achieved a significant speedup on these GPUs.

Table 3: Acceleration Ratio and Latency on low-end GPUs

Note: Acceleration Ratio and Latency are evaluated using CUDA 12.0 and PyTorch 2.1

Q1: Whether AsyncDiff can be combined with other diffusion samplers such as DPM-Solver

Thanks for the valuable feedback. AsyncDiff is a universal method that can be used with various samplers, including the DPM-Solver. Table 1 below provides the quantitative evaluation of AsyncDiff on the SD 2.1 with DPM-Solver sampler. With the same speedup ratio, AsyncDiff significantly improves generation quality compared to the baseline. We also provide the qualitative results in Figure 1 (a) of our attached PDF File. Our method achieves significant acceleration while maintaining higher consistency with the original output.

Table 1: Quantitative evaluations of AsyncDiff using DPM-Solver sampler

Note: We compared the generative quality when using the DPM-Solver sampler and AsyncDiff to achieve the same speedup. The SD 2.1 using a 25-step DPM-Solve sampler is the baseline.

Q2: Can other GPUs achieve a good acceleration ratio?

Thanks for the valuable comments. We conducted additional experiments to compare the acceleration ratio of AsyncDiff on three different GPUs (NVIDIA RTX A5000, NVIDIA RTX 3090, and NVIDIA RTX 2080 Ti). As shown in Table 2, our method achieved a significant speedup across all tested GPUs.

Table 2: Acceleration Ratio and Latency on Different GPUs

Note: Acceleration Ratio and Latency are evaluated using CUDA 12.0 and PyTorch 2.1