Communication-Efficient Diffusion Denoising Parallelization via Reuse-then-Predict Mechanism

Dear reviewer gLa1,

Thank you for providing constructive comments and valuable suggestions. Below, we address each point raised.

W1: Lacks comparison with single-device efficient inference approaches.

Reply: Thank you for your valuable comments. DDIM is a deterministic scheduler that, compared to stochastic schedulers such as DDPM, reduces sampling steps. Since ParaStep is not a scheduler-level work, it is fully compatible with DDIM and other deterministic schedulers. We demonstrate this compatibility in the table below.

To evaluate the efficiency of ParaStep, we compare it against both DeepCache and AsyncDiff. The experiments are conducted on SVD, which uses a deterministic scheduler EulerDiscreteScheduler. We denote the cache interval in DeepCache as s and the degree of parallelism in AsyncDiff and ParaStep as p.

First, SVD adopts EulerDiscreteScheduler as its scheduler. Our method achieves significant acceleration on SVD demonstrates its compatibility with deterministic schedulers. Second, ParaStep outperforms both DeepCache and AsyncDiff in terms of generation quality and latency.

W2: Requires manual tuning of hyperparameters (warm-up steps, parallelism degree), with limited discussion of robustness or automation.

Reply: Thank you for your valuable comment. Regarding the choice of warm-up steps, we find in practice that 10% of the total steps is sufficient for simpler models such as SVD and SD3, while 30% is a reasonable choice for more complex models like CogVideoX and Latte. Since generation quality and speedup are both highly sensitive to the number of warm-up steps, but remain stable across different input samples, we can efficiently profile a small set of candidates—e.g., [1, 10% × T, 20% × T, 30% × T]—to identify a good trade-off between quality and efficiency.

As for the degree of parallelism, diffusion models typically use no more than 50 inference steps, making parallelism degrees below 8 practical for achieving speedup. As shown in Figure 6 of our paper, ParaStep achieves significant acceleration with p = 8 while maintaining acceptable generation quality. Therefore, in practice, we can simply set the parallelism degree to the number of available devices on the machine (≤ 8).

W3: Needs full model replication on each GPU, limiting scalability for very large models.

Reply: Thank you for your valuable comment. We apologize for this unclear description in Section Limitations. A typical diffusion pipeline consists of several components, including a text encoder, a noise predictor (e.g., DiT or U-Net), and a VAE, among others. The model as mentioned in the paper denotes the noise predictor, and ParaStep only requires replicating the entire noise predictor across all devices. However, the text encoder, typically the primary memory bottleneck, is not replicated. Instead we split the text encoder into multiple stages across devices to reduce per-device memory consumption. We will clarify this point more explicitly in the revised version of the paper. The table below presents the memory breakdown for SD3, CogVideoX-2B, and Latte.

One may worry that splitting the text encoder increases latency. Since the text encoder is only invoked once to process the prompt, and splitting it only introduces a single communication step between adjacent stages, the additional latency incurred is minimal. To validate this, we conducted experiments on SD3. In the results, "not split" refers to using ParaStep without splitting the text encoder, while "split" means ParaStep with encoder splitting.

Q1: Can the selection of key hyperparameters be automated or made adaptive across different diffusion models?

Reply: Thank you for question. We answer this queston in W2.

Q2: For single-GPU scenarios, how does ParaStep (or BatchStep) compare to step-reduction methods?

Reply: Thank you for your question. We will answer this question in two parts, verifying the effects of ParaStep and BatchStep separately.

As discussed in W1, ParaStep is compatible with deterministic schedulers such as DDIM. Regarding the consistency model, it is indeed an effective approach for reducing the number of sampling steps in diffusion models. However, it typically requires costly and often unstable training, which limits its practicality in some scenarios. In contrast, our method is training-free. This makes it broadly applicable across various diffusion models.

To evaluate the efficiency of ParaStep in comparison with step-reduction methods, we conduct experiments on Flux1.dev, a sota diffusion model for image generation. Flux1.dev employs FlowMatchEulerDiscreteScheduler, a deterministic scheduler similar to DDIM. In the Direct Reuse method, the stride s indicates that s - 1 steps are skipped for every s steps. We also include comparisons with the original Flux model using a reduced number of sampling steps. As shown in the table below, ParaStep consistently achieves higher generation quality than both the corresponding Direct Reuse configurations and the reduced-step method. These results demonstrate the effectiveness of ParaStep relative to step-reduction methods, and further highlight its robustness even in extreme few-step settings.

To evaluate the efficiency of BatchStep in comparison with step-reduction method under a single-GPU setting, we conduct experiments on AudioLDM2-base. BatchStep achieves higher generation quality than the step-reduction, demonstrating its effectiveness in single-device scenarios.

Q3: Can ParaStep be effectively combined with single-device acceleration methods (such as DeepCache or DDIM) for further speedup?

Reply: Thank you for your suggestion regarding a hybrid approach that combines ParaStep with single-device acceleration methods. Since the steps in the denoising process are not equally important, we can apply ParaStep on the more critical steps, and use direct reuse methods (or DeepCache) on the less important ones. We believe this hybrid approach can achieve greater acceleration than ParaStep alone, while maintaining better generation quality than Direct Reuse or DeepCache.

One challenge of implementing this hybrid approach is the need for dynamic parallelism (as discussed in Section 4.4 of our paper). ParaStep is inherently a parallel method. For example, if we execute ParaStep with a parallelism degree p = 4 for the first 4 steps, and then switch to DeepCache on steps 5 and 6, only one device is required for those steps—leaving the other three devices idle. To make this hybrid method practical and efficient, we would need to develop a system that supports dynamic parallelism, allowing idle devices to serve other incoming requests and thus maximize overall utilization. This is an interesting direction that we believe is worth exploring in future work.

Dear reviewer eKuX,

Thank you for providing constructive comments. Below, we address each point raised.

W1: The proposed method has limited scalability. As the degree of parallelism increases, staleness is also increased.

Reply: Thank you for your comment. Similar to AsyncDiff and DeepCache, our ParaStep leverages the similarity across adjacent steps in the diffusion process. As the degree of parallelism increases (in ParaStep or AsyncDiff) or as the cache interval increases (in DeepCache), the generation quality of these methods may degrade. This is a limitation of techniques that rely on inter-step similarity.

However, we propose reuse-then-predict mechanism to effectively reduces the staleness caused by Direct Reuse. As a result, ParaStep achieves higher generation quality than the sota methods, as shown in Table 2 of our paper. Moreover, since diffusion models typically use no more than 50 inference steps, a parallelism degree of 4 or 8 is generally sufficient to achieve meaningful acceleration. As demonstrated in Figure 6, ParaStep delivers significant speedup with p = 8, while maintaining an acceptable quality drop.

W2: The proposed dynamic degree of parallelism seems not useful. Some compute resources are left idle, which is wasteful.

Reply: Thank you for your comment. This dynamic parallelism is particularly suitable for diffusion serving systems, where the goal is to respond to multiple requests efficiently and with low latency. For example, the system may initially assign 8 devices to a single request, allowing ParaStep to perform the first 16 denoising steps with a parallelism degree of 8. The system can then reduce the degree to 4 for subsequent steps, freeing up 4 devices to handle other incoming requests. If the parallelism degree is later increased back to 8 for the original request, only intermediate states—such as the noise ($\epsilon$) and the current sample ($X$)—need to be transferred to the newly assigned devices. The associated communication overhead is minimal.

W3: Unlike many distributed inference strategies, ParaStep replicates the full model on each device.

Reply: Thank you for your comment. We apologize for this unclear description in Section Limitations. A typical diffusion pipeline consists of several components, including a text encoder, a noise predictor (e.g., DiT or U-Net), and a VAE, among others. The model as mentioned in the paper denotes the noise predictor, and ParaStep only requires replicating the entire noise predictor across all devices. However, the text encoder, typically the primary memory bottleneck, is not replicated. Instead we split the text encoder into multiple stages across devices to reduce per-device memory consumption. We will clarify this point more explicitly in the revised version of the paper. The table below presents the memory breakdown for SD3, CogVideoX-2B, and Latte.

One may worry that splitting the text encoder increases latency. Since the text encoder is only invoked once to process the input prompt, and splitting it only introduces a single communication step between adjacent stages, the additional latency incurred is minimal. To validate this, we conducted experiments on SD3. In the results, "not split" refers to using ParaStep without splitting the text encoder, while "split" means ParaStep with encoder splitting.

W4: Methods like Ring Attention and AsyncDiff hide communication latency via asynchrony. This critical aspect is not mentioned in the paper, which is not fair.

Reply: Thank you for your comment. For Ring-Attention, it leverages asynchronous communication to overlap communication with adjacent computation. However, due to the large communication volume and limited memory bandwidth, communication cannot be fully overlapped with computation, which significantly limits the speedup achieved by Ring-Attention. As shown in Table 2 of our paper, Ring-Attention achieves only a 1.07× speedup due to this communication bottleneck, whereas our ParaStep attains a 1.82× speedup under the same configuration.

For AsyncDiff, to the best of our knowledge, AsyncDiff performs synchronous communication. To demonstrate that the speedup achieved by ParaStep primarily stems from improved communication efficiency, we conducted experiments on the SVD model.

W5: Leveraging similarities between adjacent denoising steps to accelerate or parallelize diffusion models has been extensively explored in prior work, including DistriFusion, AsyncDiff, DeepCache, and TeaCache. The concept of warmup is also explored in DistriFusion.

Reply: Thank you for your comment. The core difference between our ParaStep and the methods you mentioned lies in the reuse-then-predict mechanism we propose and adopt. This mechanism is designed to refine the quality degradation caused by direct reuse, leading to superior generation quality compared to sota acceleration methods.

To validate this, we conducted experiments comparing ParaStep with both DeepCache and AsyncDiff. As shown, ParaStep outperforms DeepCache and AsyncDiff. Specifically, with a parallelism degree of p = 2, ParaStep achieves higher generation quality than both baselines. Furthermore, ParaStep with p = 4 not only achieves lower latency than DeepCache with stride s = 8, but also delivers higher generation quality, demonstrating its effectiveness in balancing speed and quality.

Q1: How does ParaStep perform on FLUX.1, the state-of-the-art image generation model? and

Q2: What is the performance of ParaStep on diffusion models with fewer steps (e.g., 10 or 20)? Current experiments use 50 steps, which is not a typical setting in practice.

Reply: Thank you for your questions. Most of our experiments are conducted under the 50-step setting, as this is the default configuration for models such as CogVideoX-2B and Latte. Setting the number of steps of CogVideoX-2B to 35 results in a significant drop in generation quality.

To assess the performance of ParaStep in few-step settings, we conducted experiments on Flux1.dev, which supports few-step generation. Here, p denotes the degree of parallelism in ParaStep, and s represents the stride used in the Direct Reuse method. The results in the table demonstrate that ParaStep achieves speedup under few-step settings, while also outperforming both Direct Reuse and reduced-step generation in terms of output quality.

Dear reviewer p6GU,

Thank you for providing constructive comments and valuable suggestions. Below, we address each point raised.

W1: ParaStep requires replicating the entire model on each GPU. This is a significant limitation that constrains its applicability to models that can fit into the memory of a single GPU.

Reply: Thank you for your valuable comment. We apologize for this unclear description in Section Limitations. A typical diffusion pipeline consists of several components, including a text encoder, a noise predictor (e.g., DiT or U-Net), and a VAE, among others. The model as mentioned in the paper denotes the noise predictor, and ParaStep only requires replicating the entire noise predictor across all devices. However, the text encoder, typically the primary memory bottleneck, is not replicated. Instead we split the text encoder into multiple stages across devices to reduce per-device memory consumption. We will clarify this point more explicitly in the revised version of the paper. The table below presents the memory breakdown for SD3, CogVideoX-2B, and Latte.

A natural concern regarding splitting the text encoder is whether this would increase the overall pipeline latency. Since the text encoder is only invoked once to process the input prompt, and splitting it only introduces a single communication step between adjacent stages, the additional latency incurred is minimal. To validate this, we conducted experiments on SD3 to get the latency. In the results, "not split" refers to using ParaStep without splitting the text encoder, while "split" refers to using ParaStep with a split text encoder. P denotes the degree of parallelism. As shown, splitting the text encoder significantly reduces memory usage, while incurring only a minor increase in latency. By applying both ParaStep and text encoder splitting, we achieve not only lower latency but also reduced memory usage compared to the baseline.

W2: As mentioned above, the paper lacks a detailed performance breakdown (computation vs. communication). Such an analysis is crucial to definitively prove that the speedup comes from communication efficiency and to better understand the performance characteristics of both the proposed method and the baselines.

Reply: Thank you for your valuable suggestion regarding a more detailed performance breakdown. To demonstrate that the observed speedup primarily stems from improved communication efficiency, we conducted experiments on the SVD model. Specifically, we measured both the total latency and the communication-related latency for AsyncDiff and our proposed ParaStep. As shown in the table below, ParaStep exhibits significantly lower communication latency compared to AsyncDiff, which directly contributes to its higher overall speedup.

We also provide the performance breakdown on CogVideoX-2B and Latte. Since AsyncDiff does not currently support these models, we report the breakdown results for ParaStep only.

W3: Fig. 3 is slightly complex. It is suggested that the author could hide the technical details and show more of the high level idea.

Reply: Thank you for your valuable suggestion. We will simplify the Fig.3 by hiding lower-level technical details to highlight the overall design and the high level idea in the revised version of the paper.

Dear Reviewer,

We are approaching the end of the discussion period with the authors and the authors have provided you with their rebuttal.

May I please kindly ask to you check on the author rebuttal and engage in a meaningful discussion with authors?

There is also a new rule this year: you will need to tick the mandatory acknowledgement box, but only after discussing with the authors.

Thanks. AC

Dear reviewer W87m,

Thank you for providing constructive comments and valuable suggestions. Below, we address each point raised.

W1: The main concern is about the theoretical analysis of the proposed method.

Reply: Thank for your suggestion about about the theoretical analysis. We derive the upper bound of the error distribution introduced by the reuse-then-predict mechanism under a parallelism degree of 2. Specifically, we consider two devices, GPU0 and GPU1. We follow the notations and assumptions established in [1], as well as its derivation for the error bound associated with reuse. We appreciate the solid theoretical foundation provided by [1].

[1] Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy, NeurIPS 2024.

Assumptions：

• $\epsilon_\theta(x, t)$ is Lipschitz w.r.t to its parameters $x$ and $t$
• The 1st-order difference $\Delta x_i = x_i - x_{i+1}$ exists and is continuous for $0 \leq i \leq T - 1$

Notation:

• Let $x$ denote the accurate (ground-truth) sample.
• Let $\hat{x}$ denote the sample obtained via reuse (referred to as skip in [1]).
• Let $\tilde{x}$ denote the sample obtained via the reuse-then-predict mechanism.

The denoising step at iteration $i$ is defined as:

$x_i = f(i) \cdot x_{i+1} - g(i) \cdot \epsilon_\theta(x_{i+1}, t_{i+1})$

The error bound for reusing the sample at iteration $i{-}1$ is derived in [1] as follows: （equation(1)）