We appreciate the reviewer for the insightful and constructive comments. We respond to your comments as follows and sincerely hope that our rebuttal could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our paper.

W1: Misuse of the "Speculative" Concept. It incorrectly implies that the method possesses the same theoretical guarantees as speculative decoding in the LLM field.

We do apologize for causing confusion. Please let us clarify our original intent: 1) We never try to imply our method is lossless: we presented the quality drop in abstract, introduction, and evaluation in the submission. 2) We deliberately add "Speculative" before "Verification", with the intent to explain our verification uses an "approximate" way.

In the revised version, we will emphasize the difference with speculative decoding in the LLM field at the beginning of the paper and use the name of "Approximate Speculative Diffusion with Speculative Verification (ASDSV)".

W2/Q1: The Rigor of the Theoretical Analysis：1) Can you provide a more complete theoretical model for the expected speedup that incorporates the verification failure rate (as a function of δ and K)? 2) How would this more realistic model change your discussion on the importance of the cost coefficient (c)?

1) Yes. Here is a more complete formula for the expected speedup, which incorporates the verification failure rate $E(\beta)$. The expected speedup factor for each stage is as follows:

S_{expected} = \frac{(1-E(\beta))^2\gamma + (1 - (1-E(\beta))^2)}{\gamma c + 2} \tag{1}

Proof. Let the cost of running a single step of $M_p$ by $T$. An unoptimized round of SDSV costs $\gamma cT+2T$. This cost covers running the draft model $M_q$ $\gamma$ times and running the target model $M_p$ twice (at the first and last step). Assuming the two verification checks are independent, the expected number of steps advanced per cycle is given by: $E[\text{steps}] = (1-E(\beta))^2 \gamma + (1 - (1-E(\beta))^2) \cdot 1$ This calculation assumes simple failure handling, where a failed verification results in advancing a single step. Thus, the effective cost per advanced step is the total cycle cost divided by the expected steps advanced: $\frac{\gamma cT+2T}{(1-E(\beta))^2\gamma + (1 - (1-E(\beta))^2)}$ Since the cost of generating a single step with the target model $M_p$is $T$, we get the desired result.

2) As stated in our submission, $c$ is still a system-dependent constant, determined by the specific hardware and software implementation [19]. A smaller c (a faster draft model) reduces the drafting overhead. However, it may imply a simpler model that struggles to approximate the target model's output, thus increasing the verification failure rate $E(\beta)$. In our experiments, conservatively-chosen $\gamma$ leads to few failure cases.

We will add this analysis to Section 3.3 of our revised manuscript. Thank you for helping us strengthen our theoretical discussion.

Q2: The Sensitivity of the Core Hyperparameter δ

Both $\gamma_1$, $\gamma_2$ depend on the temporal correlation between draft and target model on different stages, so SDSV is sensitive to these parameters (see Figure 2). Nevertheless, since the temporal correlation can be profiled offline, we can systematically search for the best candidates for these hyperparameters for each model offline. Please check the detailed analysis below and we will add them to the revised version.

1. Analysis of $\gamma_1$ and its search method

First, we use brute force to search $\gamma_1$ because the space is small: it can only be an integer in (1,5). Below is the search space of Wan2.1. We only present one model result for simplicity, other models are the same.

2. Analysis of $\gamma_2$ and its search method

Because the search space of $\gamma_2$ is larger than $\gamma_1$ (an integer selected from the range (1,15)), we developed a two-step iterative-guided search process for $\gamma_2$ to balance the speed-fidelity trade-off:

• Profile Model Dynamics: We conduct a small-scale sampling run to profile the model's behavior, generating plots similar to Figure 2. This allows us to observe the output change dynamics and step-wise verification loss, which are crucial for setting the $\gamma_2$ and deciding the verification threshold.
• Iteratively Tune $\gamma_2$ and $\delta$: We start with a candidate $\gamma_2$ based on the theoretical speedup ceiling for our target (Section 3.3). Next, using the verification loss curve (like Fig. 2b/c), we set a corresponding threshold $\delta$ above the observed loss at the $\gamma_2$-th step. We then incrementally increase $\gamma_2$ and adjust $\delta$, evaluating the trade-off on a validation set until the desired speedup is met with minimal quality impact.

The following results show typical search results on Wan2.1 model, others are the same:

The baseline for these comparisons is the original model, corresponding to the setting $\gamma_1=0, \gamma_2=0$. The setting of $\delta=\infty$ represents an "always-accept" strategy that effectively bypasses the verification stage to achieve maximum acceleration, corresponding to our SDSV-fast variant in Figure 5.

W3: Limited Novelty Beyond Heuristics. It is conceptually similar to existing cache-based skipping methods.

First, SDSV continues the line of using a fast path to accelerate multi-modal generation but introduces a new paradigm for the "fast path" in multi-modal generation: approximate speculative generation. Compared with caching (e.g., PAB [45], Teacache [24]), the fast path is augmented with a draft model to significantly enhance the quality and fidelity (as shown in Figures 1 and 5). Compared with non-approximate speculative (e.g., Traditional speculative diffusion [2]), the generation latency speedup is improved by 2x with negligible quality drop thanks to the significantly reduced verification overheads.

Second, the design of an approximative speculation framework is far more challenging than simply deciding a threshold for skipping. As far as we know, we are the first to show that the vanilla draft-then-verify, which was designed for LLM, does not work well for SOTA, high-resolution multi-modal generation. Instead, we propose a draft-then-approximate-verify paradigm. It requires us to address which verification steps can be skipped, and how to balance the speed-fidelity trade-off. We propose systematic solutions including a multi-stage verification strategy and a system-level batched pipeline (deciding threshold is just one component of our overall design), and present promising results (e.g., 1.8-3X speedup with a minimal 0.3%-0.4% drop in VBench score) on diverse tasks using SOTA models like FLUX.1-dev and Wan2.1.

Q3: About writing: All formulas in the article are unnumbered.

Thanks for your suggestion. We will fix it in the revised version.

We thank the reviewer for the insightful and valuable comments. We especially appreciate the recognition that our work is a pratical solution and provides a thorough evaluation. We respond to your comments as follows and sincerely hope that our rebuttal could properly address your concerns.

W1: Simple failure handling may waste the computation from the draft model.

As discussed in Section 5, a binary search can be employed instead of discarding the entire K-step sequence. When a failure is detected at the final step, SDSV would recursively check the mid-step of the sequence to efficiently find the largest valid prefix of accepted steps. Since of our conservative hyperparameter setting, we encounter few failures and thus currently do not employ such a failure handling method.

W2/Q1: The hyperparameters are empirically chosen for specific model pairs. How well do they generalize to new models? Are there automated methods for tuning them?

Yes, our hyperparameters can be determined for new models through an automated process. We address each parameter below:

• For $\gamma_1$ and Initial Steps $N$ (Generalization): We found $\gamma_1$ to be a less sensitive hyperparameter for the short Stage 1, generalizing well across models. A fixed, conservative value (typically 2 or 3) is a robust choice. The initial step count N is set to 5, analogous to the warm-up phase in other methods [10,20].
• For the Stage Boundary (Profile Model Dynamics): We determine the boundary between Stage 1 and Stage 2 via a small-scale profiling run. By generating a small number of samples (as in Fig. 2a), we can observe the model's output dynamics. This empirical data is crucial for setting the stage boundary.
• For $\gamma_2$ and $\delta$ (Iterative Tuning): We start with a candidate $\gamma_2$ based on the theoretical speedup ceiling for our target (Section 3.3). Next, using the verification loss curve obtained from the profiling stage (e.g., Fig. 2b/c), we set a corresponding threshold $\delta$ slightly above the observed loss at the $\gamma_2$-th step. Finally, we iteratively increase $\gamma_2$ and adjust $\delta$, evaluating the trade-off on a validation set until the desired speedup is met with minimal quality impact.

W3/Q2: Can you provide a direct cost comparison against traditional method (Speculative Diffusion) on a high-resolution task?

Yes. Here is one direct comparison. On the Wan2.1 (Text-to-Video) model, traditional Speculative Diffusion takes 1,120s compared to the original model's 930s, i.e., increases the inference latency by 20%.

This slow down is because the core assumption of the original speculative diffusion (Unet)—efficient parallel verification of draft steps—breaks down on SOTA, compute-bound diffusion in transformer (DiT) models.

We thank the reviewer for the insightful and valuable comments. We especially appreciate the recognition that our work addresses a significant problem and represents a promising approach. We respond to your comments as follows and sincerely hope that our rebuttal could properly address your concerns.

W1/Q1: In the proposed method, for every K steps, only the first and last steps’ noises will be compared and verified.How to guarantee that the target model samples every step from N+1 to N+K-1, and the output of the (N+K-1)-th step of the target model will be the same?

As detailed in Section 3.1, our speculative verification method chooses an approximative (threshold-based) while compution-less acceptance strategy instead of guaranteeing exactly the same output. We think this is a practical tradeoff between generation fidelity and latency.

Specifically, as illustrated in Figure 2(a), after an initial warm-up phase, the step-wise change is highly consistent for draft and target models. That is, the magnitude of change between consecutive steps, measured by L1_Loss becomes highly similar(i.e., $||D\_i - D\_{i-1}||\_1 \approx ||T\_i - T\_{i-1}||_1$). Based on the draft and target model both trajectories evolve with a similar trend, if the start and last step of the current round are verified to be similar, and then the intermediate points are also highly likely to be valid (i.e., their verification loss would be below the threshold $\delta$).

This high fidelity is demonstrated by the extensive quantitative metrics (e.g., FID, IS) and qualitative visual results presented in our evaluation.

W2/Q2: Calculating FID between the generated images and the real images.

SDSV aims to preserve the generation quality of original models, so that we only present the FID compared with the original generationed images in the paper. Following your advice, we now provide the FID scores against real images on COCO Captions 2014:

The scores based ground truth here are very close, with a change of only 0.9%. Although the score for SDSV-fast is higher (0.17% higher than SDSV-slow), SDSV-slow is actually closer to the original score, which also meets our expected goal.

We thank the reviewer for the insightful and valuable comments. We especially appreciate your recognition of an important problem, a novel method, and a clear analysis. We respond to your comments as follows and sincerely hope that our rebuttal could properly address your concerns.

W1/Q1: Can the authors include standard deviations over multiple seeds in their reported results?

Our method is insensitive to the random seed. This is because the seed in the diffusion process only affects the initial noise, whereas the efficacy of SDSV depends on the temporal correlation between the draft and target models (as shown in Figure 2).

To quantitatively verify this and address the reviewer's request, we further conducted a new set of focused experiments on three additional seeds (3461, 772190542, 118270042274490399) for our main comparisons (Original vs. SDSV) during the rebuttal period. Performance was consistent across both seeds (SDSV achieves better performance than other accerlation methods). The standard deviations over multiple seeds results on Text-to-Images over are below:

Since most of the video metrics are reference-based, we cannot calculate a relative ratio (original/accelerated method). Therefore, we used a comparison between the accelerated methods (SDSV vs. Teacache), and SDSV is about 1.5x faster than Teacache.

W2/Q2: Can the authors comment on hyperparameter sensitivity to $\gamma1, \gamma2$?

Both $\gamma_1$, $\gamma_2$ depend on the temporal correlation between draft and target model on different stages, so SDSV is sensitive to these parameters (see Figure 2). Nevertheless, since the temporal correlation can be profiled offline, we can systematically search for the best candidates for these hyperparameters for each model offline. Please check the detailed analysis below and we will add them to the revised version.

1. Analysis of $\gamma_1$ and its search method

First, we use brute force to search $\gamma_1$ because the space is small: it can only be an integer in (1,5). Below is the search space of Wan2.1. We only present one model result for simplicity, other models are the same.

2. Analysis of $\gamma_2$ and its search method

Because the search space of $\gamma_2$ is larger than $\gamma_1$ (an integer selected from range (1,15)), we developed a two-step iterative-guided search process for $\gamma_2$ to balance speed-fidelity trade-off:

• Profile Model Dynamics: We conduct a small-scale sampling run to profile the model's behavior, generating plots similar to Figure 2. This allows us to observe the output change dynamics and step-wise verification loss, which are crucial for setting the $\gamma_2$ and decide the verfication threshold.
• Iteratively Tune $\gamma_2$ and $\delta$: We start with a candidate $\gamma_2$ based on the theoretical speedup ceiling for our target (Section 3.3). Next, using the verification loss curve (like Fig. 2b/c), we set a corresponding threshold $\delta$ above the observed loss at the $\gamma_2$-th step. We then incrementally increase $\gamma_2$ and adjust $\delta$, evaluating the trade-off on a validation set until the desired speedup is met with minimal quality impact.

The following results show typical search results on Wan2.1 model, others are the same:

Due to the rebuttal format constraints, we are unable to show visual results here, but they will be included in the revised version.

We thank the reviewer for the insightful and valuable comments. We especially appreciate the recognition that our work addresses an important/interesting problem. We respond to your comments as follows and sincerely hope that our rebuttal could properly address your concerns.

Q1. What draft models are used in Table 1?

The draft model used in Table 1 is Flux-SVD. As we described in Lines 232-233, Flux-SVD is the Flux.1-dev model quantized by the SVDQuant [21]. We will explicitly label "Flux-SVD" as the draft model in the caption of Table 1 in the revised version.

W1. Using an extra model increases memory consumption.

Yes, but it can be alleviated. We measured the peak memory overhead to be +9GB (29.7%) for Text-to-Image and +5GB (8.0%) for Text-to-Video. However, the target and draft models do not need to run concurrently. This allows the extra memory consumption to be almost entirely eliminated by using asynchronous swapping without affecting inference latency (i.e., swapping the inactive model's parameters between GPU and CPU).

W2. SDSV does not generalize well to low-step diffusion models; all experimental results are based on 50-step settings.

SDSV focuses on accelerating diffusion models which have more steps and thus face the issue of slow inference. Low-step diffusion models are out-of-scope. We will clarify the positioning of our work.

Our experiments are based on the 50-step setting because most SOTA text-to-image and text-to-video models operate in this setting to ensure high-fidelity results. For example, leading models like FLUX.1-dev (T2I), Wan2.1 (T2V), and CogVideoX1.5 (I2V) all specify 50 steps as their default configurations.

W3. The paper's writing needs to be improved.

Thanks for the detailed feedback. We will further proofreed our manuscript.

1. We will define all key mathematical symbols at their first appearance. For example, we will state that "$D_i$ and $T_i$ denote the predicted outputs from the draft and target models at step $i$, respectively," ensuring all notations are unambiguous throughout the paper.

2. We will provide a more precise definition for the term "output trend" in Line 134. This refers to our key empirical observation: the strong temporal correlation between draft and target models in the diffusion process. Specifically, as shown in Figure 2(a), the magnitude of change of the draft model's output and the target model's output between consecutive steps is highly similar (i.e., $||D\_i - D\_{i-1}||\_1 \approx ||T\_i - T\_{i-1}|\|_1$) after the first $N$ initial steps.

3. We will add a detailed explanation for our choice of $\gamma_2$ in Line 183. As we now illustrate with a red vertical line in Figure 2(b)(c), the verification error curve exhibits a sharp increase at the 9th step. This inflection point shows that the draft model's outputs are starting to diverge significantly from the target model's. Therefore, it represents the appropriate moment to halt the drafting phase.