Response to Reviewer 3CPQ

Thank you for your thoughtful and constructive feedback on our paper. We are particularly grateful for your recognition of our novelty and high performance.

1. Enhanced prelimiary on speculative decoding and Jacobi decoding

Thank you for your suggestion. To improve clarity and readability, we improve preliminary section and add more knowledge details related to our core design. Specifically, we refine the descriptions of speculative decoding and Jacobi decoding: (A) speculative decoding: Speculative decoding employs a small model to accelerate the sequential generation of large autoregressive model. This model is trained on the same domain as the large model and is small enough for faster generation. In each step of inference, the small model first generates a sequence with its own inference paradigm. The large model then verifies this sequence in a single forward pass, selecting a prefix to serve as part of the final output. This verification ensures that each sampled token adheres to the probability distribution parameterized by the large model. Since multiple tokens can be taken as the final output by only one forward pass of the large model, the acceleration can be achieved; (B) Jacobi decoding: Jacobi decoding deems the auto-regressive inference as a process of solving the fixed point of a nonlinear equation in a triangular system. This decoding algorithm iteratively performs multi-token decoding and can be executed without fine-tuning or auxiliary modules. The specific process of Jacobi decoding is as follows: First, given the previously pre-filled or decoded tokens, we randomly initialize a sequence of candidate tokens. Then, in each iteration, we execute one forward pass of the auto-regressive model for all the candidate tokens with a causal mask. the predicted probabilities then generate the tokens typically via greedy sampling, and these sampled tokens are taken as the inputs of the next iteration. This process can be formulated as: $x\_{i}^{(j+1)} = \arg \max\_x p\_\theta(x|\mathbf{x}\_{1:i-1}^{(j)})$, where $i$ denotes the token index and $j$ denotes the iteration index. The Jacobi decoding process continues iterating until the convergence is reached, as determined by a deterministic criterion where these tokens remain unchanged between consecutive iterations.

2. Investigations on the refinement

Thank you for your suggestion. Since we cannot include links in this rebuttal, we use a pair of tables to show the change of token categories at each sampling step and token position, so each table includes an example of the token trajectory. In SJD2 (shown by each column of Table A), over 25 sampling steps, the predicted tokens start identical (e.g., token value 167282). As the noise level decreases, the token categories become diverse, but some token categories begin to repeat, which means the trajectory becomes stabilize. In contrast, for SJD (the columns in Table B), the token categories change irregularly, appearing to oscillate and remain unstable. We will provide more detailed visualizations using figures in the camera-ready version.

Table A: The discrete token trajectory in SJD2 with Emu3 as baseline:

Table B: The discrete token trajectory in SJD with Emu3 as baseline:

3. About the possibility of batch inference

In principle, our SJD2 can inherently support batch inference as a variant of speculative decoding [A]. However, our current implementation has not yet been optimized for batch processing due to engineering challenges such as synchronization overhead. We are actively exploring engineering optimizations, including dynamic batching and asynchronous verification, to enable efficient batch inference in future versions.

[A] Qian H, Gonugondla S K, Ha S, et al. BASS: Batched attention-optimized speculative sampling[J]. arXiv preprint arXiv:2404.15778, 2024.

4. About details of Equation 3

As shown in Appendix A, we set $\alpha_t=1-t$ and $\sigma_t=t$ for Equation 3.

Response to Reviewer eMe4

Thank you for your thoughtful and constructive feedback on our paper. We appreciate your recognition of our introduction and the significant acceleration.

1. About our contribution

We feel it necessary to clarify the contribution of our work: our SJD2 introduces a new advancement by integrating denoising process into autoregressive (AR) models to stabilize the Jacobi iteration in SJD. We achieve this through fine-tuning AR models to be compatible with noisy embeddings (thus the denoising can be integrated in Jacobi iteration, shown in Eq. 3, lines 191-202), while distinctively preserving core AR mechanisms (discrete tokenization, next-token prediction with cross-entropy loss, and Jacobi iteration) via the proposed techniques like noise perturbation on normalized embeddings. Unlike existing AR+diffusion models that rely on diffusion losses and auxiliary decoders for image generation, SJD2 maintains the standard AR components for image generation (i.e. predicting the probability of discrete tokens, and training with cross-entropy loss) while delivering significant acceleration gains (shown in Table 1).

2. About details of our finetuning objective and timestep injection

Thank you for your suggestion. We will improve the clarity of the method section in the camera-ready version and we detail our finetuning objective and timestep injection here: (A) timestep injection: Since we avoid introducing additional adapters, we utilize a flexible operator, i.e., the attention mechanism, for our token-wise timestep injection. Specifically, we take the sinusoidal encodings of timesteps as a sequence of special token embeddings and append them to the sequence of input token embeddings. Then, the sequence, which comprises clean input token embeddings, noisy input token embeddings, and timestep encodings, is fed into the transformer blocks. Within the attention modules of these blocks, we use the attention mask to force each noisy token embedding to attend to the corresponding timestep encoding which indicates its noise level. To ensure the distribution of the Fourier encodings of timesteps aligns with that of the token embeddings, we apply a normalization-then-de-normalization process to these encodings (normalization with statistics of the sinusoidal encodings and denormalization with the statistics of the token embeddings), similar to Eq 4. (B) finetuning objective: SJD2 fine-tunes a pre-trained autoregressive text-to-image model to predict the next clean token from noisy token embeddings. During fine-tuning, input token embeddings are perturbed with Gaussian noise and then fed into transformer blocks followed by a prediction head which outputs logits representing the categorical probability distribution of the next clean token. The cross-entropy loss is computed between these logits and the ground truth token categories, optimizing the model to denoise inputs while maintaining autoregressive prediction. This objective enables SJD2 to handle noisy inputs during inference.

3. Difference between SJD2 and AR+diffusion methods

In our related work section (Integration of autoregression and continuous diffusion), we have discussed models like Diffusion-forcing and Transfusion. However, with the recent publication of numerous new AR+Diffusion works, we now provide an updated version of the related work part: Recent diffusion forcing models (e.g. Self-forcing [A], MAGI-1 [B], and Causvid [C]) introduce the autoregressive sampling into the temporal dimension of the continuous video diffusion models. They employ causal masks to support history-conditioned video generation. The multimodal large language models (MLLMs) (e.g. BLIP-3o [D], Emu2 [E], and Seed-X [F]) generate images via pre-trained diffusion models (e.g., SDXL) which take the output features of autoregressive models as conditions. Moreover, the unified models like Bagel [G] and Janus-flow [H] directly integrate diffusion process into the autoregressive backbones. These models train the autoregressive backbones together with lightweight decoders through diffusion loss (e.g., noise prediction or velocity prediction) for image generation, and the backbone can perform both autoregressive decoding and continous diffusion sampling. In contrast, SJD2 uniquely integrates denoising into pre-trained autoregressive models without modifying their core: preserving discrete tokenizers, next-token prediction with cross-entropy loss, and the established inference mechanisms. Our innovation lies in fine-tuning with noise-perturbed normalized embeddings, enabling ODE-based denoising in autoregressive models. We do not relay on diffusion loss and additional decoders.

[A] Huang X, Li Z, He G, et al. Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion[J]. arXiv 2025.

[B] Teng H, Jia H, Sun L, et al. MAGI-1: Autoregressive Video Generation at Scale[J]. arXiv 2025.

[C] Yin T, Zhang Q, Zhang R, et al. From slow bidirectional to fast autoregressive video diffusion models[C]. CVPR 2025.

[D] Chen J, Xu Z, Pan X, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset[J]. arXiv 2025.

[E] Sun Q, Cui Y, Zhang X, et al. Generative multimodal models are in-context learners[C]. CVPR 2024.

[F] Ge Y, Zhao S, Zhu J, et al. Seed-x: Multimodal models with unified multi-granularity comprehension and generation[J]. arXiv 2024.

[G] Deng C, Zhu D, Li K, et al. Emerging properties in unified multimodal pretraining[J]. arXiv 2025.

[H] Ma Y, Liu X, Chen X, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation[C]. CVPR 2025.

4. Comparison to modern parallel decoding methods

We compare our method with the recent state-of-the-art and classic speculative/parallel decoding methods, including Lantern (ICLR 2025) [A], ZipAR (ICML 2025) [B], Eagle [C] and Jacobi Decoding [D], on COCO2017 validation set with Lumina-mGPT as baseline. As shown in the table below, our approach achieves superior acceleration while maintaining comparable visual quality.

[A] Jang D, Park S, Yang J Y, et al. Lantern: Accelerating visual autoregressive models with relaxed speculative decoding[C]. ICLR 2025.

[B] He Y, Chen F, He Y, et al. ZipAR: Parallel Autoregressive Image Generation through Spatial Locality[C]. Forty-second International Conference on Machine Learning (ICML 2025).

[C] Li Y, Wei F, Zhang C, et al. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty[C]. Forty-first International Conference on Machine Learning (ICML 2024).

[D] Song Y, Meng C, Liao R, et al. Accelerating feedforward computation via parallel nonlinear equation solving[C]. PMLR 2021.

5. About the superscript and subscript in equation 2

The superscript in Equation (2) indicates the noise level applied to the embedding $e$, with the skip from $0$ to $t_0$ reflecting a deliberate design in the application of noise perturbation. As demonstrated in Figure 3, some regions of the embeddings remain unperturbed by noise (denoted by superscript 0), while their adjacent regions are perturbed to a minimal noise level (denoted by superscript $t_0$). The subscript $i$ identifies the position of a specific token embedding, and $i'$ is an integer in the range $[1, i]$, used to specify a certain position where the noise level shifts from $0$ to $t_0$.

6. About inputs in original autoregressive models

The pre-trained autoregressive models used as baselines are incompatible with normalized embedding inputs. Consequently, after performing denoising on normalized embedding inputs, we employ a denormalization step (Equation 4) to restore these noisy normalized embeddings to the original embedding space.

7. About the training process

Unlike standard diffusion models that rely on diffusion loss (i.e. $\mathbb{E}\_{t,\epsilon,x_0} [ ||v\_\theta(x\_t,t)-(\epsilon-x\_0)|| ]$), the training objective of SJD2 is based solely on cross-entropy loss. This is because SJD2 is not a standard diffusion model: it only leverages the denoising concept from diffusion models while not using the diffusion loss. Specifically, during training, we perturb token embeddings with noise and the cross-entropy loss is applied on the noisy embeddings. During inference, the ODE denoising process is fused with the Jacobi decoding which is also a multi-step decoding algorithm (line 191-202 with equation 3). There is no high-order gradient computations in our fine-tuning.

8. About the fine-tuning efforts

To build the training dataset for fine-tuning, we collect about 80,000 synthesized images from huggingface and recaption them with Qwen-VL. We perform fine-tuning on 8 NVIDIA GPUs (80GB memory each) with a global batch size of 64, while leveraging DeepSpeed ZeRO-3 or FSDP with gradient checkpointing to save GPU memory at the cost of increased training time. All model parameters are used for fine-tuning. The fine-tuning requires 6 epochs, which costs about $14 \times 8$ A100 hours for Lumina-mGPT and $26 \times 8$ H100 hours for Emu3.

9. About the refinement with denoising

This refinement with denoising is the key distinction between SJD2 and SJD. As shown in Table 1, SJD2 achieves greater acceleration than SJD.

Thanks for the rebuttal, my questions are partially answered. The author should make the proposed method (AR & Diffusion) clearer in paper, both training & inference procedure, text description & figures (like Fig.3).

To my view, the proposed method is to predict next token until the noisy embedding is acceptable. If so, I'd like to know how many denoising step would it take during the generation of each token?

Response to Reviewer Q75X

Thank you for your thoughtful and constructive feedback on our paper. We particularly appreciate your recognition of the clear methodological presentation and our performance of acceleration. Our primary contribution lies in enabling efficient parallelization of existing autoregressive architectures through lightweight fine-tuning, rather than pursuing state-of-the-art performance.

1. Results on more autoregressive models

Thank you for your suggestion. As our SJD2 is developed for the paradigm of next-token prediction, we choose Janus-pro-1B for experiments. The paradigm of VAR is next-scale prediction which could require different ways of acceleration and thus is beyond the scope of our paper. Specifically, we employ our pairwise text-image data to tune Janus-pro-1B to be compatiable with noisy embedding inputs and use our SJD2 for decoding. The results are in the following tables. From the results, we observe that SJD2 still can accelerate Janus-pro without sacrificing on generated image quality, as evidenced by the following Geneval metrics.

2. Comparison to modern parallel decoding methods

We compare our method with the recent state-of-the-art and classic speculative/parallel decoding methods, including Lantern (ICLR 2025) [A], ZipAR (ICML 2025) [B], Eagle [C] and Jacobi Decoding [D], on COCO2017 validation set with Lumina-mGPT as baseline. As shown in the table below, our approach achieves superior acceleration while maintaining comparable visual quality.

[A] Jang D, Park S, Yang J Y, et al. Lantern: Accelerating visual autoregressive models with relaxed speculative decoding[C]. ICLR 2025. [B] He Y, Chen F, He Y, et al. ZipAR: Parallel Autoregressive Image Generation through Spatial Locality[C]. Forty-second International Conference on Machine Learning (ICML 2025). [C] Li Y, Wei F, Zhang C, et al. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty[C]. Forty-first International Conference on Machine Learning (ICML 2024). [D] Song Y, Meng C, Liao R, et al. Accelerating feedforward computation via parallel nonlinear equation solving[C]. PMLR 2021.

3. About the lack of comparison to diffusion models

Actually, our work focuses on improving the inference efficiency of standard autoregressive (AR) models for image generation. Thus, we choose these baselines: Lumina-mGPT, Emu3, and Janus-Pro that is supplemented in this rebuttal. The AR models have advantages in the unified modeling of multimodal data (predicting discrete tokens for both the linguistic and visual tasks, different from the diffusion models) and thus it is worth studying [A, B, C]. While many AR models currently underperform state-of-the-art diffusion models in image quality and face acceleration challenges, our SJD2 narrows the speed gap. In the following table, we evaluate inference latency for several commonly-used diffusion models (smaller than 3B) and Janus-Pro at the same resolution ($384 \times 384$). We set the number of sampling steps for diffusion models as 50. Results demonstrate that our SJD2 reduces latency of Janus-Pro-1B from 9.1s to 2.5s, narrowing the gap between Janus-pro and the advanced diffusion models like SD3. Moreover, this result means Janus-Pro-1B with SJD2 already outperforms SDXL in speed (2.5s vs. 4.3s):

[A] Chen X, Wu Z, Liu X, et al. Janus-pro: Unified multimodal understanding and generation with data and model scaling[J]. arXiv preprint arXiv:2501.17811, 2025.

[B] Wang X, Zhang X, Luo Z, et al. Emu3: Next-token prediction is all you need[J]. arXiv preprint arXiv:2409.18869, 2024.

[C] Team C. Chameleon: Mixed-modal early-fusion foundation models[J]. arXiv preprint arXiv:2405.09818, 2024.

4. Regarding Performance on GenEval

This work focuses on accelerating existing pre-trained autoregressive models while preserving visual quality, not on achieving state-of-the-art image quality. As shown in Table 2, our method successfully maintains visual fidelity on GenEval. While current autoregressive baselines like Lumina-mGPT generally underperform compared to advanced diffusion-based generation models, our proposed method has the potential to be applied to stronger autoregressive base models in the future.

5. Performance of Emu3 on Geneval benchmark

In the following table, we compare our SJD2 to autoregressive decoding (AR) and SJD on the GenEval benchmark with Emu3 as the baseline. Our method achieves an overall score of 0.49, which is close to the 0.52 of AR and the 0.48 of SJD, demonstrating preserved visual quality.

6. Flops in inference

We present the average Flops per output token in the table below. The results reveal that autoregressive decoding requires fewer Flops than SJD2. Although more Flops are used for decoding, the practical latency becomes lower. Actually, this Flops overload stems from the paradigm of all the speculative decoding methods. Their drafting-and-verification mechanism, which inherently introduces computational overhead: the number of accepted tokens per sampling step is substantially lower than the number of input draft tokens. Previous studies have also confirmed this observation. For example, as demonstrated in Tables 4 and 5 of [A], Medusa (a speculative decoding method) consumes 403 GFlops per token, whereas vanilla autoregressive decoding requires only 19 GFlops.

[A] Lin C H, Tuli S, Smith J, et al. SLiM: Speculative decoding with hypothesis reduction[C]//Findings of the Association for Computational Linguistics: NAACL 2024. 2024: 1005-1017.

7. About compatibility of parallel-decoding models

Thank you for your question. Actually, speculative decoding was originally designed to accelerate standard autoregressive models that decode tokens sequentially (one token per forward pass). Its lossless acceleration relies on optimizing GPU parallelism utilization. However, the models employing parallel decoding techniques (e.g., MaskGit or the next-set-prediction of MAR) inherently utilizes GPU parallelism well. Consequently, speculative decoding methods like SJD2 would require redesign and new theoretical foundations to assist these frameworks, which can be explored for future works. Regarding the diffusion head of MAR which operates on continuous latents, the speculative decoding should be redesigned to compatiable with Gaussian inputs, and some concurrent efforsts have been conducted and still in exploration [A].

[A] Wang Z, Zhang R, Ding K, et al. Continuous speculative decoding for autoregressive image generation[J]. arXiv preprint arXiv:2411.11925, 2024.

8. Comparison of the training cost of SJD2

Regarding the fine-tuning cost, the baseline, Lumina-mGPT, requires 10 million image-text pairs for pre-training. In contrast, our fine-tuning uses only 80,000 images over 6 epochs, totaling approximately 0.5 million image samples used during training. This indicates that at most 5% of the original computational resources are used for fine-tuning our method. We only use $14\times8$ A100 hours for fine-tuning, highlighting the efficiency.

Response to Reviewer ucHV

Thank you for your thoughtful and constructive feedback on our paper. We particularly appreciate your recognition of our detailed ablation studies, clear methodological explanations, and high-quality experimental demonstrations.

1. About potential application on other domains like text/audio

Thank you for your thoughtful suggestion about extending Jacobi decoding to text and audio generation. While such expansion falls beyond the scope of current review cycle, we plan future research to pursue this direction, including curating diverse text/audio datasets and exploring integrations with autoregressive LLMs like Llama and Qwen by using SJD2.

2. About fine-tuning details and the training/inference cost

All model parameters are used for fine-tuning. To build the dataset, we collect about 80,000 synthesized images from huggingface and recaption them with Qwen-VL. We perform fine-tuning on 8 NVIDIA GPUs (80GB memory each) with a global batch size of 64, a learning rate of 2e-5, and the AdamW optimizer ($\beta_1$=0.9, $\beta_2$=0.95), while leveraging DeepSpeed ZeRO-3 or FSDP with gradient checkpointing to save GPU memory at the cost of increased training time. The fine-tuning requires 6 epochs, which costs about $14 \times 8$ A100 hours for Lumina-mGPT and $26 \times 8$ H100 hours for Emu3. The inference cost is in the following table:

3. About discussion on Rolling Diffusion Models

Thank you for your suggestion. SJD2 accelerates autoregressive text-to-image generation by integrating denoising with Jacobi iterations, using a fixed sliding window for parallel token prediction in discrete token space. It fine-tunes pre-trained models to predict conditional probability given the input of noisy token embeddings, and uses the cross-entropy loss for supervision. During inference, the token embedding is first denoised and then seamlessly enters the standard Speculative Jacobi iterations draft and verify its corresponding discrete token and probability. In contrast, Rolling Diffusion Model applies a continuous diffusion process with a rolling time-window for time-series data, trained with diffusion loss to predict noise in continuous space. While both use window-based denoising for efficiency, SJD2 focuses on the discrete tokens in autoregressive models, whereas Rolling Diffusion Model focuses on adapting continuous diffusion for temporal sequences. We will add the discussion in the camera-ready version.

4. About segment size and noise level selection

The process of sequence segment dividing and noise level selection is as follows: First, we sample a monotonically increasing timestep sequence $( 0, t\_{0}, \cdots, t\_{K'} , 1 )$ from the range of $[0,1]$, where $K'$ is randomly determined in range $[10, 50]$ (the typical number of steps in diffusion sampling). The elements $t\_i$ in this timestep sequence can adhere to the Karras timestep schedule [A]. Then, for an input token sequence of length $N$ ($N > K'$), we uniformly partition it into $n$ segments ($n$ is randomly chosen from range $[1, \frac{N}{K'}]$), and fill each segment uniformly with the timestep sequence $( 0, t\_{0}, \cdots, t\_{K'} , 1 )$ across all positions. These timesteps are used to calculate the parameters $\alpha_t=1-t$ and $\sigma_t=t$ for noise perturbation on token embeddings.

[A] Karras T, Aittala M, Aila T, et al. Elucidating the design space of diffusion-based generative models[J]. Advances in neural information processing systems, 2022, 35: 26565-26577.

5. About the writting of positional encoding

Thank you for your suggesion. Our position encoding is the same as the sinusoidal position encodings used in standard Transformers. We will add this explanation in the camera-ready version.

6. About the user preference A/B tests

We conduct user preference on the images generated by autoregressive decoding and SJD2. We use Lumina-mGPT as baseline and select 16 pairs of images with 10 users for study. The results are in the following table. We observe that the preference of the two decoding methods is close and the images generated by autoregressive decoding is slightly more preferred.

7. The influence of decreasing the 1 in the lefthand side in Equation 1

Thank you for your question. First, for simplicity, let's consider replacing the constant "1" on the left-hand side of Equation 1 with a variable $k$. Actually, decreasing this variable $k$ would reduce the acceptance rate, as a lower $k$ makes the acceptance condition $r < \min \left( k, \frac{\mathcal{P}\_\theta(x\_i^{(j)} | x\_{1}^{(j)}, \cdots, x\_{i-1}^{(j)})}{\mathcal{P}\_\theta(x\_i^{(j)} | x\_{1}^{(j')}, \cdots, x\_{i-1}^{(j')} )} \right) , ~~ r \sim \mathcal{U}[0,1]$ harder to satisfy. We also conduct corresponding experiments and the results aligns with our hypothesis. Thus, $k=1.0$ not only conforms to the theory of speculative sampling, but also guarantees a relatively large acceptance rate.

8. About relationship between normalization and the failure mode

We do not utilize high-order statistics like skewness or kurtosis for normalization and denormalization. Actually, the failure mode depicted in Figure 5 arises when the noise-perturbation-based fine-tuning (and the denoising sampling) applied directly to token embeddings without normalization and denormalization. As we discussed in lines 213-220, normalization and denormalization primarily serve to scale input values appropriately for noise perturbation and transformer blocks, respectively. The noise perturbation formula $x\_t = \alpha\_t \cdot x\_0 + \sigma\_t \cdot \epsilon$, where $\epsilon \sim N(0, I)$, can produce extreme values, i.e., $\epsilon$ may reach magnitudes of 6 or -6, because $\epsilon$ is sampled from a standard Gaussian distribution. However, we observe that the token embedding statistics are consistently small across many models (e.g., mean = 0.0179, std = 0.2709). It is challenging for models to adapt to large-value inputs with few fine-tuning epochs. Consequently, we implement noise perturbation on normalized embedding values and subsequently rescale them for model input.

9. The possibility of a training-free version of our method

Currently, it is challenging to achieve a training-free variant of our method. The difficulty stems from the fact that autoregressive models are not exposed to Gaussian-perturbed tokens during their standard training process. These models are typically optimized for next-token prediction using discrete tokens generated by tokenizer encoder (e.g., VQGAN with CNN encoders), where the inputs are not intentionally corrupted with Gaussian noise throughout pre-training. Consequently, the pre-trained autoregressive models may not encounter noisy token representations, making it difficult to generalize to such inputs without explicit training adaptation.

10. About the open-source

We will release our code upon paper acceptance.