Efficient Generative Modeling with Residual Vector Quantization-Based Tokens

We sincerely thank the reviewer for the constructive feedback. We address the specific weaknesses and questions raised below:

Regarding W1: Isolating the Efficiency Gains of Depth Modeling We agree that comparing ResGen (masked generation) directly with RQ-Transformer (autoregressive generation) makes it difficult to isolate the efficiency gains specifically from our depth modeling strategy versus the gains from the masked generation framework itself. To provide a stricter ablation, as suggested:

• New Experiment: We created an "AR-ResGen" variant. This model uses the exact same spatial autoregressive transformer as the RQ-Transformer baseline. The key difference is that it replaces RQ-Transformer's autoregressive depth transformer with our cumulative embedding prediction MLP (of similar size) to handle the depth dimension. This isolates the effect of the depth modeling approach while keeping the sequence modeling framework constant (autoregressive).
• Results: This AR-ResGen achieves better FID scores than the original RQ-Transformer, particularly with few iterative refinement steps for the depth prediction, demonstrating the efficiency of our cumulative embedding approach even within an AR framework. The results (FID) are:

  AR-ResGen Iterations	w/o CFG	w/ CFG
  1	27.45	6.47
  2	24.10	5.33
  4	23.75	5.30
  8	23.48	5.22

• Conclusion: This targeted ablation confirms that our cumulative embedding prediction for depth modeling is inherently more efficient and effective than standard autoregressive depth handling, independent of the sequence generation strategy. We will include these findings in the revised manuscript.

Regarding W2: Comparison with MAGVIT-v2 and MaskBit We thank the reviewer for highlighting the missing comparisons with MAGVIT-v2 and MaskBit. We acknowledge their strong performance (FID 1.78 and 1.62) and will add these results to Table 1 and Figure 2 for context. Our current FID is 1.93 with similar steps. While their FID scores are currently lower, ResGen offers distinct advantages:

• Theoretical Advantage (Modeling Correlations): ResGen predicts cumulative embeddings to explicitly model dependencies across the RVQ depth dimension. This differs from methods like MAGVIT-v2/MaskBit using Lookup-Free Quantization (LFQ) and predicting independent bit groups. MaskBit's own ablation (Table 3b) shows performance degrading as independent groups increase, suggesting potential scaling limitations for higher fidelity (requiring more bits/groups). ResGen's approach of modeling correlations directly may offer better scalability for deep quantization or numerous discrete outputs. To show the benifit of modeling these correlation between tokens across depth, we implemented a variant of our method which uses the same masked generation framework introduced in our method section but predicts discrete tokens directly and in parallel across all depths at each step, instead of predicting continuous cumulative embeddings. This variant achieved FID scores of 12.79 (w/o CFG) and 2.91 (w/ CFG). While this performance surpasses the other comparison methods evaluated in our study, it is slightly worse than our final proposed model which predicts cumulative embeddings, thus supporting the efficacy of the cumulative embedding strategy to capture such token correlations.
• Practical Advantage (Resolution/Depth Trade-off): Our use of 16-depth RVQ enables a lower spatial resolution (8x8) than typical 16x16 VQ methods, while achieving high reconstruction quality measure in rFID. ResGen's efficient depth handling makes this viable, offering flexibility in memory usage and model design. We will clarify these points and add the comparisons to the manuscript.

Regarding Q1: Inclusion of VAR-d30 in Figure 2 Thank you for this suggestion. To provide a more complete comparison of performance and efficiency trade-offs, we agree that including VAR-d30 in the wallclock time comparison (Figure 2) would be beneficial. We will update Figure 2 to include VAR-d30 in the revised manuscript.

Regarding Q2: Exact Coordinates for Figure 2 (left) Certainly. The exact coordinates (Wallclock Time [s], FID) for the data points in Figure 2 (left) are:

• DiT: [ (4.68, 2.27) ]
• VAR: [ (0.16, 3.60), (0.19, 2.95), (0.24, 2.33) ]
• MAR: [ (89.69, 2.31), (105.6, 1.78), (133.01, 1.55) ]
• RQTran: [ (3.61, 3.89), (3.73, 3.80) ]
• ResGen-rvq8: [ (0.84, 2.87), (1.43, 2.78), (1.86, 2.75) ]
• ResGen-rvq16: [ (0.98, 2.26), (1.67, 2.14), (2.28, 1.98) ]
• MaskGiT: [ (0.98, 6.18) ] Solid lines connect different models; dashed lines connect points for the same model with varying sampling steps. We will add this detailed information, likely in the figure caption or appendix, for clarity in the revision.

We sincerely thank the reviewer for their detailed and constructive feedback. We appreciate the recognition of ResGen's motivation and its potential to address computational costs in RVQ-based generation. We address the specific concerns and questions below:

Concerning the fair comparison with MAR and further comparison with faster MAR (Claim Question), there's a key difference in how speed is measured. MAR reports throughput (sec/image averaged over a large batch, e.g., 256), while we report wall-clock time for single image generation on one A100 GPU. This accounts for the apparent discrepancy (0.3s vs. >100s). To directly address whether ResGen outperforms faster MAR variants, we conducted additional experiments:

1. Reducing MAR's AR Steps (Diffusion=100): Even MAR-B at 16 AR steps (fastest tested, ~5.6s/image) yielded worse FID than ResGen-rvq16 (FID 4.11 vs. 1.93).

   MAR FID (Diffusion=100)	AR=64	AR=32	AR=16
   MAR-B	2.33	2.44	4.11
   MAR-L	1.81	2.10	4.32

2. Reducing MAR's Diffusion Steps (AR=256): MAR-B at 25 diffusion steps (~22.5s/image) performed worse than ResGen-rvq16 (FID 3.38 vs. 1.93).

   MAR FID (AR=256)	Diff=100	Diff=50	Diff=25
   MAR-B	2.31	2.39	3.38
   MAR-L	1.78	1.83	2.22

These results demonstrate ResGen's consistent advantage in both speed and quality, even against accelerated MAR configurations. We will add this comprehensive comparison to the revision.

To clarify the experimental environment (Potential Improvement 1, W2.2): All ResGen evaluations used a single NVIDIA A100 GPU. Training details are in the appendix. For Table 1, we reported the best performance for each model based on a hyperparameter search. Detailed sampling configurations and hyperparameters will be explicitly included in the final manuscript. For Table 2, max batch size was the largest fitting on one A100 using checkpoints' best CFG settings. Inference speed is measured as the wall-clock time required to generate a single sample. We will explicitly state these details in the final manuscript.

Regarding RQ-Transformer guidance in Table 1 (W2.1): Thank you for noting potential confusion. For the ablation in Table 1 comparing generation on 8x8x16 tokens, we used CFG for ResGen, RQ-Transformer, and MaskGiT to ensure a fair comparison under identical conditions. As the reviewer noted, the RQ-Transformer paper uses rejection sampling. Rejection sampling is applicable to any generative model, but was not used in Table 1. We will clarify it in the revision.

Concerning scaling to deeper RVQ depths (Additional Weakness 2): For images, we used depths 8 and 16 due to excellent reconstruction fidelity (RFID 1.29 and 0.67, Appendix A.4). However, in audio (Table 3), we extensively evaluated deeper depths (32, 72) and observed consistent performance improvements, demonstrating stable training and benefits from increased depth in that domain. This suggests potential benefits for vision too, though not explored here due to already high fidelity at depth 16.

Regarding clarifying the benefit of discrete diffusion over autoregressive generation: Please refer to the newly added experiments and analyses presented in our response to Reviewer aQEq ("Regarding W1: Isolating the Efficiency Gains of Depth Modeling"), which directly address this point.

Regarding generation scaling law (Q1): We investigated scaling ResGen-rvq16 from 574M to 1B parameters on images (400K iterations, batch 256 on 4 GPUs). We observed consistent FID improvements across sampling steps:

Regarding the complexity and overhead of the Mixture of Gaussians (MoG) head (Additional Weakness 1, Q2), we acknowledge the need for clarity. The MoG head's computational cost is manageable. As detailed in Appendix A.2, parameter prediction involves projecting the hidden output (size O) to K mixture probabilities, K mean vectors (size H), and affine parameters. This leads to a projection complexity dominated by O(O*K*H). For our vision model (O=1152, K=1024, H=64), this cost is comparable to a standard softmax layer with a ~64K vocabulary (V = K*H), which is practical. For higher-dimensional embeddings like in audio (H=512), we use low-rank projection for the means (μ = M * μ_tilde + s, h << H), reducing the dominant computational term to O(O*K*h + H*h), again making the effective cost similar to a ~64K vocabulary model (h=64). This technique, previously used in CLaM-TTS, significantly mitigates overhead. We recognize these details were mainly in the Appendix and will clarify the MoG parameterization, its computational cost, and the low-rank optimization in the main paper.

We sincerely thank the reviewer for their thorough assessment, positive feedback on our claims, methodology, and supplementary material, and for recognizing the conceptual innovation of ResGen. We appreciate the constructive suggestions for further improvement.

Regarding the points raised:

Limited Dataset Diversity/Scalability (W1): We appreciate the suggestion to evaluate ResGen on more diverse datasets and higher-resolution settings. Our current study focused on established benchmarks like ImageNet and standard TTS datasets to ensure the adaptability of this method across different domains. We agree that broader validation is important and plan to explore ResGen's scalability and generalization on more diverse tasks, including higher-resolution image generation, as a key direction for future work. But to provide additional context on model scalability, we conducted scaling experiments with ResGen-rvq16, increasing model parameters from 574M to 1B on ImageNet. Although these scaling experiments are ongoing (currently at 400K iterations, batch size 256 on 4 GPUs), intermediate results reveal FID improvements, as shown below:

Analysis of MAR-B vs. ResGen-rvq16 Performance (W2): Thank you for prompting a deeper analysis of the results in Table 2. While MAR-B has fewer parameters and achieves slightly better FID without classifier-free guidance (CFG), ResGen-rvq16 excels with CFG. This difference stems from their distinct generative mechanisms. MAR-B utilizes multi-token prediction followed by continuous diffusion steps, allowing iterative refinement and revision of previously generated tokens. In contrast, ResGen performs disjoint token unmasking at each step without revisiting prior decisions. This fundamental difference in modeling leads to varying performance characteristics, particularly highlighted by the impact of CFG. Furthermore, although MAR-B can perform well without CFG, achieving its best results often requires significantly more sampling steps (e.g., 100+ diffusion steps) compared to ResGen, impacting inference speed. Thus, there is a trade-off between MAR-B's potential parameter efficiency (in certain configurations) and ResGen's inference efficiency, especially when aiming for high-quality results with guidance. We will incorporate this more detailed comparative analysis into the revised manuscript.

Sensitivity to Masking Strategies (Q1): This is an excellent question regarding the robustness of our approach. We investigated ResGen's sensitivity by training ResGen-rvq16 models (400K iterations) using three distinct masking schedules during training: circle, exponential, and cosine. We then evaluated these models under two sampling conditions: using the same or different schedule during sampling than training for all models.

Our results, summarized below (FID scores, lower is better), show interesting interactions:

Notably, the best performance (FID 9.53 at 64 steps with CFG) was achieved when training with the circle masking strategy but sampling using the exponential strategy. This suggests that the different training and sampling schedules impact the final quality. The exponential schedule unmasks fewer tokens in the crucial early stages and more tokens later. This coarse-to-fine unmasking during inference appears beneficial for ResGen, likely allowing the model to establish a more stable initial prediction before revealing finer details. The strong performance under exponential sampling, even when trained differently, indicates its effectiveness for inference with ResGen. We will include a detailed discussion of these findings in the revised manuscript.

We sincerely thank the reviewer for the positive evaluation and constructive feedback.

We address the specific points raised below:

Essential References Not Discussed (HART): Thank you for bringing HART to our attention. We agree it is a relevant recent work and will incorporate a discussion into our Related Work section. HART proposes a hybrid approach, distinct from purely discrete tokenizers like ours (ResGen, VAR) or continuous ones (MAR, GIVT). It decomposes latents into discrete tokens (modeled autoregressively) and continuous residuals (modeled via diffusion). This hybrid strategy aims to reduce sampling steps compared to continuous methods like MAR.

Theoretical Claims (Sec 3.2, Lines 263-265): Thank you for the careful reading and request for clarification. There might be slight confusion in terminology depending on perspective (encoding vs. decoding of VQ). Let us clarify our notation:

• z represents the continuous cumulative embeddings corresponding to the target clean tokens x^(0). Hence the term q(z | x^(0), x^(t)) represents a form of dequantization or embedding lookup.
• p(x^(0)|z, x^(t)) represents the inference of the true clean tokens x^(0) given this embeddings z and the current masked tokens x^(t). This involves finding the discrete tokens x^(0) whose corresponding embeddings are z. This step effectively performs the quantization of the continuous embedding z back into the discrete RVQ code space.

So, p(x^(0)|z, x^(t)) indeed relates to determining the discrete tokens x^(0) from the continuous embedding z, which involves the RVQ quantization mechanism applied to z.

Weakness 1: Potential for Further Inference Speed-up: Thank you for this suggestion. In the vision domain, reducing sampling steps to 18 (Appendix B.2) improves inference speed but slightly decreases generation quality (FID 3.94). However, experiments in the audio domain show that fewer steps (8 and 16 steps) still achieve comparable performance (see Table below).

Weakness 2: Elaborating Differences with VAR: Thank you for this suggestion; a clearer comparison with VAR will indeed strengthen the paper. While both ResGen and VAR utilize the hierarchical nature of RVQ, they differ:

• Structure & Resolution: VAR assigns different spatial resolutions to different RVQ depths (e.g., 1x1, 2x2, ... up to original resolution), requiring a predefined hierarchy. RVQ uses all quantization depths to refine the representation at a single resolution which is identical to the length of output sequence of the VAE encoder.
• Generative Process: VAR generates tokens autoregressively across depths, although sampling within a depth can be parallel given the previous depth. ResGen predicts cumulative embeddings representing multiple depths simultaneously within a masked generation framework, allowing parallel prediction across the sequence length dimension.
• Task Adaptability: VAR's predefined resolution hierarchy might be less straightforward to adapt to tasks with arbitrary output lengths, such as text-to-speech, where ResGen is easily applied.
• Resolution Flexibility: RVQ allows us to achieve lower final spatial resolutions (e.g., 8x8) by increasing quantization depth (e.g., 16), offering flexibility in balancing sequence length and depth. Achieving similarly low spatial resolution might be less direct in VAR's depth-resolution coupled structure.

We will incorporate a more detailed discussion contrasting these aspects in the related work.

We thank the reviewer for their insightful comments, which help improve our work's clarity. We address each point below:

Q1: Rationale for Predicting Quantized Discrete Tokens vs. Continuous Embeddings

Our approach of predicting discrete RVQ tokens via cumulative embeddings offers significant inference efficiency advantages over directly predicting high-dimensional continuous embeddings.

While direct continuous prediction (e.g., DiT) is valid, it often requires many inference steps. Our method targets discrete RVQ tokens (8x8x16) for efficiency, even though it uses continuous cumulative embeddings internally.

Generating masked tokens generally demands fewer inference steps than refining high-dimensional continuous vectors, as shown by MaskGiT achieving strong results faster than many continuous methods. Quantization, despite information loss, simplifies the target space, reducing the burden of precisely predicting large continuous vectors (e.g., 8x8x64 in our case).

Predicting cumulative embeddings enables a few step iterative prediction of all 16 token depths. This contrasts with single-step embedding prediction (e.g., GIVT) or continuous diffusion models needing many refinement steps. Quantization is crucial for this multi-depth strategy's feasibility.

In essence, we leverage internal continuous representations for modeling flexibility but target discrete tokens for efficient inference, particularly suited for our multi-depth prediction approach.

Q2: MaskGiT Reimplementation Baseline in Ablation Study (Section 5.2.1)

The MaskGiT variant in Table 1 uses 8x8x16 RVQ tokens specifically for a controlled comparison within our ablation study on multi-depth token generation, not to replicate the original MaskGiT's performance on its native 16x16x1 format. The experiments aimed to compare strategies for generating multi-depth RVQ tokens (8x8x16): (i) fully autoregressive (RQ-Transformer), (ii) masked sequence + autoregressive depth (our MaskGiT variant), and (iii) fully masked (ResGen). Fair comparison required all models to operate on the same 8x8x16 RVQ format. The relatively lower performance of the MaskGiT variant (predicting depth-by-depth) compared to ResGen highlights the challenge of extending single-depth masked models to multi-depth RVQ tokens and supports our approach of predicting all depths collectively.

To further strengthen this analysis and provide a more precise ablation, we conducted additional experiments:

1. As detailed in our response to Reviewer aQEq (W1), we implemented an "AR-ResGen" model which combines an autoregressive sequence model (like RQ-Transformer's) with ResGen's efficient iterative depth prediction using cumulative embeddings. This isolates the benefit of our depth modeling strategy.
2. We also implemented a variant of our method which uses the same masked generation framework introduced in our method section but predicts discrete tokens directly and in parallel across all depths at each step, instead of predicting continuous cumulative embeddings. This represents a more direct application of masked diffusion to RVQ depths. This variant achieved FID scores of 12.79 (w/o CFG) and 2.91 (w/ CFG).

These additional experiments demonstrate that while variations of our method can perform well, our proposed final method using cumulative vector embedding prediction achieves the best overall performance within our ablation study, validating its effectiveness.

We will revise Section 5.2.1 to clarify the baseline's purpose (controlled comparison on 8x8x16 RVQ tokens), implementation (sequential depth prediction), its role within the ablation, and add the results of our new ablation experiments as requested.

Q3: Clarity of the Term 'Discrete Diffusion Model'

We acknowledge the ambiguity regarding 'discrete diffusion model' and thank you for highlighting it.

Our discussion of 'discrete diffusion models' in the related works primarily referred to models like VQ-Diffusion, GIVT, and conceptually MaskGIT, which learn to reverse a corruption process (like masking) on discrete data (tokens). This was mainly confined to the first paragraph of the related works.

Other mentioned models (e.g., VAR, RQ-Transformer) were included for broader context in token-based modeling but were not necessarily classified by us as discrete diffusion models.

We also aimed to highlight 'masked diffusion' (corruption via masking only) as an effective subset of the discrete diffusion model, following D3PM and VQ-Diffusion.

We will revise the paper to provide a precise discrete diffusion model definition as used, clearly delineate this category from related models, explicitly refer masked diffusion, and ensure consistent terminology throughout.

We hope these clarifications and the planned revisions adequately address the reviewer's concerns. We appreciate the constructive feedback.