HART: Efficient Visual Generation with Hybrid Autoregressive Transformer

Q1: Clarification of condition in residual diffusion.

We clarify that the hidden states of the last layer contain information from all previous sampling steps. This is because the input at each sampling step is the cumulative feature map from all prior steps (see Figure 6). As a result, the residual diffusion module has the full context necessary to predict the final continuous residual tokens: discrete tokens from steps 1 to N-1, as well as the newly predicted discrete tokens from step N.

Q2: Description on efficiency improvement.

We have revised our description. Our original intention was to convey that naively adding the residual diffusion module introduces overhead in both training time and memory. However, our partial supervision strategy significantly accelerates training and reduces memory footprint.

Q3: The great advantage of alternating training still exists in the end of training.

We train the HART model with a separate-decoder hybrid tokenizer until convergence and benchmark its performance against the original HART model.

Clearly, the hybrid tokenizer trained with our alternating strategy maintains a huge performance advantage even after the hybrid transformer has converged.

Q4: Updating Figure 6.

Thanks for your helpful comments. We've updated Figure 6 accordingly.

Q1: Analysis on the correlation between rFID and gFID in HART.

We sincerely thank the reviewer for the positive comments.

An important principle for achieving both good rFID and gFID simultaneously is ensuring that the decoder used in generation can achieve strong discrete rFID. We assume that continuous rFID will generally perform well across different designs. We apply this principle to explain the two designs discussed in the paper:

1. HART's Tokenizer: A single decoder is used for generation. Its discrete rFID is competitive (Table 1), and the gFID is also strong.
2. Separate Decoder Approach (Figure 7, middle and right): Two decoders are used—one for continuous features and one for discrete features. The discrete decoder is frozen, and the continuous decoder is trained solely on continuous features. During generation, the continuous decoder is used, but its discrete rFID is poor (since it is not trained on discrete tokens), which results in a lower gFID.

We have found this principle useful in explaining other unsuccessful designs, such as using a single decoder but only fine-tuning on continuous tokens. This leads to the same issue as the separate decoder approach, where the decoder used for generation has very poor discrete rFID.

Q1: This paper is not about random selection between two mature pipelines.

HART's contributions are twofold:

• Proposing a hybrid tokenizer that addresses the reconstruction quality challenge in discrete tokens;
• Demonstrating that by predicting discrete tokens with a scalable-resolution autoregressive (AR) transformer and learning residual tokens with a compact residual diffusion model, we achieve 4.5–7.7x higher throughput than traditional diffusion models when generating 1024px images. This represents the first time AR models have shown efficiency advantages over diffusion models in high-resolution image synthesis.

We would like to politely highlight that while the reviewer focused primarily on our contribution in hybrid tokenization, they overlooked our substantial contribution to visual generation.

Below are quotes from other reviewers that recognize our significant contribution to visual generation:

1. Reviewer 5LBN highlights that our model architecture "exemplifies an effective design choice that balances efficiency and performance", and that "this hybrid approach not only achieves comparable speeds to VAR in class-conditional image generation tasks but also scales effectively to large datasets, as evidenced by its application to tasks like text-to-image generation."

2. Reviewer yud3 mentions that our model "demonstrates efficiency improvements over state-of-the-art diffusion models", our residual diffusion approach "minimizes memory and processing costs, achieving a reduction in latency and MACs", and our method's "latency and throughput metrics are efficient, offering a speed and computational advantage."

3. Reviewer y4uu mentions that the residual diffusion idea "is both simple and effective" and this design "will offer valuable insights to the community".

4. Reviewer a2Ay mentions that compared with existing AR and diffusion models, HART "achieves superior performance on text-to-image and class-conditioned image generation tasks" and "can generate 1024px image efficiently".

Even with our tokenization design, the choice is not a random selection between two established pipelines. Please refer to our comments below for detailed ablation studies.

Q2: 50%-50% sampling is a superior design choice in tokenizer training.

We perform an ablation study on the impact of different sampling ratios for continuous and discrete tokens during tokenizer training, as shown in the table below:

As shown, a 50%-50% sampling ratio achieves the best rFID and gFID. 30%-70% sampling proposed by the reviewer is still better than the discrete-only baseline.

Q3: HART's tokenizer training pipeline is better than alternative strategy for balancing continuous and discrete tokens.

First, we would like to clarify that our hybrid tokenizer training strategy is defintely not a ''cheating scheme''. The decoder is trained to reconstruct original images using both discrete and continuous tokens, meaning there is no way to bypass learning discrete tokens. This is supported by our results in Table 1, where the discrete rFID of our hybrid tokenizer is highly competitive.

Second, we experimented with an alternative design that interpolates randomly between continuous and discrete tokens during tokenizer training. This method avoids 'random selection' mentioned by the reviewer. However, the resulting generation FID is significantly worse compared to our approach.

Additionally, our pipeline demonstrates strong stability. As shown in Q2, varying sampling ratios for discrete and continuous tokens during training consistently yield competitive rFID and gFID results.

Q1: HART contrasts sharply with MAR.

We benchmark MAR-B with different number of sampling steps in the following table:

Reducing the sampling steps in MAR results in a significant drop in both FID and inception scores. In contrast, a similar study conducted for HART-1B revealed that the number of diffusion steps has little impact on generation quality:

We argue that the 'similar diffusion adaptations' mentioned by the reviewer essentially refer to decomposing image tokens into discrete and residual tokens and training a hybrid tokenizer to support this decomposition. This approach would effectively make MAR a special case of HART.

Furthermore, KV caching cannot be applied to MAR because the authors explicitly state in their paper that MAR employs bi-directional attention rather than causal attention. As a result, tokens generated later modify the features of previously generated tokens, rendering KV caching inapplicable to MAR.

Q2: HART's text conditioning method is better than cross attention.

We compare cross-attention-based text conditioning with HART's text conditioning under 256x256 resolution in the table below:

HART's text conditioning method outperforms cross-attention, achieving better FID with 1.3x better parameter efficiency and 1.1x larger throughput.

Q3: HART is fundamentally different from STAR and VAR-CLIP.

We summarize the difference between HART and the concurrent research papers, STAR and VAR-CLIP as follows:

1. Hybrid vs. Discrete-Only Tokenization

HART employs hybrid tokenization, where discrete tokens capture the broader structure and residual tokens represent fine-grained details. In contrast, STAR and VAR-CLIP rely solely on discrete tokens. Table 5 in our paper highlights the significant performance gains achieved by HART's residual diffusion module, which effectively models the residual continuous tokens.

2. High-Resolution Generation

While VAR-CLIP and STAR support image generation only up to 512px resolution, HART can efficiently generate 1024px images with 4.5–7.7x higher throughput and comparable quality to traditional diffusion models.

3. Text Conditioning

HART employs a unique text conditioning approach, differing from STAR (which uses cross-attention) and VAR-CLIP (which relies on the embedding of the EOS token as a single text condition token). As discussed in Q2, HART’s text conditioning method delivers superior results with 1.3x better parameter efficiency and 1.1x faster training efficiency compared to cross-attention-based methods.

Q4: HART has strong theoretical foundation.

We argue that HART is grounded in theoretical foundation. Hu et al. [1] (NeurIPS 2024) demonstrated that transformers are universal approximators for the score function in DiTs. When HART's diffusion head is implemented using a DiT (a special case of HART with heavier diffusion head), it can theoretically achieve universal approximation of the residual tokens under conditions in [1]. This enables HART to generate results as accurate as reconstruction outputs.

In contrast, discrete-only methods are fundamentally limited to modeling categorical distributions, or the space spanned by discrete tokens. As reflected in the quantitative rFID results (Table 1) and qualitative reconstruction outcomes (Figures 4 and 14), discrete tokenizers cannot achieve as good reconstruction performance as continuous or hybrid tokenizers. Consequently, discrete-only methods are theoretically incapable of matching HART’s generation upper bound.

[1] Hu et al., On statistical rates and provably efficient criteria of latent diffusion transformers (dits), in NeurIPS 2024.

Q1: Residual tokens are easier to learn than full image tokens.

(1) Most of the parameters in HART are used to model the discrete tokens, while only 37M parameters are used to model the residual tokens. In fact, MAR-L uses 72M parameters for the diffusion head, which is 1.9x more more than ours.

(2) We argue that runtime and MACs, rather than the number of parameters, are better measurements for modeling efficiency. HART achieves better FID and inception score than MAR with 12.9x higher throughput and 10.9x lower MACs.

(3) The diffusion head in HART requires only 6–8 sampling steps to achieve optimal FID and inception scores, whereas MAR typically requires 30–60 steps (see Figure 7, left). This difference in sampling steps further demonstrates that residual tokens are inherently easier to model.

Q2: Clarification on Table 4 and diffusion sampling steps.

For clarity, we have revised Table 4. The original #steps column refers to autoregressive sampling steps, not diffusion head sampling steps for HART and MAR. In HART, the number of autoregressive sampling steps is fixed by the tokenizer's resolution hierarchy and cannot be adjusted. Below, we benchmark HART-2B with additional diffusion steps:

The residual diffusion model achieves approximately a 10% improvement in FID when the sampling steps reach at least 6. Further increasing the sampling steps yields no significant gains in quantitative performance.