Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots

Q1: Training-inference discrepancy issue

Yes. We mainly focus on the design of hierarchical masked autoregressive model which addresses the training-inference discrepancy for passing tokens from phase 1 to phase 2, while the discrepancy caused by the inherent autoregressive process in each stage still remains. Such discrepancy is also occurred on most existing autoregressive models. We will discuss this.

Q2: Speed and accuracy behave for longer visual sequences

As suggested, we experimented on larger 512 resolution and Hi-MAR-L (FID: 1.62) outperforms MAR-L (FID: 1.73), while its computational cost is 20.9% less than MAR-L. We will add this.

Q3: More comprehensive benchmarks

Thanks. As suggested, we evaluate T2I model on T2I-CompBench and GenEval benchmarks. Due to limited computational resources, here we compare our Hi-MAR with other state-of-the-art methods (e.g., U-ViT-S and AutoNAT-S) which are trained on MS-COCO with similar parameter size. The larger models with billions of parameters trained on billions of images (e.g., SDXL, SD3) are not included. As shown in the following tables, our Hi-MAR consistently outperforms other baselines with comparable parameter size. We will add this.

Q4: Configuration of Hi-MAR-S

For fair comparison, Hi-MAR-S follows the configuration of U-ViT-S/2 (Deep) and its exact configuration is shown in the following table. We will add this in Table 1.

Q5: Training setup

To be clear, we only employ the diffusion loss as the training objective on both stages. Following MAR, we set the maximum timestep as 1,000 for the Diffusion heads. The models in Figure 4 follow this training setup.

Q6: The gap between Hi-MAR and MAR reduces. The scalablility of Hi-MAR to larger images and larger models

The performance in ImageNet is almost saturated and it is relatively difficult to introduce large margin of improvement. Considering this comment, we conducted the suggested experiments on larger resolution (i.e., 256 -> 512) and the FID achieves 1.62, which improves over MAR-L by 0.11. Furthermore, we scale both MAR and Hi-MAR to 2B parameters, and the FID of MAR and Hi-MAR is 1.49 and 1.45 respectively. The results basically demonstrate that Hi-MAR is scalable to both larger images and larger models. We will add this.

Q7: Concurrent works

We appreciate the suggested concurrent works and we are also happy to discuss them in revised version.

Q8: Safe innovation

Please refer to Q1 of Reviewer SRgF for more discussion on technical contribution against existing works.

Q9: More visualizations

As suggested, we provide more visualization results on the link. We will add this.

Q10: Impact on phase 1 resolution

We experimented with smaller resolution (i.e., 64x64) on phase 1 and the FID score is degraded into 2.06 due to the quality of images generated in such smaller resolution is relatively low. Therefore, we choose 128x128 resolution for the first phase. We will add this.

Q11: Other design choices of the diffusion heads

We also experimented by replacing the self-attention layer with cross-attention in the Diffusion Transformer head to mine the context among all tokens, and the FID is slightly dropped by 0.05 compared to the final version of Hi-MAR. We will discuss this in ablation study.

Q1: Novelty

Thanks. We summarize the differences between Hi-MAR and conventional multi-scale generation models as follows:

1. During training, the conventional models (i.e., Muse, VAR, Hart) commonly utilizes the ground-truth low-resolution visual tokens directly to guide the next phase prediction. Instead, Hi-MAR takes the conditional tokens estimated from the Hi-MAR Transformer of low-resolution visual tokens as the condition. Such design can mitigate the training-inference discrepancy as discussed in section 3.2. As shown in Table 4, the FID score improves from 2.28 to 2.07 when adopting the conditional tokens to mitigate such discrepancy.
2. Both VAR and Hart models multi-scale probability distribution via a shared Transformer without additional guidance. This way leaves the inherent different peculiarities of each scale in autogressive modeling not fully exploited, resulting in a sub-optimal solution for multi-scale token prediction. In contrast, our Hi-MAR incorporates a scale-aware Transformer block that provides scale guidance to the Transformer tailored to each phase.
3. Both VAR and Muse discretize images by VQGAN, resulting in severe information loss. Instead, Hi-MAR adopts continuous tokenizer via a diffusion loss, overcoming the poor generation upper bound caused by the vector quantization.
4. Muse utilizes two different models for low/high-resolution image generation and the two models are trained separately. Instead, our Hi-MAR jointly optimizes the probability distribution for low/high-resolution tokens with a shared scale-aware masked autoregressive Transformer and two small diffusion heads, which is more parameter-efficient.
5. In contrast to Hart that utilizes an MLP-based diffusion head to model the each token probability distribution individually, Hi-MAR devises Diffusion Transformer head by exploiting the self-attention to model the interdependency among tokens. Note that we appreciate the suggested reference of concurrent work Hart (pulished on ICLR 2025 with camera ready deadline on Mar 14, 2025). We will add the discussion.

Q2: Speed comparison with VAR and HART

As suggested, we compare the speed of our Hi-MAR with mentioned VAR and HART in the following table. The results basically demonstrate that Hi-MAR achieves superior performance against VAR and HART with comparable computational costs. We will add this in revision.

Q3: Other resolution settings

Thanks. We experimented by equipping Hi-MAR-L with larger resolution (i.e., 256 -> 512), and the FID score achieves 1.62, which outperforms MAR-L with 512 resolution by 0.11. The result again validates the effectiveness of exploiting hierarchical autoregressive and modeling interdependency among tokens. We will add this.

Q1: Discussion on VAR and comparisons to "VAR w/ diffusion heads"

Thanks. We summarize the contributions of our Hi-MAR against VAR in two points:

1. VAR utilizes the low-resolution visual tokens directly to guide the next phase prediction, which would cause training-inference discrepancy as discussed in section 3.2. During training, VAR takes the ground-truth low-resolution tokens as the condition for the next phase. Since no ground-truth token is available at inference, VAR has to take the predicted noisy low-resolution tokens as the condition, resulting in training-inference discrepancy. Instead, to mitigate such discrepancy, Hi-MAR takes the conditional tokens estimated from the Hi-MAR Transformer of low-resolution visual tokens to trigger the second phase. As shown in Table 4, when replacing the pivots of visual tokens (similarly used in VAR) with our conditional tokens, the FID score clearly improves from 2.28 to 2.07, which validates the effect of the conditional tokens that mitigate training-inference discrepancy.
2. VAR models multi-scale probability distributions via a shared Transformer without additional guidance. This way leaves the inherent different peculiarities of each scale in autoregressive modeling not fully exploited, resulting in a sub-optimal solution for multi-scale token prediction. In contrast, our Hi-MAR incorporates a scale-aware Transformer block that provides scale guidance to the Transformer tailored to each phase.

Moreover, as suggested, we experimented by implementing VAR with diffusion heads, and "VAR w/ diffusion heads" (FID: 2.67) manages to outperform VAR-d16 (FID: 3.30). Nevertheless, the performance of "VAR w/ diffusion heads" (FID: 2.67) is still inferior to our Hi-MAR (FID: 1.93), which demonstrates the effectiveness of hierarchical autoregressive modeling. We will add all discussions in revision.

Q2: Would further stages yield additional improvements

Appreciate this comment. We experimented by stacking one more stage (64x64 resolution) ahead of the low-resolution stage (128x128 resolution), and the FID score only fluctuates within the range of 0.03. We speculate that the low-resolution stage (128x128 resolution) has already provided sufficient global structure guidance for the next high-resolution stage. The use of an additional stage (64x64 resolution) might introduce unnecessary/redundant global structure information. We will discuss this in the revised version.

Q3: Autoregressive relationship of Figure 1 (a)

Thanks. To be clear, Figure 1(a) correctly illustrates the left-to-right autoregressive relations among the image token sequence. That is, each predicted token at position i (see the bottom output sequence) can only be emitted conditioned on the previous input tokens at positions less than i (see the top input sequence). We will clarify this in revision.

Q1: Add a new row of "Pivots + MLP-based diffusion heads" in Table 4

Appreciate this comment. As suggested, we conducted experiments by including a new ablated run of "Pivots + MLP-based diffusion heads" in Table 4. This ablated run enables hierarchical modeling with shared MLP-based diffusion heads, without using any additional Diffusion module. As shown in this table, the FID of this new ablated run (the third row) manages to outperform MAR (the first row) by 0.17, which clearly validates the effectiveness of the hierarchical modeling. We will add the discussion in revision.