Continuous Visual Autoregressive Generation via Score Maximization

We sincerely appreciate the Reviewer's time and efforts in reviewing our work. We provide discussions about the concerns as follows.

In Table 1, it is essential to show the results of MAR and GIVT. They are two important baselines to compare.

We fully agree with the Reviewer and will include comparisons with ​MAR and ​GIVT in the main table.

In Figure 2, the speed for MAR maybe not correct. To my knowledge, the speed of MAR in Figure 2 is slower than MAR paper. Could you provide more details about inference settings?

The difference in reported speeds arises from differing evaluation protocols:

• In the MAR paper, the authors measure inference time using the ​maximum batch size and report the ​average time per image, which benefits from parallel computation.

• In our experiments, we follow the more common practice of benchmarking with a ​batch size of 1 to reflect real-world deployment scenarios where latency matters.

Under the setting of batch=1, MAR’s diffusion process incurs significant decoding latency due to its iterative nature. While increasing the batch size improves MAR’s throughput, EAR remains approximately ​2× faster than MAR under maximum batch size conditions.

This paper only shows the comparisons on ImageNet 256x256 conditional generation. It is better to show more results to assess the proposed method, like ImageNet conditional generation.

We appreciate the Reviewer’s valuable suggestion. We are currently conducting experiments on ​ImageNet 512×512 conditional generation to further validate the effectiveness of our method. These results will be included in the final version of our paper.

The ablation studies are insufficient. It is better to investigate more settings, such as the choice of random noise and mask ratio.

We thank the Reviewer for pointing out these concerns. Regarding the mask ratio, we directly adopted the range of [0.7, 1] from MAR, without further tuning. Regarding the random noise, we empirically observe that uniform noise yields slightly better performance compared to Gaussian noise. Specifically, for EAR-B: w/o CFG, FID increases from ​5.46 (uniform) to ​5.74 (Gaussian); w/ CFG, FID increases from ​2.83 (uniform) to ​2.89 (Gaussian). We will ablate this choice in the revised manuscript.

We sincerely appreciate the Reviewer's time and efforts in reviewing our work. We provide discussions about the concerns as follows.

While the paper mentions that “GIVT is confined to the pre-defined family of Gaussian mixtures,” this is not necessarily a drawback. Instead, it may be the result of deliberate selection by the GIVT authors after thorough comparisons. The paper fails to explain why this limitation is a concern and to motivate its ideas.

We appreciate the reviewer’s thoughtful feedback. While we acknowledge that the Gaussian mixture assumption in GIVT may indeed represent a deliberate and well-justified choice for modeling continuous tokens, we respectfully disagree with the assertion that this assumption "is not necessarily a drawback".

The fundamental limitation arises when using likelihood maximization to train continuous autoregressive models: the predictive distribution must be constrained to a predefined parametric family (e.g., Gaussian mixtures) to enable tractable likelihood estimation. While Gaussian mixtures may offer the best fit within this constrained framework, this approach inherently restricts the model’s ability to represent arbitrary target distributions—a limitation that cannot be fully mitigated by the choice of distribution family.

In contrast, our proposed EAR framework circumvents this challenge through its inherently expressive architecture. By implicitly defining the predictive distribution via its sampling process, EAR avoids restrictive parametric assumptions and enables automatic learning of complex data distributions. We will clarify this distinction more explicitly in the revised manuscript to better motivate our method.

The authors claim that continuous AR is superior to discrete AR, but the performance of their proposed EAR (Table 1) does not consistently support this assertion. For instance, EAR underperforms compared to MAGVIT-v2. Moreover, the training cost of EAR is significantly higher than existing approaches: EAR requires 800 training epochs, whereas VAR only needs 200 to 350 epochs.

We appreciate the reviewer’s attention to the empirical results. Our claim regarding the advantages of continuous AR modeling is primarily supported by the comparison in ​Figure 3, where EAR outperforms its discrete counterpart (using the VQ tokenizer in [1]). This demonstrates the benefits of continuous modeling when controlling for other architectural factors.

Regarding MAGVIT-v2, we acknowledge that EAR currently uses a ​three-year-old continuous tokenizer architecture, which may limit its performance compared to state-of-the-art discrete tokenizers. We hypothesize that integrating a ​more advanced, fine-grained continuous tokenizer could better showcase the potential of continuous AR modeling.

As for training efficiency, we agree that EAR currently requires more epochs due to the ​inherent complexity of learning continuous distributions without restrictive parametric assumptions. Future work could explore optimizations to reduce training costs while retaining EAR’s expressiveness.

[1] High-Resolution Image Synthesis with Latent Diffusion Models.

The experiments in pages 7-8 mainly include obvious experiments or hyperparameter tuning studies, without exploring more critical aspects, for example, temperature or random noise in the MLP generator.

We thank the Reviewer for pointing out these concerns. Regarding the temperature, we have tuned it on EAR-B and found that τ=0.98 works better than other choices. Regarding the random noise, we have conducted experiments comparing uniform noise and Gaussian noise, and observed that uniform noise leads to marginally better performance. We will include these ablation experiments in the revised manuscript.

We sincerely appreciate the Reviewer's time and efforts in reviewing our work. We provide discussions about the concerns as follows.

The ablation for masked autoregressive should be provided with causal and full attention like setting in MAR.

We thank the Reviewer for raising this concern. We will enrich the ablation studies and provide comparisons between causal and full attention.

The experiment in table 1 lacks of comparison with continuous AR like [1], [2]. The paper states that in [2] GMM could limit expressivity line 155 2nd column. However, the performance of proposed method still lag behind GIVT. The author should include the performance of [1] and [2] in the main table and make a comprehensive comparison.

We will follow the suggestion and include comparisons with continuous AR models (MAR and GIVT) in the main table. We would also like to respectfully clarify that our method (EAR) does not lag behind GIVT in performance. Specifically:

• Base-scale models: GIVT (304M params) achieves an FID of ​3.35, while our EAR-B (205M params) achieves ​2.83.

• Large-scale models: GIVT-L (1.67B params) achieves an FID of ​2.59, while our EAR-H (937M params) achieves ​1.97.

Why the random noise in MLP is uniform but not normal. Why uniform is better choice ?. The author should ablate this choice.

We empirically observe that uniform noise yields slightly better performance compared to Gaussian noise. Specifically, for EAR-B: w/o CFG, FID increases from ​5.46 (uniform) to ​5.74 (Gaussian); w/ CFG, FID increases from ​2.83 (uniform) to ​2.89 (Gaussian). We will report this comparison in the revised manuscript.

4. The inference process is significantly affected by variance σ in section 5.4. This could be similar to Truncated trick in Big GAN. How the author think about it?

We would like to clarify that Section 5.4 actually presents results using an MSE loss function (rather than our proposed EAR method), which can be interpreted as modeling continuous token distributions with Gaussian distributions. In this setup, the standard deviation σ is specified during inference.

Our experiments show optimal performance occurs around σ=0.4, likely because this value best matches the inherent variance of the continuous token distribution. The reviewer's analogy to BigGAN's Truncation Trick is indeed insightful - they share the commonality of potentially trading sample diversity for sample quality. The Truncation trick operates on latent space sampling, while the variance adjustment operates more directly on the predicted distribution's shape.