FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction

Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments, and here are the answers to your concerns.

W1: Motivation.
Residual modeling cannot achieve the flexibility of generating, as it relies on a weighted sum of all scales. This dependence requires a consistent scale design during both training and inference. Additionally, residual modeling mandates using every scale in the weighted sum, thus preventing step dropping during training. We analyze the differences between two structures in L50-L52: residuals at different scales often lack semantic continuity, and this implicit prediction approach may limit the model’s ability to represent diverse image variations. Furthermore, the ablation study in Tab. 4 experimentally demonstrates that predicting ground truth is an effective approach.

Q1: VAR performance clarification.

1. We perform evaluations using the official VAR model, and the results presented in Tab. 2 align with the performance reported by the VAR team on GitHub. Additionally, concurrent work M-var [36] also verifies the performance of VAR (Tab. 2 in [36]). Therefore, at an equivalent inference budget, FlexVAR demonstrates superior performance compared to VAR.
2. Step dropping is to enable the model to adapt to varying steps, which does not result in significant performance gains, as discussed in W1.

Q2: VQVAE performance.
Due to limited training resources, we train FlexVAR-VAE with fewer epochs and smaller batch sizes. We observe its reconstruction quality slightly inferior to VAR-VAE. Thus the improvement is not due to the quality of the discrete tokens, using a more robust FlexVAR-VAE might further improve the quality of generated images.

Q3: Scaling capacity

As shown in Tab. 2, the FID trends when scaling up FlexVAR and VAR are similar (VAR: 3.55 -> 2.95 -> 2.33). Additionally, according to the FID formula, the decrease in FID score is non-linear, indicating that it becomes increasingly challenging to achieve further reductions as the score lowers:

Where $\mu$ and $\Sigma$ represent the mean and covariance matrix of two distributions, respectively, and $\text{Tr}(\cdot)$ denotes the trace of the matrix.

Q4: Explication of aspect ratio generation.
We control the height and width for each scale through approximate rounding. e.g., to generate an $H × W$ image, the corresponding VAE latent feature size is $h × w$, where $h = \frac{H}{8}$, $w = \frac{W}{8}$. We use VAR's default 10-steps ($K =${$1, 2, 3, 4, 5, 6, 8, 10, 13, 16$}) to determine the size of each scale. Consequently, the H × W image corresponds to ten scales with sizes {$(\text{int}(h \times \frac{i}{16}), \text{int}(w \times \frac{i}{16}))$}$_{i \in K}$. We will add more detailed explanations in the final version.

Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments, and here are the answers to your concerns.

W1: Concern about the flexibility.
Residual modeling cannot achieve the flexibility of generating, as it relies on a weighted sum of all scales. This dependence requires a consistent scale design during both training and inference. Additionally, predicting residual does not allow for dropping scales because each scale is utilized during the weighted sum stage. We analyzed the differences between residual modeling and GT modeling in L50-L52: residuals at different scales often lack semantic continuity, and this implicit prediction approach may limit the model’s ability to represent diverse image variations.

W2: Replacing residual with ground-truth modeling does not yield meaningful performance gains

1. FlexVAR demonstrates higher Inception Scores (IS) across different scales (d16, d20, d24) when compared to VAR. Our FlexVAR enhances IS further by utilizing zero-shot inference with more steps.
2. Precision and Recall metrics do not accurately reflect image quality; i.e., the Precision score of VAR across different scales (VAR-d16, VAR-d20, VAR-d24) shows a decreasing trend: 0.84 -> 0.83 -> 0.82. Therefore, critiquing the ground-truth modeling paradigm based on IS, Precision, and Recall is not accurate.

W3: FlexVAR’s image visual quality lags behind VAR’s
Due to NeurIPS guidelines prohibiting anonymous links, we are unable to present side-by-side visual comparisons at this time. However, we have included a selection of generated samples in the supplemental material to showcase the visual quality of FlexVAR's images.

W4: FlexVAR’s scalability isn’t fully demonstrated
FlexVAR at scales d16, d20, and d24 outperforms the corresponding VAR models, which sufficiently demonstrates the scalability. The performance of FlexVAR-1B being weaker than VAR-d30 (2B parameters) should not be seen as our weakness. Due to limitations in computational resources, we are currently unable to train FlexVAR-d30. Therefore, to ensure a fair comparison, we only present models with parameters less than or equal to 1B, as explained in Line 198-199.

Q1: Performance of the densest scale and sparsest scale
We present the FID and IS of FlexVAR-d24 during inference with 3-steps ({1, 4, 16}), 5-steps ({1,2, 4, 8, 16}), 10, 13, 14, 15, and 16-steps in the table below. The results indicate that using too few inference steps (e.g., 3 steps) results in poor performance, but there is a noticeable improvement in image quality with 6 steps. Both FID and IS achieve optimal values at step 14, followed by a slight decline in performance with more steps.

Q2: Why adopt VAR’s scale configuration by default
We use the same scales as VAR for a fair comparison.

Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments, and here are the answers to your concerns.

W1: Concerns about bias in specific resolution
The randomness inherent in the sampling process during training prevents the introduction of bias towards specific resolutions. Additionally, sampling only includes scales with resolutions $\leq$ 256. This approach does not encompass higher resolutions, thereby ensuring zero-shot inference at those higher resolutions.

W2: Attention mask during inference
During inference, each scale is conditioned on all previous scale tokens. We believe that the information from previous scales is valuable, as it helps prevent error accumulation.

W3: Why shifting the paradigm to GT modeling brings benefits to gFID.
We analyze the differences between two structures in L50-L52: residuals at different scales often lack semantic continuity, and this implicit prediction approach may limit the model’s ability to represent diverse image variations. Furthermore, the ablation study in Tab. 4 experimentally demonstrates that predicting ground truth is an effective approach.

W4: Concerns about the training flops.
We perform sampling at resolutions $\leq$ 256, deliberately avoiding larger scales such as 512. As a result, compared to VAR, no additional computational overhead is introduced. Specifically, while VAR employs 10 fixed steps for sampling, we utilize random sampling within a range of 6 to 10 steps. Consequently, the training FLOPs may vary within a certain range but will not surpass the computational demands of VAR.

Thanks for the rebuttal. Most of my concerns have been addressed. After reading other reviews, I decide to maintain my original score. The reasons are listed as follows:

1. I still have a question why simply training on resolutions smaller than 256, the model is able to generate 512*512 images. I think this part should be discussed.
2. The quality of zero-shot inference on 512*512 is not promising. And generating larger resolution (above 512) images is the main focus point.

Overall, the proposed flexible design provides a new insight into how to generate a high-resolution image with only training on low-resolution ones.

Thank you for the valuable discussion and constructive suggestions.
Autoregressive generation models in the NLP domain inherently possess a degree of zero-shot capability, allowing them to extend to longer sequences beyond those seen during training. However, the design of VAR, specifically its use of residual weighted summation, completely eliminates the model's ability to generalize in a zero-shot manner to larger scales. In our proposed FlexVAR, we address this limitation by removing both residual prediction and residual weighted summation. This modification overcomes the fixed-resolution constraint of VAR and enables image generation at resolutions beyond those encountered during training.
On the other hand, while FlexVAR demonstrates some zero-shot capability for generating high-resolution images, this does not imply that it can outperform methods that are fully supervised and trained directly at the target resolution. In Tab. 3, our evaluation at 512px resolution is conducted in a zero-shot setting, whereas all comparison methods are explicitly trained on 512px images.

Thank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments, and here are the answers to your concerns.

W1 & Q1: Analysis of error-correction
VAR predicts residuals at each step and weights the sum of all step residuals. These residuals are not equivalent to errors, thus there is no theoretical basis for VAR explicitly correcting errors. Both VAR and our FlexVAR can see all previous scale tokens, allowing them to implicitly correct previous errors. From this perspective, predicting ground truth and predicting residuals show no significant difference in error accumulation. Additionally, Tab. 2 demonstrates that predicting ground truth is an effective approach.
To further analyze error propagation, we progressively added Gaussian noise within 1% of the maximum response scale at the early steps (e.g., step {1} or steps {1,2,3}) to simulate the error generation process during inference. The FID results under different noise patterns are shown in the table below. It is evident that our FlexVAR consistently produces better quality than VAR under varying noise patterns.

Note: FID is used for evaluation, where a lower score indicates better performance.

W2: The proposed VAE lack innovation.
We do not emphasize VAE as a fundamental architectural innovation. Our VAE follows the architecture of Llamagen and VQGAN, and the training of VAE is aimed at fitting the architecture that removes residuals. Its training primarily relies on data augmentation methods rather than structural improvements.

W3: The relative inference time favorable for FlexVAR.
There may be some misunderstandings about the relative inference time. We would like to clarify it as follows. Relative time is a time evaluation metric used by VAR, which provides an intuitive reflection of inference time. This metric is not favorable for FlexVAR. Relative time represents the multiple of inference time required in comparison to VAR-d30 (as shown in Tab. 2 caption). i.e., on an A100 GPU, the inference time for VAR-d30 is 0.5s. Therefore, multiplying 0.5 seconds by the corresponding relative time metric yields the actual image generation time for each model.

Q2: Zero-shot inference with more steps.
A 256-resolution image corresponds to a VAE latent feature of 16 × 16, allowing for a maximum of 16 sampling steps, ranging from 1 to 16. We present the FID and IS of FlexVAR-d24 during inference at steps 14, 15, and 16 in the table below, along with results for 6-step, 10-step, and 13-step for comparison. The results indicate that both FID and IS achieve optimal values at step 14, followed by a slight decline in performance with more steps.