TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generation

We sincerely thank the reviewer QM2g for your insightful suggestions, and will add your advice to the camera-ready version. We appreciate your crucial questions, including inference efficiency, and we have provided analysis to address them below.

1. [W1] Computation-Performance Tradeoff of Different Designs

We evaluate our method with $N=2,4$ on the GPU Nvidia A800-SXM4-80GB. Below is the ablation of different designs and corresponding gains in inference time and peak memory utilization. Because the upper boundary of the 80G memory, BoN with $N=8$ requires two inferences and doubles the consumption.

While these methods introduce some computational overhead, the impact is minimal. The adaptive batch size schedule, for instance, only adds a slight expense because it primarily increases the sample count when the sequences are short and the tokens are limited. Furthermore, by applying low-rank PCA (pca_lowrank) for dimensionality reduction before K-Means and utilizing a compact reward model on bf16, we effectively control filtering costs, ensuring that total resource consumption remains acceptable.

Compared with BoN under similar time boundaries, for example, BoN with $N=4$ and Ours with $N=2$, there is a clear advance in GenEval Score, indicating the effectiveness and efficiency of our designs.

2. [W2] Other Benchmark Results

We acknowledge that GenEval's evaluation primarily focuses on text-image alignment and does not offer a comprehensive assessment of the overall generation quality. To provide a more complete result, in Table 6 of the Appendix, we also report on metrics related to image quality, such as ImageReward and HPS, in addition to GenEval. These results are presented in the table below.

Furthermore, we have incorporated test results from T2I-Compbench++ [1]. This benchmark assesses attributes such as color and shape, which, to some degree, reflect the quality of the generated images.

We also introduced a user study on Google Forms, asking participants to select the best image from those generated by different methods based on three criteria: Image Quality, Image Rationality, and Consistency with the Prompt. This study consisted of 15 sample groups, and we collected 21 responses, resulting in 315 total votes. The percentages of votes received by each approach are summarized in the table below.

In addition to receiving high marks for text consistency, these results also demonstrate that our method holds a competitive advantage over other approaches in terms of image quality and image rationality.

3. [W3] Worse Performance with Higher Resampling Frequency

We discuss this phenomenon in Section 5.3 of the main submission. As shown in Figure 5, increasing the frequency of resampling does not always lead to performance improvements. This is because, as illustrated in Figure 5(b), the images decoded during the VAR generation process may not be effective representatives of the final generated image's quality. Consequently, resampling at early stages can lead to the erroneous selection of low-potential samples, thereby limiting the upper bound of scaling.

4. [Q1] Performance with only Clustering-Based Diversity Search

In the table above, we present the ablation, including results of using only Clustering-Based Diversity Search for the cases where $N=2$ and $N=4$. Without resampling, we cannot filter intermediate states in the subsequent steps. Therefore, we cluster directly down to $N$ samples from the outset. Specifically, the adaptive batch size schedule used is {8N, 8N, 6N, 6N, 6N, 1N, 1N, 1N, 1N, 1N}.

As shown, the clustering process effectively selects diverse intermediate states, ensures sample diversity, and provides more possibilities for the final selection, resulting in a significant improvement over the standard Best-of-N (BoN) method.

Reference

[1] T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation. Huang, Duan, et al.

Dear Reviewer QM2g,

Thank you for your review and for acknowledging our work as the first general TTS framework for VAR models with excellent clarity.

Following your insightful concerns, especially regarding efficiency, we have provided a detailed rebuttal. In summary, we have:

• Added a direct comparison with Best-of-N under a matched inference speed / compute budget. The results confirm that our method's superiority comes from its efficient design, not just a larger sampling budget.
• Included results on additional qualitative metrics to better demonstrate the improvement in image detail, addressing the limitations of GenEval.
• Provided a deeper analysis of the resampling frequency phenomenon and included the requested ablation study on using the clustering alone.

We believe these additions thoroughly address your concerns about the fairness and efficiency of our framework. As the discussion period is drawing to a close, we would be grateful to know if our response has resolved your questions.

Thank you for your time and consideration.

Sincerely,

Authors of TTS-VAR

Thanks for providing additional details for the paper. Most of my concerns are addressed by the authors and I would like to maintain my score.

We appreciate the reviewer 4YxR’s constructive feedback and will add your advice to the camera-ready version. We are encouraged that you acknowledged the originality of our method and the rigorous experiments demonstrating consistent gains across various metrics. We answer your questions below.

1. [W1, Q1] Output distribution diversity

The concept of output diversity can be divided into two components. The first is the diversity of the final output results, which we term Final Semantic Diversity. The second is the structural diversity among the intermediate states during the inference process, which we refer to as Intermediate Structural Diversity.

1.1 Final Semantic Diversity

In reference [1], the authors test IS and FID scores under the Class-to-Image (C2I) task on ImageNet. Since IS is based on the classification of generated images, it is not suitable for the T2I task. In addition, FID requires the reference distribution of real images, which is rarely adopted in open-domain T2I tasks.

Therefore, we follow [2] to adopt an evaluation on the CLIP latent space, which is more in accordance with the T2I task. The metric CLIP-Div, as formulated below, with CLIP encoder $f_\theta$ and $k$ samples ${\\{x_0^i\\}}_{i=1}^k$, calculates the difference between samples with the same prompt and directly measures the diversity on text-aligned features.

$\\text{CLIP-Div} \\left( \\{\\mathbf{x}\_0^i\\}\_{i=1}^k \\right) := \\sum\_{i=1}^k \\sum\_{j=i}^k \\frac{2}{k(k-1)} \\left\\|f\_{\\theta}(\\mathbf{x}\_0^i) - f\_{\\theta}(\\mathbf{x}\_0^j) \\right\\|\_2^2$

Though the metric is under a relatively small range, we can judge the diversity by comparing the relative differences between different methods, as shown in the table below. Generally, with the growing $N$, the generated results gain a higher possibility of convergence to adjacent superior features, resulting in a lower CLIP-Div score. However, the difference is quite small, indicating that our method enhances the performance with little sacrifice of diversity.

1.2 Intermediate Structural Diversity

We have visualized the generation process and analyzed the role of clustering in filtering these structural features. We found that clustering effectively groups samples with similar structures, thereby selecting for different structural characteristics and increasing the likelihood that various samples advance to later filtering stages.

Take the intermediate process for the prompt, "Three origami cranes in white, gold, and blue are perched on a black lacquered shelf above a blooming bonsai tree," as an example. During generation, issues such as incorrect numbers of cranes, repeated colors, or wrong color sequences may arise. We observed that clustering can efficiently group samples with similar flaws into one category while grouping the generally correct samples into others. This filtering preserves representative samples, ensures structural diversity, and maximizes the probability that a correct sample exists for the subsequent reward-based filtering.

Due to the inability to submit images during the rebuttal phase, we will incorporate visualizations of this filtering process in the camera-ready version of the paper.

2. [W2, Q2] Computation-Performance Tradeoff of Different Designs

We evaluate our method with $N=2,4$ on the GPU Nvidia A800-SXM4-80GB. Below is the ablation of different designs and corresponding gains in inference time and peak memory utilization. Because the upper boundary of the 80G memory, BoN with $N=8$ requires two inferences and doubles the consumption.

Our method takes less time and memory to achieve comparable or even superior performance, for example, BoN with $N=4$ and ours with $N=2$. This proves the effectiveness and efficiency of TTS-VAR.

Building upon Table 3 in the main submission, we have also included an analysis of the computation-performance tradeoff when applying resampling at different scales. As the results show, increasing the number of resampling steps incurs additional inference costs.

3. [Q3] Limited Improvements on Other Indicators

In our view, these limited improvements can be attributed to two main factors.

1. Different metrics possess inherent biases due to their distinct training data. Although most metrics (excluding Aesthetic) evaluate text-image alignment to some degree, they have different tendencies and priorities. This is evidenced by the inconsistent rates of improvement across various metrics. In the majority of our experiments, we exclusively used ImageReward to guide the selection process. This method, however, primarily favors cases that align with the preferences of the ImageReward model itself. Consequently, while we observed notable improvements on GenEval, which focuses purely on text-image consistency, the gains on other metrics with specific biases, such as HPS and CLIP, were less pronounced.

2. The metrics exhibit varying levels of numerical sensitivity. As shown in Table 9 of the Appendix, even when using HPS as the reward for selection, the corresponding improvement in the HPS metric itself is not substantial. This is because, while all metrics are designed on the principle that a higher score indicates better quality, their specific numerical ranges and the magnitude of improvement represented by a score increase are not uniform across metrics.

We also introduced a user study on Google Forms, asking participants to select the best image from those generated by different methods based on three criteria: Image Quality, Image Rationality, and Consistency with the Prompt. This study consisted of 15 sample groups, and we collected 21 responses, resulting in 315 total votes. The percentages of votes received by each approach are summarized in the table below.

As shown, though some metrics did not show significant improvements, the results of the user study keenly capture the enhanced performance of our method across all three criteria, particularly in terms of consistency with the prompt.

4. [Q4] Efficiency of Investing in Different Architectures

Overall, the benefits of scaling on different model architectures are not yet clearly established. Our work aims to provide a feasible solution for the alternative architecture VAR, and we are willing to share our perspective on this matter.

1. Different architectures possess distinct characteristics and advantages. Diffusion models, benefiting from their progressive denoising process and modeling of continuous latent spaces, tend to achieve superior generative quality. In contrast, the VAR architecture more closely resembles that of large language models, which may be more conducive to unifying generation and understanding, like [3]. Depending on the specific goal, scaling efforts on a particular architecture might yield distinct advantages.

2. We believe that current scaling strategies for diffusion models are still in their early stages. In contrast to the significant improvements our method shows over Best-of-N (BoN), the search method used in reference [1] offers only a marginal improvement over Random Search. We hypothesize that this is because the multi-step denoising process in diffusion models creates a much larger search space, likely requiring more sophisticated methodological designs. Therefore, based on the current works, it is difficult to make a direct judgment on which approach is definitively superior.

Reference

[1] Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps. Ma, Nanye, et al.

[2] A General Framework for Inference-time Scaling and Steering of Diffusion Models. Singhai, Horvitz, et al.

[3] VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning. Zhuang, Xie, et al.

Dear Reviewer 4YxR,

Thank you for your thorough review and positive feedback. We are very pleased you recognized our work's rigorous experiments, clarity, and novelty.

Following your excellent questions, we have provided a detailed rebuttal. In summary, we have:

• Added a quantitative analysis of output diversity and an analysis of visualizations, as you suggested.
• Included a detailed compute-performance tradeoff analysis for our key design choices.
• Provided further discussion on the behavior of different evaluation metrics and efficiency between diffusion and VAR.

We believe these additions directly address your main questions. As the discussion period is drawing to a close, we would be grateful to know if our response has resolved your questions.

Thank you for your time and consideration.

Sincerely,

Authors of TTS-VAR

We thank Reviewer XWDr for the insightful feedback. We will incorporate the suggestions in the camera-ready version and appreciate the recognition of our work. Our responses follow.

1. [W1] Contribution of Adaptive Batch Sampling

To illustrate the motivation for adaptive batch sampling, we compare it with fixed batch sizes below. While our method has higher initial FLOPs and memory usage, its computational growth is limited, and peak memory does not exceed that of a fixed batch size of 1. This design expands the search space at a minimal cost and prevents out-of-memory (OOM) errors.

• FLOPs (TFLOPS)

• Memory (GB)

We will incorporate the curves of other fixed batch sizes into the figure in the camera-ready version.

2. [W2, Q1] Comparison with BoN & Computation-Performance Tradeoff

1. The VRAM limit is a critical bottleneck in test-time scaling, especially in VAR models with quadratic memory growth. A key challenge, therefore, is how to effectively expand the search space under severe memory constraints, which is also the primary motivation for our design of Adaptive Batch Sampling. While a BoN approach with $N=8$ should, in theory, perform greater than or equal to our method with $N=1$, it requires ~8x the VRAM. In practice, this leads to Out of Memory (OOM) errors even on an 80GB GPU (we used multi-GPU parallelism for our $N=8$ experiments).

2. In terms of efficiency, our method's time overhead is minimal compared to increasing BoN's sample count. We tested the time and corresponding memory footprint for each component on the NVIDIA A800-SXM4-80GB GPU. Owing to the OOM error, BoN with $N=8$ requires two inferences and doubles the consumption. As shown, our method not only surpasses BoN with $N=8$ when ours is with $N=2$ in performance, but also shows a significant advantage in both time and memory consumption.

3. [W3] Design Discussion

3.1 Effectiveness of Clustering

The primary objective of clustering is not to find higher-quality structures, but to preserve the diversity of crucial structural features, especially when the batch size is reduced due to VRAM limits. This approach ensures that a broader range of possibilities is available for evaluation in the subsequent resampling stage, enabling the selection of the most promising samples from a more diverse pool.

Take the intermediate process for the prompt, "Three origami cranes in white, gold, and blue are perched on a black lacquered shelf above a blooming bonsai tree," as an example. During generation, issues such as incorrect numbers of cranes, repeated colors, or wrong color sequences may arise. We observed that clustering can efficiently group samples with similar flaws into one category while grouping the generally correct samples into others. This filtering preserves representative samples, ensures structural diversity, and maximizes the probability that a correct sample exists for the subsequent reward-based filtering.

Due to the inability to submit images during the rebuttal phase, we will incorporate visualizations of this filtering process in the camera-ready version of the paper.

3.2 Using Reward Model as Metric Indicator

We acknowledge that using the reward score itself as an evaluation metric can indeed lead to a biased assessment. This is why we stated at the beginning of Section 5 that we use GenEval, which is an objective metric without bias on a specific indicator, as our main metric. We also report a detailed breakdown of its sub-metrics in the Appendix. For a more comprehensive evaluation, we also present results on another mainstream benchmark, T2I-CompBench++ [1].

4. [W4] Other Experimental Results

4.1 Ablation on Clustering

We compare our clustering method with a random drop process in the table below. As shown in the table below, even though resampling is in place to help ensure image quality, the effectiveness of scaling is significantly diminished when clustering is removed. This shows clustering is an effective selection mechanism.

4.2 Results on DPG-Bench

We evaluate the performance of the baseline Infinity model, the Best-of-N (BoN) method, and our approach on the DPG-Bench, specifically for the case where $N=4$. For each prompt, we generated four corresponding images for evaluation using seeds 0 through 3. The results are presented in the table below.

4.3 $\lambda$ > 10 Results

We conduct an ablation study on $\lambda \in \{50, 100\}$ for both $N=2$ and $N=4$, with the complete results presented in the table below. The rows labeled $\lambda=-$ represent the baseline results from BoN.

A higher $\lambda$ promotes selecting better intermediate states, but an excessive value (e.g., 100) can hinder performance by prematurely pruning potential samples. The optimal setting varies with $N$: $\lambda=10$ performs best for $N=2$, whereas $\lambda=50$ achieves slightly better results for $N=4$.

5. [W5, Q2] Notations and Words

We apologize for the incorrect notations on lines L108 and L158 and will correct them in the camera-ready version. Regarding the variable $N$ on lines L137-138, it represents the same as the $N$ in Table 1—the final number of generated samples. We will add a clarification on lines L137-138 to prevent any confusion.

As for using clustering instead of pruning and selection, this decision is based on our observation of structural features that emerge early in the generation process. We found that these features can be effectively extracted, and by using clustering, we can preserve a diverse set of representative structures. This, in turn, provides a richer pool of candidates for the subsequent selection phase. Furthermore, direct pruning and selection are difficult at these early stages due to the lack of consistency between the early intermediate images and the final outputs. For these reasons, we chose and named our design 'clustering'.

Reference

[1] T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation. Huang, Duan, et al.

Dear Reviewer XWDr,

Thank you for your highly detailed and critical review. We are encouraged that you recognized our work as an "interesting and promising direction," and we have worked extensively to address the numerous concerns you raised.

In our rebuttal, we have made substantial additions to address your concerns. In summary, we have:

• Strengthened our core motivation and comparisons: We've added compute-performance tradeoffs and new analyses to justify our adaptive approach better.
• Clarified and justified key designs: We provided deeper justification for our clustering method and discussed the potential evaluation bias from the reward model.
• Added extensive results: We reported more ablation studies and results on the DPG benchmark.

We hope these substantial revisions have addressed the core issues you identified. As the discussion period is drawing to a close, we would be very grateful to know if our response and revisions have helped clarify our contributions or if you have any further questions.

Thank you for your time and consideration.

Sincerely,

Authors of TTS-VAR

Thank you for your positive feedback and for raising your score. We are glad to see that our rebuttal has addressed your questions. As promised, we will incorporate the improvements, including extra indicators of average FLOPs and memory, in the updated version. Your suggestions have helped us improve the clarity and presentation of our work.

We sincerely thank the reviewer wUiv for positive feedback and insightful comments, and will add your advice in the camera-ready version. We are particularly encouraged that you found our work to be well-motivated and recognized its potential to unlock a wide range of real-world applications. We answer your questions below.

1. [W1, Q1] More Results for validation

1.1 Other Model

Our test-time scaling framework is designed based on VAR-based Text-to-Image (T2I) models. Since Infinity is currently the only provider of an open-source T2I model with a VAR architecture, in addition to the 2B Infinity model used as base model in the paper, we selected another latest 8B model (resolution 512) to validate the effectiveness of the framework. Specifically, since the generation process for the 512x512 resolution involves only 10 scales, we use an adaptive batch size schedule of {8N, 8N, 6N, 6N, 6N, 4N, 2N, 2N, 1N, 1N}. Clustering is applied to the scales in the range [2, 5], and resampling is performed on the scales in the range [6, 8].

The experiments were conducted using the same settings as in Table 1 of the main submission and Table 6 in the Appendix. The results are shown in the table below.

As shown in the table, our framework also achieves excellent results on this model. Notably, when $N=4$, our method even surpasses the 0.80 threshold, delivering exceptional performance. We look forward to the emergence of more text-to-image models based on the VAR architecture, which will enable us to further validate our approach in the future.

1.2 Other Benchmarks

Recognizing the limitations of a single benchmark in fully assessing a method's effectiveness, we performed further evaluations on the Infinity-2B model using T2I-Compbench++ [1], a prominent benchmark in text-to-image generation. The corresponding results are detailed in the following table.

As displayed, our approach yields substantial performance gains over the baseline Infinity model. Echoing the results on GenEval, our method with just $N=2$ outperforms BoN even when it uses $N=8$. While our method does not surpass SD3 on this benchmark, it significantly narrows the performance gap, achieving this result on a considerably smaller model.

2. [W2, Q2] Computation-Performance Tradeoff of Different Designs

We evaluate our method with $N=2,4$ on the GPU Nvidia A800-SXM4-80GB. Below is the ablation of different designs and corresponding gains in inference time and peak memory utilization. Because the upper boundary of the 80G memory, BoN with $N=8$ requires two inferences and doubles the consumption.

Our method takes less time and memory to achieve comparable or even superior performance, for example, BoN with $N=4$ and ours with $N=2$. This proves the effectiveness and efficiency of TTS-VAR.

We display the comparison between Infinity+Ours ($N=2$) and SD3, because of similar memory consumption and GenEval score under the same resolution. As shown above, a 2B Infinity model with our test-time scaling achieves similar performance as an 8B SD3 model with less inference time (2.51s vs 13.24s).

3. [Q3] Human Feedback

To gather user feedback, we selected a small set of 15 image groups and compiled them into a questionnaire on Google Forms. For each group, the images are randomly shuffled, and participants were asked to select the best generated image based on three criteria: Image Quality, Image Rationality, and Consistency with the Prompt. In total, we collected 21 completed questionnaires, resulting in 315 sets of survey data for each indicator. The percentages of votes received by each approach are summarized in the table below.

As demonstrated, the results generated by our method received the most votes across all three criteria, highlighting its effectiveness, particularly regarding its consistency with the prompt.

4. [W3, Q4] Hyperparameters

In K-Means, we just set the next batch size $b_{i+1}$ in the adaptive descending batch sizes as the number of centers, to find $b_{i+1}$ different structure categories. For PCA, we only select the first major component, as we utilize it for dimension reduction only. In all benchmarks, we use seeds from 0 to 4 for different results. For experiments requiring multi-GPU, like $N=8$, we set the seed from 100*process_index to 100*process_index+4 for each device.

About other parameters related to Infinity, we just keep them the same as the original public settings.

Reference

[1] T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation. Huang, Duan, et al.

Dear Reviewer wUiv,

Thank you once again for your constructive review. We are very encouraged that you recognized our work's solid gains and clear methodology.

Following your valuable suggestions, we have updated our paper and provided a rebuttal. In summary, we have:

• Expanded our evaluation with new model and benchmarks as you suggested.
• Added concrete compute-performance tradeoffs and all hyperparameters for reproducibility.
• Conducted a human study that confirms our method's qualitative advantage.

We believe these additions have thoroughly addressed your concerns. As the discussion period is drawing to a close, we would be grateful to know if our response has resolved your questions.

Thank you for your time and consideration.

Sincerely,

Authors of TTS-VAR