LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding

First of all, we sincerely appreciate the detailed review of our paper and the constructive feedback provided. Thank you for highlighting the clarity of our presentation and recognizing the novelty and elegance of our approach to tackle token selection ambiguity. We believe your feedback has significantly strengthened our work. To address the concerns and questions, we have conducted additional experiments and revised the manuscript as follows:

Weakness 1 & Question 1: Finding Neighborhood

To identify the proximity sets A and B of quantized image tokens, we employed both a dynamic approach and precomputation. Specifically:

• Precomputation of $B$: The set $B$ comprises the $k$ nearest tokens in the latent space with $\ell_2$ distance for each quantized image token. Since $B$ is independent of token probabilities (and preceding tokens), it is precomputed and stored as tensors before inference begins.
• Dynamic Calculation of $A$: The set $A$ is a subset of $B$, determined dynamically based on token probabilities and an upper bound $\delta$. This ensures $A$ adapts to the current decoding context.

We have summarized the overall procedure as pseudocode and included it as Appendix D.2 in the revised manuscript.

To evaluate the impact of these computations on overall latency, we measured the time required to compute $A$ during a single inference step and compared it with the forward times for the target model and drafter. Target model is LlamaGen Stage I and randomly selected 1000 caption from MS-COCO 2014 validation set are used. The results are summarized below:

Our analysis shows that the time required to compute $A$ is 32 times smaller than drafter model forward and 9.1 times smaller than target model forward for $k=1000, \delta=0.4$, which is negligible in entire process. Furthermore, the significant speedup enabled by $A$ also confirms the efficiency of our method.

These findings have been added to Appendix F.2, along with further discussion on implementation details. We appreciate your suggestion, which has enhanced the clarity and comprehensiveness of our work.

Weakness 2: Misstatement in Line 105

We acknowledge the incorrect description of EAGLE-2 as a drafter in Line 105. This has been corrected to EAGLE-2 as a base speculative decoding method at Line 105 in the revised manuscript. Thank you for pointing this out, allowing us to rectify this misstatement.

Weakness 3 & Question 3: Generalizability of the Token Selection Ambiguity and LANTERN

Thank you for your valuable suggestion on the generalizability of token selection ambiguity and our method LANTERN to other visual AR models. To demonstrate the generalizability of the token selection ambiguity and our method, we conducted additional experiments on other visual AR models, including LlamaGen Stage II model and Anole [1] with MS-COCO 2017 Validation set.

As shown in the table, like LlamaGen, Anole exhibits low average top-1 and top-10 probabilities, with values of $0.064$ and $0.204$, respectively. However, these probabilities are even lower than those of LlamaGen ($0.206$ and $0.520$), suggesting that Anole faces a more severe degree of token selection ambiguity. This increased ambiguity also affects drafter performance, with Anole’s drafter achieving a test accuracy of only $27.60\%$, compared to $38.80\%$ for LlamaGen.

By extending the analysis to Anole, we confirm that the token selection ambiguity is a recurring challenge across different models, with varying degrees of severity. This further substantiates our claim that token selection ambiguity is a critical factor influencing the performance of speculative decoding. The results have been incorporated into the Figure 2(b) and (c) in the revised manuscript.

Thank you for the detailed feedback and experiments. I think most of my concerns have been addressed, so I will raise my score to an Accept.

We sincerely thank you for your thoughtful feedback and constructive suggestions. We appreciate your recognition of the motivation and soundness of our approach, as well as your valuable insights on evaluation metrics and image quality. Your comments provide important guidance, and we address them in detail below.

Metrics for Evaluating Basic Generation Quality (Precision and Recall)

We appreciate your suggestion to include precision and recall (P/R) metrics to better understand the quality and diversity trade-offs in our approach. To this end, we have evaluated LANTERN with varying values of the hyperparameters $\delta$ and $k$, and compared its performance to the baseline. The results are as follows:

• Precision / Recall for standard AR: $0.4781$ / $0.5633$

• LANTERN ($\tau=1.0$)

  Precision / Recall	$\delta=0.05$	$\delta=0.1$	$\delta=0.2$	$\delta=0.4$
  $k=100$	$0.4867$ / $0.5389$	$0.4796$ / $0.5303$	$0.4789$ / $0.5140$	$0.4825$ / $0.4946$
  $k=300$	$0.4856$ / $0.5367$	$0.4834$ / $0.5231$	$0.4894$ / $0.4901$	$0.4895$ / $0.4719$
  $k=1000$	$0.4865$ / $0.5334$	$0.4869$ / $0.5172$	$0.4880$ / $0.4888$	$0.4909$ / $0.4497$

The results demonstrate that LANTERN achieves comparable or slightly improved precision relative to the baseline across various settings, highlighting its ability to maintain the quality of generated images. While recall decreases slightly with increasing $\delta$, this indicates that the quality of individual images is preserved, though there is a modest reduction in diversity.

The slight decrease in recall can be attributed to token selection ambiguity in the drafter. When the drafter is not optimally trained, it may struggle to produce sufficiently diverse or accurate predictions. Consequently, increasing the acceptance probability enhances acceleration but can lead to reduced image diversity. Nevertheless, this trade-off does not significantly impact overall image quality, as evidenced by consistent FID, Precision, and HPS v2 scores (in the next section). These precision and recall results illustrate that our method effectively balances quality and diversity across different hyperparameter configurations.

Overall, these evaluations provide further evidence that LANTERN maintains generation quality within acceptable bounds while achieving substantial speed improvements.

Incorporation of Modern Image Quality Metrics (PickScore and HPS v2)

To further quantify the aesthetic quality of generated images, we have evaluated our approach using HPS v2 [1], as suggested. Evaluating PickScore [2] was challenging due to its high time requirements, which involve measuring Elo ratings or win rates for each individual samples to properly evaluate it. Both PickScore and HPSv2 are metrics for assessing human preference; however, HPSv2 reportedly aligns more closely with human preference scores. Therefore, we chose to use only HPSv2 in our evaluation. This modern image quality metrics provide a more holistic measure of perceptual quality and aesthetic appeal. The updated findings are as follows:

| HPS v2

These evaluations confirm that while there is a slight reduction in aesthetic quality compared to the baseline, the trade-off is well-justified by the significant improvements in generation speed. This aligns with the intended design of LANTERN, which emphasizes efficiency while preserving acceptable quality.

The results have been incorporated to our main table (Table 2 in the revised manuscript) and full results have been appended to Appendix H. The added P/R analysis complements our FID results, offering a more comprehensive view of the method’s performance.

Weakness 2 : Is speculative decoding still necessary in such a case?

We sincerely thank you for raising this important question. To explore the necessity of speculative decoding, we conducted an experiment using a randomly initialized drafter model to evaluate its ability to accelerate decoding. For this experiment, we utilized LlamaGen Stage I model as the target model and randomly sampled 1000 captions from the MS-COCO 2014 validation set to measure each model's performance. The results are summarized in the tables below:

The results clearly demonstrate that a randomly initialized drafter model fails to improve the mean accepted length, achieving a value of 1.00 for both $\tau=0$ and $\tau=1$, equivalent to standard decoding without speculative decoding. In contrast, the trained drafter significantly enhances performance, achieving a mean accepted length of 1.60 and 1.20 for $\tau=0$ and $\tau=1$, respectively, after 20 epochs of training. These findings highlight the necessity of speculative decoding and the importance of having a well-trained drafter to achieve meaningful acceleration. Without proper training, the drafter cannot generate effective predictions, rendering speculative decoding ineffective. We are grateful for your insightful suggestion, as it allowed us to further validate the critical role of the drafter in speculative decoding and to provide a clearer understanding of its importance in our approach. This experiment reinforces the robustness and continued relevance of speculative decoding in achieving both speed and quality improvements.

Once again, we thank you for your thoughtful comments and suggestions, which have significantly contributed to refining our work. We hope the additional experiments and explanations provided above adequately address your concerns. Please do not hesitate to reach out if further clarification or additional results are needed.

We deeply appreciate your positive evaluation of our work and your thoughtful feedback. Your recognition of the strengths in our methodology and the potential contributions of our approach is highly encouraging. Additionally, your detailed comments and constructive suggestions have been invaluable in guiding us to improve the quality and clarity of our work further. Below, we address each of the points you raised and provide additional experiments and explanations.

Weakness 1 : Evaluation of the image quality

We have conducted additional experiments using a broader set of evaluation metrics. The updated results are presented below (which is the same as in the general response), expanding on the main table in our original submission with same setting:

As shown in the table above, LANTERN achieves comparable or slightly higher precision while exhibiting a modest reduction in recall. This suggests that the quality of individual images is preserved, albeit with a slight decrease in diversity.

This modest decline in recall is likely due to the token selection ambiguity faced by the drafter. When the drafter is not optimally trained, it may struggle to generate predictions that are both diverse and accurate. As a result, increasing the acceptance probability to enhance acceleration could inadvertently reduce the diversity of the generated images. Nevertheless, this trade-off between speed and diversity has minimal impact on overall image quality, as demonstrated by stable FID, Precision and HPS v2 scores. Further improvements in drafter training could mitigate this effect, enhancing recall while maintaining the efficiency gains achieved by LANTERN. Additionally, the HPS v2 score, which is derived from a preference model trained on a human preference dataset, does not exhibit any significant degradation, further supporting the robustness of LANTERN's performance.

We also understand your concern regarding the stylistic similarity and lack of text-image consistency in the generated samples. While we acknowledge this issue, we believe it primarily stems from the base visual AR model (LlamaGen Stage II) rather than being specific to LANTERN. To address this, we revised the text prompts used for generation of qualitative samples and have conducted additional experiments focusing on cases where the base model generates images that meet the following criteria: (1) stylistically dissimilar, (2) consistent with the text prompt, and (3) free of visual errors. For these cases, we provide samples generated by LANTERN to demonstrate its performance under such conditions. In addition, we provide samples generated by random replacement decoding as well. The qualitative samples are available at here : LANTERN samples, random replacement decoding, and qualitative samples. Each image have been incorporated into the Figure 1, 3, and 4 in the revised manuscript.

We hope these additional results provide a more comprehensive perspective on the image quality generated by LANTERN.

Thank you for your kind words and for maintaining a positive recommendation for our research. We believe that your feedback has helped make our work even more robust.

We sincerely thank you for your thoughtful and constructive feedback. Your comments have highlighted critical areas for improvement and provided valuable insights that have strengthened our work. We are especially grateful for your positive remarks regarding the clarity and intuitiveness of our proposed method and its effectiveness in accelerating image generation. Below, we address each of the points raised with detailed responses.

Weakness 1: Gap between the introduction and methodologies in writing

We agree that the gap between the introduction and methodologies could have been improved. To address this, we revised the paper to include a concise preliminary section (Section 2 in the revised manuscript) in the main body, bridging the introduction and methodologies sections. Due to ICLR's page limit policy, we kindly ask for your understanding as we were unable to include the entire related work section. Additionally, Sections 2.1 and 2.2 have been merged into a single section (Section 3 in the revised manuscript) and refined to eliminate overlaps and improve conciseness. We believe these updates enhance the coherence of the paper, making it more accessible to readers and better aligning the introduction with the subsequent sections.

Weakness 2: Insufficiency in evaluating efficiency

First and foremost, we sincerely apologize for the misstatement in the original manuscript regarding the measurement of mean accepted length. While we had stated that the mean accepted length was measured on 100 images, this was incorrect. In fact, mean accepted lengths were evaluated on the full MS-COCO dataset, and this has been corrected in the revised manuscript.

Additionally, we acknowledge that using only 100 captions to measure actual speedup (wall-clock time) may be insufficient for providing convincing evidence. To address this, we have re-measured the actual speedup using a larger sample size of 1000 captions in the revised version. We deeply regret any confusion caused by this oversight and thank the reviewers for bringing it to our attention, allowing us to improve the clarity and accuracy of the manuscript.

In addition, we performed an analysis to confirm that speedup results stabilize as the number of captions increases. The table below illustrates this stability, showing that measurements with 1000 captions are consistent with those using larger sets:

Please note that since the captions were randomly sampled, the results for the 100 captions may differ slightly from the acceleration reported in Table 3 of the original paper (Table 2 in the revised version). We updated our main result in Table 2 (in the revised version) with this re-measured speedup, and we hope these additional evaluations provide a clearer understanding of the robustness of our efficiency claims. The analysis of a number of captions has been added to Appendix F.1 in the revised manuscript.

Weakness 3: Further Analyses

• Further analysis on $\delta$ and $k$

  As part of an extended evaluation prompted by another reviewer's question, we conducted additional experiments with image quality metrics (e.g., precision/recall and HPS v2) under the same settings as our main results. Through this process, we observed that increasing $k$ values does not consistently improve performance for small $\delta$, contrary to our initial claim. The table below illustrates this behavior:

  Configuration	Precision / Recall	HPS v2
  $k=100, \delta=0.05$	$0.4867$ / $0.5389$	$24.01$
  $k=300, \delta=0.05$	$0.4856$ / $0.5367$	$23.97$
  $k=1000, \delta=0.05$	$0.4865$ / $0.5334$	$23.91$

Anole exhibits significantly lower next-token prediction probabilities (average top-$1$ probability of $0.064$ and average top-$10$ probability of $0.204$) compared to LlamaGen ($0.206$ and $0.520$, respectively), as shown in Figure 2 (c) of the revised manuscript. These results indicate that Anole suffers from more severe token selection ambiguity, which inherently affects the drafter’s training process. Consequently, Anole’s drafter achieves a test accuracy of only $27.60\%$, markedly lower than the $38.80\%$ test accuracy observed for LlamaGen’s drafter, as shown in Figure 2 (b) in the revised manuscript. This discrepancy in drafter performance directly impacts the effectiveness of both EAGLE-2 and LANTERN on Anole.

It is important to emphasize that this variation does not reflect a limitation of LANTERN itself but rather underscores the influence of model-specific characteristics, such as token selection ambiguity, on overall performance. Despite these challenges, LANTERN continues to outperform EAGLE-2 on Anole in terms of acceleration while maintaining acceptable image quality, highlighting its robustness across diverse models.

Furthermore, these findings suggest a promising direction for future research. Addressing token selection ambiguity through improved drafter architectures or incorporating ambiguity-aware training techniques could enhance the drafter’s performance, further bolstering LANTERN’s effectiveness across models with severe token selection ambiguity. This indicates that while the current results demonstrate LANTERN's robustness and general applicability, there is potential for even greater improvements with advancements in drafter training and model design.

We have included the these results to appendix E in the revised manuscript, as we aimed to minimize significant changes to the main content of the manuscript. However, if the reviewers collectively feel that incorporating these results into the main table would enhance clarity and comprehensiveness, we would be happy to expand the main table in the next revision. We sincerely appreciate all of your valuable feedback and remain committed to improving the manuscript in line with your suggestions.

Major changes in the revised manuscript

Again, we sincerely thank the reviewers for their valuable suggestions and thoughtful feedback, which have greatly contributed to improving the quality of our work. Based on these insights, we have carefully revised the manuscript to address the concerns raised and incorporate additional analyses.

We have made several key revisions to enhance the clarity, organization, and comprehensiveness of the manuscript:

1. Preliminaries Section (Section 2): Added a new section to bridge the Introduction and Methodologies, providing essential background on visual AR models and speculative decoding to improve reader understanding.
2. Consolidation of Sections 2.1 and 2.2: These sections were merged and rewritten to eliminate overlapping content and improve the flow of information. While the text was significantly revised to achieve a clearer and more concise presentation, the underlying content and key messages remain unchanged.
3. Expanded Experimental Section: Incorporated additional results using new evaluation metrics into the main table (Table 2) and corresponding explanations in the Experimental Setup (Section 5.1) and Results sections.
   • Updates in ablation study : To provide a clear comparison between TVD and JSD, we compared the differences in image quality between TVD and JSD when both achieved the same level of acceleration. This analysis has been reflected in Table 3 and Section 5.3.2 of the revised manuscript.
4. Updated Qualitative Samples: Replaced the samples in Figures 1, 3, and 4 with more representative examples to better align with text prompts, address visual errors, and showcase stylistic diversity. The updated figures also can be found in these anonymous links: LANTERN samples, random replacement decoding, and qualitative samples.

We hope these additional experiments further support the contributions of our work and provide clarity on its robustness. Once again, thank you for your time and effort in reviewing our submission. We are grateful for your constructive feedback and look forward to your continued insights.

References

[1] Kynkäänniemi, Tuomas, et al. "Improved precision and recall metric for assessing generative models." Advances in neural information processing systems 32 (2019).

[2] Wu, Xiaoshi, et al. "Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis." arXiv preprint arXiv:2306.09341 (2023).

[3] Chern, Ethan, et al. "Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation." arXiv preprint arXiv:2407.06135 (2024).

As previously mentioned, we further evaluate the performance of LANTERN under greedy decoding ($\tau=0$) and summarize the results in the following table. Similar to the case of sampling ($\tau=1$), we report both acceleration and image quality across various metrics. Similar to the sampling, the findings demonstrate that LANTERN achieves substantial speedup while maintaining competitive image quality compared to standard AR decoding in the greedy decoding as well. These results reaffirm LANTERN's capability to balance efficiency and quality across different decoding strategies. This additional analysis has also been incorporated into the main table (Table 2) of the revised manuscript.