Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression

We extend our gratitude for your insightful feedback and suggestions, which have helped us improve the clarity and completeness of our work.

Q1: The proposed method is specially designed for VAR. Therefore, considering the rapid development of AR image generation, the effectiveness for future AR models maybe limited.

A1: Thank you for the question. Our method generalizes beyond VAR to AR architectures. We demonstrate this by evaluating ScaleKV on LlamaGen [1], an image autoregressive model, showing consistent performance with minimal degradation while achieving significant memory reduction:

Table 1: DPG score of ScaleKV-compressed LlamaGen model

[1] Autoregressive model beats diffusion: Llama for scalable image generation

Q2: Too many hyper-parameters making the proposed method much complex and less easy to extend and generalize.

A2: Thank you for the valuable feedback. Our method has two tunable hyper-parameters: Top-K scores in drafter/refiner classification and observation window size in KV cache selection. Both show robust performance across wide ranges:

Table 2: Ablation of Top-K score

Table 3: Ablation of observation window size

This stability demonstrates good generalizability and practical usability.

Q3: Some works share very similar ideas of this paper for improving VAR's generation efficiency, for example [1]. I recommend the authors to discuss with these works, if considering they are concurrent works?

A3: Thank you for raising this important point. While both methods optimize the KV cache for VAR models, HACK [1] allocates fixed budgets by classifying individual attention heads, whereas ScaleKV dynamically categorizes entire layers as "drafters" or "refiners" to adaptively manage memory at each generation scale. Since HACK hasn't been open-sourced yet, we directly report their DPG score from the original paper.

Table 4: Comparison of ScaleKV and HACK on DPG benchmark

[1] Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling.

Q4: As the rapid development for AR image models, it is necessary to include more representative AR image models like [2][3].

A4: We appreciate this constructive feedback. We have evaluated ScaleKV on LlamaGen in A1 and Table 1, demonstrating effectiveness on large-scale text-to-image AR models beyond VAR framework. This validates our method's broad applicability to the AR family.

Q5: Including the original uncompressed VAR model's performance in Table 1 and Table 2 will help authors to a more comprehensive understanding.

A5: We appreciate this observation. We have included performance of the uncompressed models (Infinity-2B and 8B) in the paper's Table 2. Table 1 specifically measures consistency between compressed and original uncompressed models, so only compressed model metrics (FID, LPIPS and PSNR) are relevant there.

Q6: Add the difference map between generation results of compressed and uncompressed models will be helpful, such as in Figure 1.

A6: Thank you for this helpful suggestion. We will include difference maps in the revision to provide visual insights into generation quality preservation.

Q7: The identification of Drafter and Refiner Layers are decided by the value of Attention Selectivity Index, will there exist a hybrid layer around the identification threshold?

A7: Thank you for the valuable comment, this is indeed a question that needs further explanation. We use a strict threshold ensuring layers are classified as drafters only when cache requirements are definitively large. This conservative approach eliminates ambiguous hybrid cases, ensuring robust layer categorization.

We're grateful for the feedback you've provided.

Q1: Niche Application Scope: VAR models (e.g., Infinity) are not the mainstream in text-to-image generation, which is currently dominated by diffusion models (e.g., SDXL, SD3, SANA, ...). The paper does not demonstrate whether ScaleKV’s core strategies (e.g., layer classification, scale-aware allocation) can generalize to non-VAR frameworks, limiting practical impact

A1: Thank you for the valuable feedback. We acknowledge that diffusion models currently dominate text-to-image generation, but we believe ScaleKV addresses fundamental challenges that extend beyond VAR models:

Broader Applicability Beyond VAR: Our method generalizes beyond VAR to traditional AR frameworks. We validate this by applying ScaleKV to LlamaGen [1], an image autoregressive model, demonstrating robust performance with minimal quality loss while achieving substantial memory savings:

Table 1: DPG score of ScaleKV-compressed LlamaGen model

Strategic Focus on VAR: VAR models are gaining remarkable attention due to their efficiency, scalability, and potential for unified vision understanding and generation. ScaleKV effectively solves the memory bottleneck that makes high-resolution modeling challenging, enabling these promising architectures to scale to ultra-high resolutions such as 4K.

Future Extensibility: Our drafter/refiner classification and scale-aware allocation principles provide a methodology that could adapt to future architectures, including potential hybrid approaches combining visual autoregressive and diffusion techniques.

[1] Autoregressive model beats diffusion: Llama for scalable image generation.

Thank you very much for your review of our manuscript and attachment. We appreciate your time and effort in evaluating our work.

Q1: How can this method generalize to other datasets? Currently, the method's static, binary classification of layers has only been evaluated on the Infinity model family and requires further validation for its generalization across diverse inputs and other model architectures. I would like to see some discussions or empirical results.

A1: Thank you for this valuable feedback. We evaluated ASI robustness across three distinct visual domains using 128 calibration prompts each from HuggingFace datasets: flickr30k_captions_simCSE (human faces), architecture_house_building_prompts_SDXL (architecture details) and nature-dataset (natural landscapes). Results show consistent performance across domains, confirming that drafter-refiner distinctions reflect model characteristics rather than domain-specific patterns. We also demonstrate applicability to traditional AR architectures like LlamaGen (Table 5).

Table 1: Cross-domain validation results

Q2: It is still unclear about selecting core hyperparameters, such as the K' in the ASI formula and the observation window size. Some key experimental setup details should be complete.

A2: We appreciate your suggestions on clarifying these details. Our method demonstrates robust performance across varying hyperparameters, requiring minimal tuning. Tables 2-3 show stable results across different Top-K values (1%-20%) and window sizes (4-32 tokens).

Table 2: Top-K parameter ablation

Table 3: Observation window size ablation

Q3: The paper did not discuss the method's integration and compatibility with critical performance optimizations, such as Flash Attention.

A3: Thank you for pointing this out. ScaleKV is fully compatible with FlashAttention. We report the inference latency of Infinity-8B measured on a single NVIDIA A100-SXM4-80GB GPU. Table 4 shows combined optimization achieves 2.7× speedup while maintaining compression benefits.

Table 4: Inference latency comparison (Infinity-8B)

Q4: Why not perform the drafter/refiner identification on a per-sample basis? I am concerned about the method's adaptability to other models.

A4: Thanks for raising this important concern. Our drafter/refiner identification process is conducted before model deployment. Performing calibration on a per-sample basis would require capturing full attention maps for each sample, which would eliminate the computational benefits of KV cache compression.

Our compression method is also applicable to traditional AR image generation frameworks, including LlamaGen [1].

Table 5: DPG scores of ScaleKV-compressed LlamaGen model

[1] Autoregressive model beats diffusion: Llama for scalable image generation.

Q5: Also, when the calibration set size varies, do other metrics besides FID also remain consistent?

A5: Thank you for this question. We tested calibration sizes from 1 to 128 prompts using GPT-4o generated data. Table 6 demonstrates stable performance regardless of calibration size, confirming that minimal data suffices due to fundamental attention pattern differences and cache requirements between layer types.

Table 6: Calibration data size ablation

Q6: What is the dominant component of the Attention Selectivity Index (ASI)—the 'current' attention ratio or the 'history' Top-K ratio, and how are they balanced? How is the Top-K' parameter selected, and what is the basis for choosing the observation window size (mentioned as typically 16 tokens)?

A6: Thanks for the valuable comment. The Attention Selectivity Index (ASI) multiplies the current attention ratio and history Top-K ratio, ensuring both components contribute equally to layer classification. The robustness experiments above demonstrate minimal parameter tuning requirements (see A2 and Tables 2-3). We chose 16 tokens as the default window size to ensure fair comparison with existing baselines, as this is the standard setting used in most KV cache compression methods.

Q7: The baselines for comparison might not be recent enough; could you compare against more current cross-layer KV allocation methods?

A7: Thanks for the suggestion. We compared ScaleKV against two recent cross-layer KV cache allocation methods: PyramidKV [1] (employs pyramid-shaped budget allocation) and SimLayerKV [2] (attention-based selective cache allocation). ScaleKV significantly outperforms both methods across all metrics.

Table 7: Comparison with cross-layer KV cache allocation baselines

[1] PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

[2] SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction

Q8: I'm also wondering whether a more granular, per-layer management strategy would be better than the current simple distinction between 'drafters' and 'refiners'.

A8: We appreciate this constructive feedback. We implemented a per-layer granular management strategy that allocates individual budgets based on the proposed Attention Selectivity Index (ASI). As shown in Table 8, granular allocation does not provide additional benefits over our current method, supporting our hypothesis that cache demands are similar within groups but different between drafter/refiner groups.

Table 8: Comparison with per-layer management strategy

Q9: Additionally, what was the image resolution used for the main results presented in the tables?

A9: Thanks for the question. All experiments use 1024×1024 resolution to validate ScaleKV's effectiveness for high-resolution generation, where memory bottlenecks are most critical.

I appreciate the authors' solid rebuttal, which has addressed most of my concerns. I have also read other reviews and the authors' responses to them. I agree with other reviewers on accepting the paper. Therefore, my final rating is an Accept. Congrats!

Thank you for your valuable feedback on our work. Your constructive comments on our work are invaluable.

Q1: Lack of discussion of generalization of the proposed method: While the paper thoroughly evaluates on Infinity models, it provides insufficient discussion of transferability. The evaluation does not include VAR variants with different attention mechanisms. It remains unclear whether the drafter/refiner would hold for architectures with different layer configurations or attention head designs.

A1: Thank you for this valuable feedback. Our method generalizes beyond VAR architectures. We validate this by applying ScaleKV to LlamaGen [1], an autoregressive image model with fundamentally different attention mechanisms. Table 1 shows consistent performance with minimal degradation while achieving significant memory reduction. We also tested across different model scales (Infinity-2B/8B), confirming robustness across layer configurations in the paper.

Table 1: DPG score of ScaleKV-compressed LlamaGen model

[1] Autoregressive model beats diffusion: Llama for scalable image generation

Q2: Additionally, the paper neither analyzes if the ASI metric (Eq. 2) requires adjustment for models with varying attention mechanisms nor discusses its adaptability to future VAR architectures with novel designs.

A2: We appreciate this important concern. For future VAR architectures that use Grouped-Query Attention (GQA) for efficient computation, ASI should be adjusted to measure attention diversity between entire query groups rather than between individual heads. This involves shifting the core unit of analysis from the head to the group level, as heads within a group are no longer independent.

Q3: Evaluation on larger models: As acknowledged in Appendix D, the evaluation covers models up to 8B parameters, with Infinity-8B as the largest tested. Thus, empirical validation is missing for truly large-scale (e.g., 20B) models, where attention patterns and drafter/refiner ratios might differ. Additionally, the linear scaling assumption for memory savings remains unverified at larger scales.

A3: Thanks for the valuable comment. We tested our method on Infinity-8B, currently the largest available VAR model. Results demonstrate improved scaling behavior—performance increases from Infinity-2B to 8B (FID: 2.53→2.12 at 10% compression), suggesting larger models benefit more from ScaleKV due to increased attention redundancy. The linear scaling assumption holds because memory reduction is directly proportional to pruned KV cache tokens. We will evaluate on larger models when available.

Q4: Trade-off analysis: While Figure 7 (a) provides a trade-off curve showing FID scores across five compression ratios (1%, 4%, 10%, 16%, 20%), the analysis could be more comprehensive. The paper focuses primarily on three ratios (4%, 10%, 20%) in most experiments and does not provide guidance on choosing appropriate compression levels for specific use cases. There is no principled method offered for practitioners to determine optimal compression ratios given their quality constraints.

A4: Thank you for your constructive suggestion. We provide comprehensive trade-off analysis with clear guidance for practitioners. Table 2 demonstrates ScaleKV's effectiveness across five compression ratios on Infinity-2B, showing a clear quality-efficiency trade-off. Extreme compression levels (1-4%) achieve dramatic memory reduction suitable for resource-constrained environments like mobile devices, while moderate compression (10-16%) maintains high visual quality for standard deployments. Conservative compression (20%) preserves exceptional quality for professional applications requiring high fidelity.

Table 2: Quality & efficiency trade-off (Infinity-2B)

Q5: Besides, the paper lacks analysis of which image types or generation tasks are most sensitive to compression. For example, whether architectural details, human faces, or natural landscapes show different degradation patterns.

A5: Thank you for raising this important point. We evaluated ScaleKV across three distinct visual domains using 5,000 prompts each from HuggingFace datasets: flickr30k_captions_simCSE (human faces), architecture_house_building_prompts_SDXL (architectural details) and nature-dataset (natural landscapes). Table 3-5 demonstrate consistent performance across all domains, confirming ScaleKV's robustness for diverse visual content.

Table 3: Output consistency on human faces

Table 4: Output consistency on architecture details

Table 5: Output consistency on nature landscapes

Q6: Architecture variations: The paper's evaluation focuses exclusively on Infinity models. How does ScaleKV perform on VAR models with architectural variations? Specifically, I'm curious about models using cross-attention mechanisms between different scales rather than just self-attention within scales, as these might exhibit fundamentally different attention patterns.

A6: Thank you for the question. VAR models employ complex attention mechanisms beyond simple self-attention: causal cross-attention between current scale tokens and previous prefix tokens in KV cache, combined with bidirectional attention within current scale tokens. Our successful evaluation on both VAR and LlamaGen with different mechanisms (discussed in A1 and Table 1) demonstrates broad applicability across diverse attention patterns.

Q7: Integration with other optimizations: Can ScaleKV be effectively combined with quantization methods (e.g., LiteVAR [1])? Cache compression and reduced precision can either help each other or get in each other’s way.

A7: Thank you for the valuable suggestion. ScaleKV is designed to be complementary to other optimization techniques. Both ScaleKV and LiteVAR can optimize inference efficiency with negligible quality degradation, so combining them could further enhance overall efficiency. However, since LiteVAR has not been open-sourced yet, we could not test this integration experimentally.

Beyond quantization, ScaleKV could be integrated effectively with FlashAttention [1] to further optimize computational efficiency. We report inference latency of Infinity-8B measured on a single NVIDIA A100-SXM4-80GB GPU in Table 6, showing that the combined approach achieves 2.66× speedup while maintaining the same memory compression benefits.

Table 6: Inference latency of Infinity-8B

[1] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Q8: Minor Issues

A8: Thank you for the careful review. We will address all minor issues including typos, formatting inconsistencies, and notation clarifications in the revised manuscript.

We sincerely appreciate the valuable feedback and constructive suggestions. Thanks so much for taking time and effort to review our paper.

Q1: The Attention Selectivity Index (ASI) requires minimal calibration data (10 prompts) to distinguish Drafters and Refiners. However, the method's sensitivity to calibration prompt diversity and content is not fully explored. For example, if prompts are biased toward specific visual domains, the layer classification might not generalize to broader scenarios, potentially affecting cache allocation accuracy.

A1: Thanks for the valuable feedback. We evaluated ASI robustness across three distinct visual domains using 128 calibration prompts each from HuggingFace datasets: flickr30k_captions_simCSE (human faces), architecture_house_building_prompts_SDXL (architecture details) and nature-dataset (natural landscapes). Table 1 shows consistent performance across all domains, confirming that drafter-refiner distinctions reflect intrinsic VAR model characteristics rather than domain-specific patterns.

Table 1: Ablation of calibration data diversity and content

Q2: Additionally, I am curious whether the experimental conclusions are consistent with different amounts of calibration data. ScaleKV only tested on 10 prompts to determine the drafters and refiners—are there more conclusions supporting this?

A2: Thanks for the valuable question. We tested calibration sizes from 1 to 128 prompts using GPT-4o generated data. Table 2 demonstrates stable performance regardless of calibration size, validating that minimal data suffices to identify layer roles due to fundamental attention pattern differences and cache requirements.

Table 2: Ablation of calibration data size

Q3: Regarding actual running speed rather than computational load or GPU usage, I noticed in Fig7(b) that the latency for 1024×1024 resolution is only 52ms vs. 71ms compared to full cache. Are there any optimization potentials or observations here?

A3: Thanks for the valuable observation. ScaleKV prioritizes memory efficiency over raw speed, as VAR's main bottleneck is exponential KV cache growth at high resolutions. VAR is already significantly faster than diffusion transformers or traditional AR models of comparable size. Nevertheless, our compression method can be integrated with FlashAttention [1] to further optimize inference speed, achieving combined 2.66× speedup (77ms → 29ms) on Infinity-8B with NVIDIA A100-SXM4-80GB.

Table 3: Inference latency of Infinity-8B

[1] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Q4: What other drawbacks does the current KV cache method have?

A4: Thanks for raising this important concern. As a post-training compression framework, ScaleKV's output quality is fundamentally bounded by the original VAR model's capabilities. Therefore, if the baseline quality of the original VAR models is unsatisfactory, achieving high-quality results with our method could be challenging.