FreqExit: Enabling Early-Exit Inference for Visual Autoregressive Models via Frequency-Aware Guidance

We sincerely thank you for the recognition and detailed comments. Below, we address each point and clarify the motivation, assumptions, and design choices of our method, hoping to better convey its contributions and resolve your concerns.

W1: "Since the proposed acceleration strategy needs to retrain the VAR model from scratch ..."

W1-Ans: In practice, FreqExit does not require retraining the VAR model from scratch. For example, we start from the official VAR-d20 checkpoint (trained for 250 epochs on ImageNet-1K) and fine-tune for only 80 additional epochs (~30% extra cost) to enable early-exit capability. Our rotating supervision further reduces computation by selectively activating losses across layers. Thus, the added training overhead is modest and far from the cost of full model training.

This small investment yields sustained inference benefits: ~30% latency reduction without FID loss, and up to 2× speedup with mild quality trade-offs. FreqExit is deployment-friendly and allows runtime adaptation to diverse resource budgets.

Furthermore, FreqExit is the first to apply early exit in next-scale generation, revealing a progressive low-to-high frequency pattern and addressing frequency conflicts via wavelet-based supervision. Our training strategy is general to transformer-style architectures with layered outputs and could extend to autoregressive models in other domains where frequency-phase behaviors are common.

W2: "The method needs to be evaluated on a broader range of VAR models before it can be claimed as a “unified” training framework."

W2-Ans: We supplement our analysis with additional experiments on both VAR-d16 and VAR-d24 models. As shown in the tables, our method significantly reduces the FID scores of intermediate layers, demonstrating its effectiveness in enabling early exits with high-quality outputs.

Table 5: VAR-d16 experimental results

Table 6: VAR-d24 experimental results

W3: "The motivation is not strong enough."

W3-Ans:

Table 7: Token Matching Rate with Final Transformer Layer Output (VAR-d20, Last Step)
At the last generation step, we apply greedy sampling to each layer’s output and compute the per-token match rate with the final layer’s output tokens.

Motivation: Our motivation stems from a mismatch between dynamic inference assumptions and next-scale generation behavior in VAR models. CALM-style confidence signals (e.g., Softmax Max, Entropy) remain far below usable thresholds, revealing that training-free heuristics are unreliable. This is because assumptions like semantic stability, monotonic locality, and granularity consistency break down. FreqExit addresses this by introducing training-time supervision to enable reliable early exits.

Semantic stability: assumes that token representations converge across deeper layers. However, as shown in our analysis (Table 1 in Rebuttal ), softmax confidence remains low and unstable throughout the layers of VAR models. This is likely due to the increasing resolution and complexity of token maps, which evolve from coarse global structures to fine-grained high-frequency content. This continual refinement prevents logits from stabilizing and invalidates the assumption that later layers produce redundant outputs.

Monotonic locality: assumes that tokens generated at each step can serve as a fixed and verifiable prefix. In next-scale generation, however, token maps are not immutable—tokens generated at step $i$ are frequently updated in step $i{+}1$ through upsampling and refinement. To validate this, we measured the token-wise matching rate between intermediate layers and the final output at the last generation step of VAR-d20. As shown in Table 7, most layers between 10 and 17 yield match rates below 5%, and even Layer 19 reaches only ~60%, confirming the instability of intermediate outputs and the failure of prefix-based verification.

Granularity consistency: assumes that dynamic inference operates over fixed, bounded computation units (e.g., one token or one layer). This assumption breaks in next-scale generation due to VAR’s hierarchical structure: each step generates a $p_n \times p_n$ token map, which is upsampled to full resolution and then downsampled for the next step. These tokens are intermediate and overwritten, leading to unstable computation units and growing step-wise cost. As a result, fine-grained control (e.g., per-token or per-layer gating) becomes ineffective.

We believe this unified analysis and supporting experiments clearly demonstrate that effective dynamic inference in next-scale generation requires rethinking beyond inference-time heuristics.

W4: "It is quite hard to identify which loss function..."

W4-Ans: Our ablation（Table 8-10）shows that removing either HF-loss or FGSR significantly degrades early-exit quality (higher FID), while final-layer outputs are only marginally affected—supporting our claim that high-frequency mismatch is the main constraint on early-exit performance.

Table 8: VAR-d20 with Early Exit loss

Table 9: VAR-d20 with Early Exit loss and HF loss

Table 10: VAR-d20 with Early Exit loss, HF loss, and FGSR loss (FreqExit)

Q1: "Can you explain more about what the assumptions of dynamic inference are: semantic stability, monotonic locality, and granularity..."

Q1-Ans: VAR models exhibit unstable intermediate tokens, overwritten outputs across steps, and variable computation granularity, undermining conventional early-exit heuristics. These failures motivate our training-time solution. Please refer to W3-Ans for full analysis and evidence.

Q2: "How does removing the low-frequency content affect quality?"

Q2-Ans: We clarify the roles of low- and high-frequency components in image generation: low-frequency (LL) sub-bands capture global semantics, while high-frequency (LH/HL/HH) sub-bands encode fine textures. Evaluating VAR-d16/d20/d24/d30, we found that removing low-frequency components leads to FID > 200, indicating major semantic distortion, whereas removing high-frequency components mainly affects perceptual quality (Fig.~2(b)). This confirms that low-frequency governs semantics, while high-frequency governs realism.

As shown in Fig.~2(a), high-frequency energy rises steadily during generation, reaching ~50% in later stages. This makes high-frequency mismatch a key source of residual error, as shallow layers struggle to capture the needed fine details. In such cases, enhancing low-frequency features helps little, while improving high-frequency modeling yields significant FID gains. FreqExit introduces two high-frequency enhancement strategies:

(1) HF Consistency Loss: aligns the wavelet-domain high-frequency outputs of shallow layers with the final output.

(2) FGSR Loss: provides stage-aware, spectrum-adaptive supervision, guiding the model to adjust to frequency behavior at each generation step and improve texture reconstruction.

Our ablation（Table 8-10）shows that removing either HF-loss or FGSR significantly degrades early-exit quality (higher FID), while final-layer outputs are only marginally affected. In conclusion, Observation 2 highlights a critical insight: once low-frequency information is learned through early-exit supervision, remaining quality gaps are due to insufficient high-frequency modeling.

Q3: "What is the setting of each strategy in Table 2?"

Q3-Ans: The strategies in Table 2 represent different early-exit paths applied to the same trained FreqExit model. Each is a 10-dimensional vector specifying the number of layers used at each generation step. For example, “7” = [16, 12, 10, …, 10] uses 16 layers at the first step, 12 at the second, and 10 for the rest. This enables flexible quality–latency trade-offs without retraining or extra modules.

This early-exit capability is not achieved by Layer Dropout. It is enabled by our training design, which equips each layer with independent generation ability via supervised learning. Layer Dropout is only used during training as a regularizer to improve robustness and has no effect at inference time.

We appreciate the reviewer’s thoughtful follow-up comments and address each point in detail below.

Q1: "...but VAR is more than just next-scale, like next-token. To me, this cannot be claimed as “unified” unless the authors can justify its applicability to other VAR paradigms."

Ans-1: Thanks for the suggestion! To validate the applicability of our method to next-token generation VAR models, we add the experiments on LlamaGen [^1].  We finetune LlamaGen-B (∼111M parameters, 12 hidden layers) on the ImageNet dataset using the proposed FreqExit method and evaluate the FID of intermediate-layer outputs. Results are shown below.

As shown in Table 1, the FID scores consistently decrease across epochs for layers 8–11, suggesting that FreqExit is also applicable to next-token generation models.

Table 1: FID scores of intermediate-layer outputs for FreqExit on LlamaGen-B.

Besides, additional experiments are conducted on the larger LlamaGen-L model (∼343M parameters, 24 hidden layers). For comparison with the baseline used in the main paper, we also fine-tune LlamaGen-L using the LayerSkip method, which applies only cross-entropy loss without frequency-aware supervision. Both models are fine-tuned for 20 epochs, and the FID results are shown in Table 2 (FreqExit) and Table 3 (LayerSkip). Table 2 demonstrates that FreqExit remains effective on a larger next-token generation model. Compared to LayerSkip (Table 3), FreqExit achieves lower FID scores and faster convergence. These results validate the applicability of our method beyond next-scale VAR.

Table 2: FID scores of intermediate-layer outputs for FreqExit on LlamaGen-L.

Table 3: FID scores of intermediate-layer outputs for LayerSkip on LlamaGen-L.

References:

• [1] Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation, Arxiv: 2406.06525

Q2: "Early Exit loss (Eq.(2)), HF loss (Eq.(6)), and FGSR loss (Eq.(11)) are used as the loss functions? It is unclear how the distillation loss in Eq.(3) and the alignment regularizer in Eq.(12) function differently. It would be good to present the final training objective, which is currently lacking in the paper."

Ans-2: We appreciate the suggestion to clarify the overall training objective. As requested, the final training loss used for FreqExit is defined as:

• $L_{\text{CE}}$ and $L_{\text{KD}}$ (Eq.(2) and Eq.(3)) together constitute the early-exit supervision loss, which provides the primary learning signal by aligning intermediate predictions with ground-truth labels and final-layer outputs.
• $L_{\text{HF}}$ and $L_{\text{FGSR}}$ (Eq.(6) and Eq.(11)) serve as the frequency adaptation loss, which further enhances the high-frequency reconstruction capability of early exits and improves their perceptual quality.

Note that $L_{\text{FGSR}}$ includes an alignment regularizer $L_{\text{align}}$ (Eq.(12)), which enforces approximate orthogonality in the projection matrices to stabilize spectrum reconstruction. Since it is an internal component of $L_{\text{FGSR}}$, we do not treat it as a separate loss term.

We hope the additional results and clarifications can fully resolve your concerns. If you have extra critical concerns, we would be grateful if you could let us know early so that we have enough time to address them.

Finally, thank you so much for helping us improve the paper so far! Looking forward to your further feedback.

Thank you for the response. That clears up my concerns. I will raise the score, assuming the authors will make adjustments to the writing and integrate the answer to my previous questions in the revised version accordingly.

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we provide point-by-point responses to the comments and questions.

W1: "In Equation (2), the meaning of $Y$ is not explained. The term $\tilde{e}$ seems to be a dynamic weight, but a more detailed explanation is needed. Moreover, is the total loss simply a summation of all the losses proposed in the Method section?"

W1-Ans: We thank the reviewer for pointing this out. We provide the following clarifications and will annotate the definitions in the revised version:

• Definition of $Y$: $Y$ denotes the ground-truth token sequence obtained by encoding the target image with a pre-trained VQ-VAE, which serves as the supervision target for predictions.

• Definition of $\tilde{e}(t,\ell)$: $\tilde{e}(t,\ell)$ follows the formulation in Eq. (2), and is a normalized dynamic weight that determines the contribution of each intermediate layer $\ell$ to the early-exit loss at training step $t$:

where

• $C(t,\ell)$ is a curriculum gate that selects a subset of layers to be supervised at each training step. Specifically, all layers are divided into $R$ blocks (with $R=4$ for VAR-d20), and one block is activated per step in a round-robin schedule: block_id = $t$ mod $R$.

• $e(\ell)$ is a depth-dependent static weight defined as: (1) If $\ell < L-1$ , $e(\ell) = \sum_{i=0}^\ell i$, (2) if $\ell= L-1$ , $e(\ell) = (L{-}1) + \sum_{i=0}^{L{-}2} i$. This formulation ensures that deeper layers receive quadratically increasing weights, with additional emphasis on the final layer to promote more comprehensive supervision. This design corresponds to Eq. (14) in Appendix A.1.

• The normalization ensures that the sum of $\tilde{e}(t,\ell)$ over all active layers is 1. This design emphasizes deeper layers during training, ensuring that updates to shallow layers do not compromise the performance of deeper ones.

Total loss structure: The total loss is a weighted combination of the main task loss (CE + KD), the high-frequency (HF) loss, and the FGSR loss. The main loss is applied to both the final and intermediate layers using $\tilde{e}(t,\ell)$. The HF loss is activated after 25% of training and dynamically scaled (see Appendix B, Eq. (20)), while the FGSR loss is applied with a fixed weight of 5.0, along with an orthogonality term weighted by 0.1.

Details on the layer-wise loss weights and training schedule can be found in Appendix B (FreqExit). We hope this clarifies the formulation and thank the reviewer again for the valuable feedback.

W2: "Although many losses are introduced in the Method section, the ablation study only investigates the HF and FGSR losses. It would be helpful to see how other components affect performance—for instance, the role of the $\tilde{e}$ and gate terms in the early-exit loss, and the temperature scheduling in Equation (3)."

W2-Ans: The roles of the dynamic weighting $\tilde{e}(t,\ell)$ and the gating mechanism $C(t,\ell)$ have been detailed in our response to Comment 1. Briefly, $e(\ell)$ emphasizes deeper layers to protect their optimization from shallow-layer interference, while $C(t,\ell)$ reduces per-step computational cost and gradient conflict by supervising only a subset of layers in a rotational schedule. Together, they help stabilize training and enhance early-exit performance.

We also provide further clarification on temperature scheduling in Eq. (3). The temperature $T_\ell$ decreases linearly from 4.0 at shallow layers to 1.0 at the final layer. This design is based on the observation that shallow layers have limited semantic capacity during early training. A higher temperature smooths the teacher distribution, providing softer and more tolerant supervision targets. As depth increases, semantic representations become more refined; hence, lower temperatures help retain sharper predictions and improve alignment. This strategy aligns with prior work on progressive distillation and improves intermediate representation quality.

W3: "The overall framework feels a bit complex, making it hard to pinpoint which parts contribute most to the performance. A more concise and clearer explanation would greatly improve readability."

W3-Ans: We thank the reviewer for the helpful suggestion. To clarify the contribution of individual components, we conducted additional ablation studies to further isolate the effects of the FGSR module and temperature scheduling.

W4: "If possible, please provide some visual comparison with the baseline model (e.g., CoDe and LayerSkip)."

W4-Ans: We thank the reviewer for the helpful suggestion. We agree that visual comparisons with baseline methods would provide valuable qualitative insight. Due to rebuttal constraints, we are unable to include figures at this stage. However, we will incorporate representative visual comparisons in the revised version of the paper to highlight the differences in generation quality.

Thank you for your feedback. Due to the rebuttal deadline extension, we have extra time to include additional quantitative experiments to address your concerns. we add new ablation studies on the effects of the layer weighting scheme $\tilde{e}$ and the temperature scheduling strategy. The experiments are conducted on the LlamaGen-L model (343M, 24 layers). Table 1 reports the results comparing the uniform-weight baseline ($\tilde{e}=1$) with our depth-aware layer weighting scheme. Table 2 compares the effect of using a fixed temperature ($T=1$) versus our progressive scheduling strategy, where $T$ decreases linearly from 4.0 at shallow layers to 1.0 at the final layer.

Table 1. FID scores of intermediate-layer outputs under two layer weighting settings: (1) uniform weighting ($\tilde{e}=1$), and (2) depth-aware weighting computed according to Eq.~(14) in the paper. Results are reported at epochs 8, 16, and 20 on LlamaGen-L.

Table 2. FID scores of intermediate-layer outputs under two temperature settings: (1) fixed temperature $T=1$, and (2) progressive temperature scheduling from $T=4.0$ (shallow layers) to $T=1.0$ (final layer). Results are reported at epochs 8, 16, and 20 on LlamaGen-L.

• Layer weighting

Our model involves multi-layer supervision, which can be seen as a multi-objective optimization problem. To prevent shallow-layer gradients from dominating and degrading deep-layer performance, we assign progressively increasing weights to deeper layers. As shown in Table 1, using uniform weights ($\tilde{e}=1$) yields similar FID at shallow layers (e.g., layer 20). However, it leads to worse performance at deeper layers (e.g., layers 22–23), indicating interference from shallow-layer updates.

• Temperature scheduling

In early-exit settings, aligning shallow-layer outputs with the final-layer teacher logits can be overly strict due to limited representational capacity. Applying a higher temperature smooths the teacher distribution, reducing the difficulty of distillation for shallow layers. As shown in Table 2, progressive temperature scheduling reduces FID across all layers, with the effect especially pronounced at deeper ones. This reduction is likely related to more stable alignment at intermediate layers.

We hope these additional analyses help clarify our design choices and address your concerns. If you find the improvements satisfactory, we would be grateful if you would kindly consider raising the score.

If you have any further suggestions or questions, we would be happy to hear them and will do our best to respond within the available time. Thank you again for your thoughtful feedback and we truly appreciate your time and consideration.

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we provide point-by-point responses to the comments and questions.

W1: Role of High Frequencies

W1-Ans: We thank the reviewer for raising this point. To clarify upfront: image compression (e.g., JPEG) and visual image generation pursue fundamentally different goals. Compression removes redundant information from complete images to reduce bitrate. In contrast, generative models synthesize content from scratch and rely on high-frequency details to achieve perceptually convincing results. In generative methods such as GANs [1] and diffusion models [2], insufficient high-frequency modeling typically leads to over-smoothing and loss of texture, degrading visual quality.

Our work investigates this issue in autoregressive visual generation, with a focus on next-scale generation, where images are produced progressively across resolution scales. We identify a previously underexplored frequency pattern: low-frequency components dominate early stages, while high-frequency content becomes dominant in later steps. This frequency inconsistency limits the effectiveness of confidence-based, training-free early-exit strategies, which assume uniform information distribution across layers. To address this, we propose FreqExit, a frequency-adaptive training framework that explicitly aligns learning dynamics with this progression. It integrates frequency-aware supervision and a frequency-gated residual mechanism to promote earlier modeling of high-frequency features, enabling more effective early exits with minimal loss in image quality.

W2: Distillation Overhead

W2-Ans: In our setup, the teacher model is frozen and performs only a single forward pass per batch. Its activations are not stored, and memory is immediately released after inference, resulting in minimal resource usage. To quantify this, we conducted a profiling experiment using PyTorch’s torch.cuda.max_memory_allocated() to capture peak memory during the teacher’s forward pass. As shown in Table 4, the teacher model accounts for only 5.3% of peak memory usage and 8.8% of total training time.

Table 4: Resource Usage Breakdown During Distillation Training

This confirms that our distillation strategy imposes minimal overhead. The training process is also fully compatible with widely-used optimization tools such as mixed-precision training, ZeRO, and DeepSpeed, which can further reduce the cost if necessary. The modest additional training cost is acceptable, and this frozen-teacher strategy is widely adopted in model compression research [3-4].

W3: Dataset and Resolution Generalization

W3-Ans: We chose ImageNet-256 as our evaluation benchmark because existing VAR-style models are trained and publicly released on this dataset. This ensures reproducibility and allows for direct, fair comparison with baseline methods. In addition, ImageNet’s 1,000-category diversity provides rich variations in texture, scale, and semantics, making it a strong testbed for evaluating convergence, detail recovery, and long-range dependencies.

FreqExit is resolution- and dataset-agnostic, as it operates solely along the depth axis by adjusting the number of active layers at each generation step, without modifying the patching scheme, tokenization process, or data modality. For example, in VAR models, increasing resolution from 256×256 to 512×512 enlarges the token maps at each step (e.g., step-wise patch_nums from (1, 2, …, 16) to (1, 2, …, 32)), but the per-step computation graph remains unchanged, allowing FreqExit to scale without modification. Recent work such as Infinity [5] has already extended VAR-style architectures to 1024+ resolutions while preserving their core structure, indirectly supporting the transferability of our method.

We agree that broader evaluation across datasets and resolutions would further demonstrate generalizability. We plan to extend FreqExit in these directions, but due to time and resource constraints, we are unable to provide new experiments during the rebuttal phase and appreciate the reviewer’s understanding.

Q1: Generalization of FreqExit

Q1-Ans: All publicly available VAR baselines are trained on ImageNet at 256 × 256, so we used the same setting for reproducible, fair comparison. FreqExit itself is resolution- and dataset-agnostic because it only adjusts the active‐layer depth. Training higher-resolution models (e.g., 512 × 512) would take about 24 days on 8 × A100 GPUs, which is infeasible during the rebuttal period. We therefore defer larger-scale and cross-dataset experiments to future work.

Q2: Discussion on High-Frequency Behavior

Q2-Ans: VAR generates images in a scale-by-scale manner, where each step operates on a low-resolution token map that is upsampled before the next step. In early steps, the coarse spatial resolution limits the range of frequency components that can be represented—only large-scale structures and smooth regions (i.e., low-frequency information) can be reliably synthesized. This limitation aligns with the Nyquist sampling principle, which states that higher-frequency content requires denser spatial sampling. As the token map is upsampled, the spatial granularity increases, allowing later layers to generate finer textures and high-frequency details. This leads to a natural “structure-first, detail-later” generation process.

From an information-theoretic perspective, natural images typically follow a $1/f$ power spectrum, where most of the signal energy is concentrated in the low-frequency range [6]. During training, cross-entropy loss tends to reduce large-magnitude errors first—primarily from low-frequency components. As structural errors diminish, high-frequency residuals become more prominent in the gradient signal, which explains the gradual emergence of fine details in later stages.

Q3: Wavelet Details

Q3-Ans: We use a single-level 2D Haar wavelet transform (wave=‘haar’) in FreqExit. Each $2\times2$ spatial block in the token map is linearly projected into four subbands: LL (low-frequency structure) and LH/HL/HH (directional high-frequency details). This fixed-stride decomposition preserves a one-to-one spatial correspondence between inputs and outputs, allowing frequency-aware gating and reconstruction in a unified coordinate system without extra alignment.

Compared to global Fourier transforms or block-wise DCT, Haar wavelets are better suited to our setting for three main reasons:

(1) Spatial-frequency alignment: FFT and DCT distribute frequency information globally or within local blocks, weakening spatial localization. In contrast, Haar wavelets retain both spatial and frequency information, which is better suited for frequency-aware control in spatially structured generation;

(2) Hierarchical consistency: The structure-to-detail nature of next-scale generation in VAR naturally aligns with Haar’s hierarchical LL → LH/HL/HH decomposition, in contrast to the less localized zig-zag ordering used in DCT.

(3) Computational simplicity: Haar wavelets are computationally lightweight and easy to integrate.

Overall, Haar-DWT offers aligned coordinates, hierarchical structure, and low overhead, making it a practical fit for our frequency-aware training design. We will include this discussion in the paper.

Paper Formatting Concerns: We thank the reviewer for pointing out the typos and the formatting issue with Figure 3. We will correct these issues in the paper.

Reference

[1] Zhang B, Gu S, Zhang B, et al. Styleswin: Transformer-based gan for high-resolution image generation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 11304-11314.

[2] Yang X, Zhou D, Feng J, et al. Diffusion probabilistic model made slim[C]//Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2023: 22552-22562.

[3] Huang T, You S, Wang F, et al. Knowledge distillation from a stronger teacher[J]. Advances in Neural Information Processing Systems, 2022, 35: 33716-33727.

[4] Jin Y, Wang J, Lin D. Multi-level logit distillation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 24276-24285.

[5] Han J, Liu J, Jiang Y, et al. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 15733-15744.

[6] Torralba A, Oliva A. Statistics of natural image categories[J]. Network: computation in neural systems, 2003, 14(3): 391.

Dear Reviewer NPSH,

This is a reminder that the author-reviewer discussion period is about to close.

Could you please provide your comments on the author rebuttal? We would appreciate your contribution to the discussion.

We sincerely thank the reviewer for the constructive and insightful feedback. We greatly appreciate your recognition of the strengths of our work and your valuable suggestions. We address each point in detail below.

W1: Lack of High-Resolution Experiments

W1-Ans: We thank the reviewer for the insightful comment. We regret that we are unable to provide high-resolution results during the rebuttal period. Training at 512×512 resolution is prohibitively expensive. Running 80 epochs with VAR-d36 would take about 24 days on 8×A100 GPUs (approximately 23% of the original VAR-d36 training time on A100 GPUs). We plan to include high-resolution experiments in the revision.

Notably, FreqExit is resolution-agnostic: it adjusts layer depth per generation step without modifying patching or tokenization. Although higher resolutions produce larger token maps, the layer-wise structure within each step remains unchanged, allowing seamless application of our method. Moreover, our baseline CoDe is also only evaluated at 256×256 resolution.

Finally, recent work such as Infinity [1] adopts the same next-scale generation paradigm and per-step Transformer structure at resolutions beyond 1024, further supporting the generalizability of FreqExit to high-resolution settings.

W2: Limitations of CALM under Next-Scale Generation

W2-Ans: We evaluated CALM-style exit criteria (Softmax Max, Diff, and Entropy) on VAR-d16 and VAR-d20 (see Table below), and found that all metrics remain far below typical thresholds, indicating the failure of confidence-based signals to determine viable exits.

Table 1: Layer-wise CALM Confidence Metrics Comparison

Softmax Max = probability of the top-1 token; Softmax Diff = difference between top-1 and top-2 probabilities

This is due to CALM’s core assumption that representations stabilize across layers, which does not hold in VAR. As next-scale prediction introduces high-frequency details in later steps, each layer continues to substantially update token representations. This leads to large inter-layer changes, also reflected in the low cosine similarity shown in Fig. 2(c).

These results will be included in the revision.

W3: FGSR Closed-Form Derivation

W3-Ans: We will revise the main text to improve the explanation of the FGSR module and simplify the derivation of the closed-form update in Eq. (9).

W4: Lack of Definition

W4-Ans: All frequency band notations (e.g., HH, LH) will be defined upon first use to ensure clarity. Thanks for pointing out the issue!

W5: Early Exit Strategy Clarification

W5-Ans: We report typical exit paths to enable comparison across typical Latency/FLOPs–FID trade-offs, serving as consistent baselines for evaluating inference cost and quality.

W6: Lack of benchmark Figures

W6-Ans: We will include visual plots showing performance vs. average depth and latency in the revised version. These cannot be included in the rebuttal per the NeurIPS rebuttal guidelines.

W7: Discussion of Shared LM Head

W7-Ans: We thank the reviewer for raising this point and are glad to clarify. Sharing the LM head promotes consistent representations and serves as a form of regularization. It also avoids significant parameter overhead. Adding separate heads from layer 10 onward increases parameters by 23.8%–29.9% (see Table 2 below) without improving performance.

Table 2: Overhead from adding individual LM heads to intermediate layers starting at layer 10 (excluding the final layer).

W8: Impact of Early Exit on Multi-Token Generation

W8-Ans: We thank the reviewer for raising this important concern. FreqExit adjusts inference depth but does not alter the underlying autoregressive generation scheme. The reported FID reflects the quality of full images, thereby capturing coherence across token sequences. Moreover, Fig.4 in the appendix (Zero-Shot Inpainting) shows that local consistency is well preserved.

W9: Generalization of FreqExit

W9-Ans: FreqExit targets next-scale autoregressive visual generation (e.g., image synthesis, inpainting) and already offers practical, resource-aware inference within vision. Because it operates purely at the architectural level, the same depth-adaptive strategy can transfer to hierarchical coarse-to-fine generators in NLP, audio, and video without changing the core training scheme.

Q1: Inference-Time Exit Criterion

Q1-Ans: In our current paper, early exits follow fixed depth paths chosen offline. We benchmark several representative paths and report their speed (latency / FLOPs) and quality (FID) so readers can make direct trade-off comparisons. No confidence or entropy threshold is applied at inference time. These fixed paths serve as practical heuristics and reproducible baselines for typical deployment budgets. In future work, we will explore adaptive per-sample exit rules, such as confidence signals or lightweight policy networks, to further improve flexibility.

Q2: Sensitivity Analysis of λ

Q2-Ans: FGSR uses two mechanisms for stable training: (i) a learnable sub-band gate γ_b and (ii) an auxiliary reconstruction loss. The hyper-parameter λ scales only the orthogonal alignment term L_align, regularizing sub-band projections without perturbing the main gradients. We have added a sensitivity analysis on VAR-d16 that varies λ and confirms training remains stable across a broad range of values.

Table 3 reports FID scores at two checkpoints (Epoch 8 and Epoch 16) for three λ values and three early-exit layers.

The differences across λ in [0.05, 0.20] are marginal, indicating that FGSR is insensitive to reasonable choices of λ. We therefore set λ = 0.1.

Q3: Applicability of FreqExit Beyond VAR

Q3-Ans: FreqExit is an architectural mechanism: every layer learns to emit a full sample, and inference depth is selected on demand. This layer-centric design is token-agnostic and thus transferable. We have already confirmed its effectiveness on visual generation with VAR models. We believe the same depth-adaptive scheme could be applied to other hierarchical or next-scale generators, including those in NLP, audio, and video. We will explore these modalities in future work.

Q4: Adaptive Early-Exit Strategy

Q4-Ans: We agree this is a valuable suggestion. FreqExit presently uses a few fixed exit paths to cover typical latency versus FID trade offs. Learning a per sample exit rule by using confidence cues is a promising next step. We plan to investigate it in future work.

Q5: Runtime Savings Benchmarks

Q5-Ans: We clarify that Table 1 reports latency, throughput, and GFLOPs together with FID for each exit schedule, providing a comparison of speed-quality trade offs. To make these trends clearer, We will include a performance vs. latency plot in the revision.

Q6: Impact of Shared and Sperate LM Heads

Q6-Ans: A shared LM head keeps representations consistent, serves as implicit regularization, and avoids the extra parameters incurred by per-layer heads. As shown in Table 2, adding separate heads from layer 10 onward would raise parameters by roughly 24%–30%. Consistent with prior work [2-3], we therefore keep a single shared head.

Reference

[1] Han J, Liu J, Jiang Y, et al. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 15733-15744.

[2] Elhoushi M, Shrivastava A, Liskovich D, et al. LayerSkip: Enabling early exit inference and self-speculative decoding[J]. arXiv preprint arXiv:2404.16710, 2024.

[3] Liu F, Tang Y, Liu Z, et al. Kangaroo: Lossless self-speculative decoding for accelerating llms via double early exiting[J]. Advances in Neural Information Processing Systems, 2024, 37: 11946-11965.