DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding

We thank the reviewer for the constructive feedback and for highlighting both the strengths and limitations of our work. Below we address your main concerns.

Q1: On the anonymous repository link:

We apologize for the confusion. The hyperlink shown in the OpenReview webpage abstract appears to be misformatted due to translation from latex to markdown, but the link provided in the paper’s abstract is correct and functional. The repository can be accessed here without issue: https://anonymous.4open.science/r/DEL-F162

Q2: On DEL's adaptability beyond LayerSkip

DEL is designed to operate within early-exit self-speculative frameworks such as LayerSkip, providing dynamic, per-input control over speculation granularity using cached hidden states. As long as the baseline speculative decoding provides these inputs DEL can be adapted.

Early-exit self-speculation schemes like LayerSkip offer an efficient foundation by eliminating the need for a separate draft model, avoiding additional memory for KV cache or model parameters, and preventing recomputation of draft layers during verification. As early-exit capabilities become increasingly common, we hope DEL provides an interesting solution that other researchers may build on and adapt to such systems.

Q3: On evaluation beyond LLaMA family models

We focused on the LLaMA family due to the availability of early-exit (LayerSkip-tuned) checkpoints. However, DEL is model-agnostic and can be applied to any decoder-style transformer that supports early exits and exposes intermediate activations, as we demonstrated in the additional experiments on the latest Llama-3-8B model (see responses for reviewer #SrG8). We anticipate that more out-of-the-box LayerSkip-tuned models beyond the LLaMA family will become available, all of which can be directly used with DEL.

We thank the reviewer for the thoughtful feedback and for recognizing DEL’s practicality and potential impact. Below we address the raised concerns and questions.

Q1: On the model selection being limited to LLaMA-2 and CodeLLaMA

We focused on LLaMA-2 and CodeLLaMA due to the availability of LayerSkip-tuned versions. That said, DEL is model-agnostic and we believe can be adapted to any decoder-style transformer that supports early exits and exposes intermediate activations.

To demonstrate DEL’s applicability to newer models, we now include additional results using LayerSkip-tuned LLaMA-3-8B, expanding beyond the previously reported LLaMA-3.2-1B results in the main paper. These experiments confirm that DEL continues to deliver consistent improvements across tasks, even on latest models:

Q2: On compatibility with LoRA adapter-based models

DEL does not impose any assumptions on the underlying model architecture beyond early-exit support and access to intermediate hidden states. It is fully compatible with models fine-tuned via parameter-efficient methods such as LoRA.

In fact, early-exit capabilities can be incorporated during fine-tuning using LoRA adapters, and LayerSkip training itself can be adapted for PEFT methods. As these training approaches become more common, we expect DEL to be widely deployable, serving as a general plug-and-play controller for a range of self-speculative frameworks regardless of whether the base model uses full fine-tuning, adapters, or other lightweight tuning strategies.

Thank you for your detailed and thoughtful review. We are grateful for your recognition of DEL's practical significance and empirical strength. Below, we address each of your concerns and suggestions.

Q1: On reliance on LayerSkip-tuned models

DEL is designed to operate within early-exit self-speculative frameworks such as LayerSkip, providing dynamic, per-input control over speculation granularity using cached hidden states. As long as the baseline speculative decoding provides these inputs DEL can be adapted.

Early-exit self-speculation schemes like LayerSkip offer an efficient foundation by eliminating the need for a separate draft model, avoiding additional memory for KV cache or model parameters, and preventing recomputation of draft layers during verification. As early-exit capabilities become increasingly common, we hope DEL provides an interesting solution that other researchers may build on and adapt to such systems.

Q2: On missing fidelity metrics

For greedy decoding scenarios, we perform exact match to accept a speculated token by the target model. As a result, the generated outputs are identical, and thus fidelity is always preserved.

For sampling-based decoding, we provide ROUGE-2 fidelity results for DEL on HumanEval using CodeLLaMA-7B. Results are averaged across 10 runs:

The results indicate consistent output fidelity across methods, including DEL. We will include these results and clarifications in the final revision.

Q3:

On the TPL cost model and its memory-bound assumption. Section 5.3 (Figure 3b) provides a runtime breakdown showing that verification latency remains nearly flat as the number of verified tokens increases (1 to 32 tokens), confirming the memory-bound behavior. This justifies TPL as a suitable proxy for decoding cost and latency.

On the ablation of $\omega$. As reported in Appendix D.2 (Table 5), DEL is robust to the decay factor $\omega$. Speedup is stable in range 2.54X to 2.60X for different $\omega$ in range [0.5, 1]. This stability supports ease of DEL's deployment without requiring hyperparameter tuning.

Theoretical guarantee of TPL. While we do not present a formal derivation, Section 4.1 defines TPL as a speculation utility metric under the assumption that latency is dominated by the memory bandwidth of the underlying hardware, a common assumption among Speculation Decoding works, that is empirically validated for our setting in Section 5.3 (Figure 3b). Furthermore, the effectiveness of TPL is empirically verified by the results reported in Section 5.2 (Table 1) indicating the strong correlation between TPL improvements and actual wall-clock speedups observed across models and tasks.

Q4: On coverage of model architectures and language settings

DEL currently targets decoder-only models, which are common in generative tasks. However, the formulation is architecture-agnostic and can extend to encoder-decoder models with early-exit support.

To explore multilingual applicability, we ran WMT14-DE-EN translation with LLaMA-2-7B. Results:

DEL achieves a 2.56X speedup over vanilla, exceeding the best baseline (2.31X). The full set of experiments with the multilingual task will be included in the final revision.

Q5: On controller stability under out-of-distribution prompts

DEL does not assume in-distribution prompts. The controller makes decisions based on observed acceptance rates and internal confidence signals at inference time. However, like other speculative decoding methods, DEL works best when an alignment exist between draft and target model distributions, which is generally needed for any speclative decoding to work reasonably. Under significant distribution shift, draft errors may increase, reducing effective speculation length.

OOD robustness is an orthogonal direction for Speculative Decoding domain. We will expand our discussion of this point in Section 6 of the paper.

We thank the reviewer for the thoughtful and detailed feedback. We're encouraged that you found the empirical study and dynamic control of early-exit based self-speculative decoding valuable. Below, we address your comments and questions.

Q1: On the Novelty of DEL

DEL is, to our knowledge, the first controller to jointly and dynamically select both the exit layer ($\mathcal{E}$) and speculation length ($\gamma$) at runtime, per input, without any retraining, architecture modifications, or offline hyperparameter finetuning.

What sets DEL apart is its context-based, plug-and-play design. It leverages cached hidden states already produced during LayerSkip-style inference to evaluate draft utility in real-time with negligible overhead. This allows DEL to adaptively optimize the tradeoff between latency and acceptance rate using a unified control policy, rather than static or separately tuned heuristics for $\mathcal{E}$ and $\gamma$.

We believe this unified, context-aware formulation allows effective dynamic speculative decoding, particularly in practical deployment settings where per-input efficiency is critical.

Q2: On the performance gains

Recent infrastructure studies [1, 2] demonstrated that performance optimizations in AI inference, even if modest, have high impact in industrial settings. DEL improves the decoding speed from 2.23X (FS baseline) to 2.5X over no speculation decoding baseline. DEL achieves these speedups over SOTA baseline (FS) and across a wide range of models (1B–70B) and tasks (reasoning, summarization, language modeling, code generation), all with zero offline overhead and no need for model modifications.

These gains are especially compelling given that it requires no retraining, no hyperparameter tuning, and integrates seamlessly into early-exit self-speculation systems, like LayerSkip. We respectfully argue that ease of adoption and consistent gains are valuable in practical deployment scenarios.

[1] https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference

[2] https://cloud.google.com/blog/products/compute/accelerating-ai-inference-with-google-cloud-tpus-and-gpus

Q3: On expanding baselines

Based on the request for expanding baselines, during the rebuttal window we compared DEL with another SOTA baseline SWIFT (Xia et al., ICLR 2025), a recent self-speculation method that, like DEL, requires no retraining or architectural modification. However, SWIFT operates without early-exit capability. Below are results comparing DEL to SWIFT and LayerSkip on LLaMA-2-13B:

These results show that DEL achieves stronger gains atop LayerSkip than SWIFT does over vanilla decoding. We will include the full set of experiments and clarifications in the final revision.

EAGLE and S3D make training adaptations to the model architecture and token search process for improving drafting, which is an orthogonal approach to DEL. For example, EAGLE performs tree-structured decoding to increase acceptance rate, and train specific head atop the target model to generate drafts. DEL on the other hand is a plug-and-play controller that dynamically adjusts the exit layer and speculation length without requiring any changes to the training process.