SkipGPT: Each Token is One of a Kind

Q1: The reviewer suggests including experiments on LLaMA3, which was released in April 2024.

Response: To address the reviewer’s suggestion, we conducted additional experiments on LLaMA3-8B, evaluating SkipGPT under both 25% and 40% sparsity settings. Due to space constraints, we compare SkipGPT with two of the most competitive baselines: ShortGPT and Shortened LLaMA (PPL). The table below presents the results under 25% sparsity, comparing SkipGPT-RT with the baselines across several downstream tasks and language modeling benchmarks:

As shown, static pruning methods like ShortGPT suffer significant degradation—e.g., ShortGPT shows extremely poor language modeling (PPL > 2700). In contrast, SkipGPT-RT maintains performance close to the dense model, highlighting the importance of dynamic pruning for large, well-trained models like LLaMA3-8B, where full layer removal is no longer viable. We observe the same trend under 40% sparsity, where SkipGPT continues to outperform static baselines by a large margin:

Here, ShortGPT fails completely as a language model, producing meaningless outputs. In contrast, SkipGPT-RT maintains over 80% of the dense model’s performance, despite having no fine-tuning beyond router tuning. For the LoRA fine-tuning phase, Shortened LLaMA (PPL) fails to converge and is thus omitted. We compare SkipGPT-RT-L (router tuning + LoRA) with ShortGPT under 25% sparsity:

SkipGPT-RT-L effectively recovers nearly all the performance of the dense model after LoRA adaptation. This conclusion holds under 40% sparsity as well:

Q2: The reviewer recommends including results on generation tasks to further evaluate the effectiveness of the method.

Response: Thank you for the helpful suggestion. To evaluate our method on generation tasks, we conducted zero-shot experiments on GSM8K (flexible match) and MBPP using LLaMA3-8B under a 40% sparsity setting. We compared SkipGPT-RT-L (router tuning + LoRA) with a strong baseline ShortGPT + LoRA, using the same sparsity level. The results are as follows:

While there is some drop in performance compared to the dense model, SkipGPT still significantly outperforms the pruning baseline, demonstrating its advantage in generation tasks.

Q3: Can you measure end-to-end latency with and without the router?

Response: Please refer to our response to Reviewer hvhB Q2. The router adds negligible overhead, as shown in our end-to-end latency and module-level breakdown. We’ll highlight this in the revision.

Q4: The reviewer asks for clarification on the LoRA configuration used in the experiments.

Response: Thank you for the suggestion. In our experiments, we use a LoRA rank of 16, applied to $W_q$, $W_v$, and $W_{\text{gate}}$. We will clearly specify these details in the main text in the revision.

Q5: Do Table 1 ratios include router/LoRA parameters?

Response: Table 1 includes router parameters. Table 2 includes both router and LoRA. We'll clarify in the revision.

Q1: The phrase "novel sparsity concept" seems exaggerated.

Response: We appreciate the reviewer’s feedback regarding the “novel sparsity concept.” Our intention was not to overstate novelty in this regard. Rather, we aimed to highlight that SkipGPT enables fully dynamic layer skipping, where both the number of layers each token passes through and the participating tokens in each module can vary dynamically across the sequence.

That said, we agree that similar ideas of global token- and layer-level budget allocation have been explored in prior works such as PyramidKV. In our case, this sparsity definition is primarily used as a control mechanism during training to specify and regulate the sparsity target, and is not the main technical contribution of our paper. We will revise the wording in the final version to better reflect this and avoid overstating novelty.

Q2: Can the method be evaluated on recent LLMs such as LLaMA-3 or Gemma-2?

Response: Thank you for the suggestion. We refer you to our response to Reviewer kVh7 Q1, where we present detailed results and analysis on LLaMA3-8B.

Q3: Does the method maintain strong performance under higher pruning ratios beyond 30%?

Response: We appreciate the reviewer’s attention to the method’s robustness under higher pruning ratios. While we have already demonstrated in Q2 that SkipGPT remains effective at pruning ratios beyond 30% using LLaMA3-8B, we further conducted a comprehensive set of experiments on LLaMA2-7B, the model originally used in our paper, under a 40% sparsity setting.

The results after router tuning are shown below:

We observe that our method outperforms all pruning baselines on both downstream accuracy and language modeling perplexity under 40% sparsity, demonstrating its superior adaptability and robustness. To further validate effectiveness, we also apply LoRA fine-tuning under the same 40% sparsity setting. The results are summarized below:

As shown, after LoRA fine-tuning, our method nearly fully recovers the performance of the dense model.

Q4: How does the method perform on long-context benchmarks such as LongBench?

Response: Thank you for the question. To evaluate our method on long-context understanding, we conducted experiments on LongBench. Specifically, we applied router tuning followed by supervised fine-tuning (SFT) on the LLaMA 3.1 8B base model using the Alpaca dataset. We compare our model (SkipGPT-RT-SFT, with 40% sparsity) against the official LLaMA-3.1-8B-Instruct, as well as a strong pruning baseline ShortGPT, which uses the same SFT configuration as our method. The following table summarizes the results (without CoT prompting):

Although LongBench remains a challenging benchmark, our method significantly outperforms the pruning baseline (ShortGPT) across all difficulty levels and context lengths. While SkipGPT-RT-SFT does not fully match the dense model’s performance, it shows strong capability in long-context scenarios—even under 40% sparsity—highlighting the effectiveness of our compressed approach.

Q5: Can you provide actual execution time comparisons rather than just theoretical speedup ratios?

Response: Thank you for your question. Please refer to our response to Reviewer hvhB Q2.

Q1: Is the definition of sparsity in the paper appropriate and reflective of practical inference efficiency?

Response: Our sparsity metric, based on skipped modules, is a simple and consistent proxy for dynamic pruning, though it doesn't directly reflect FLOPs. While actual efficiency depends on sequence length and module type, MLP and attention FLOPs are roughly comparable on average (e.g., in LLaMA2-7B), making our proxy a reasonable simplification. For fair comparisons, we match parameter sparsity with the pruning baseline. Since attention has fewer parameters and is skipped more often, this leads to lower actual FLOPs—especially for long sequences. We plan to include FLOP and latency metrics in future versions.

Q2: Can the proposed method demonstrate real-world inference speedup, and are latency measurements compared to dense baselines reported?

Response: Thank you for highlighting this important point. In practice, LLM inference consists of two phases: prefilling (processing the initial prompt) and decoding (generating tokens step by step).

(1) Prefilling phase

To quantify the latency overhead introduced by our method, we conduct detailed timing analysis on an A800 GPU using a SkipGPT model (LLaMA2-7B) with 25% sparsity. For an 80-token input, the additional operations per layer introduced by SkipGPT include attention/MLP routing, argmax, and slicing. The average per-layer latency (in seconds) is as follows:

This yields an average ~10% reduction in per-layer latency during prefilling. We also report end-to-end prefilling latency across sequence lengths for both dense and 25% sparse models, showing the sparse model’s runtime as a fraction of the dense model’s:

(2) Decoding phase

Due to token-by-token generation, SkipGPT often achieves greater-than-theoretical speedups. FlashAttention’s per-step overheads—like linear projections, RoPE, KV cache updates, and kernel setup—can’t be amortized, making attention the dominant cost. Since our router frequently skips attention, SkipGPT yields significant latency gains. The table below shows results for a 25% sparsity SkipGPT model (LLaMA2-7B) with an 80-token prompt.

As shown above, the actual speedup closely matches or even exceeds the theoretical upper bound, particularly as the sequence length increases.

(3) Hardware-aware Optimization (Ongoing FPGA Work)

We recognize that current GPUs struggle with token-wise dynamic sparsity due to memory bottlenecks and kernel overhead. To address this, we're developing a custom FPGA backend for SkipGPT that separates compute- and memory-bound phases for targeted optimization:

• Prefilling (Compute-bound): We use a dataflow PE array with sparsity-aware scheduling to boost sparse matrix throughput and reduce idle cycles.

• Decoding (Memory-bound): We design a hierarchical memory system with local buffers, lightweight scheduling, and KV cache-aware prefetching, outperforming GPU’s unified memory pipeline.

Using a 25% sparse SkipGPT (LLaMA2-7B), our FPGA prototype achieves the following normalized speedups:

For a fixed prompt length of 80 tokens, we measure the total decoding time for generating different output lengths:

Lastly, we have shown that the sparsity ratio can be further reduced to 40% without significantly copromising the model performance (See our response to RGDzU Q3 and Reviewer kVh7 Q1). We will add latency measurements and provide results under higher sparsity ratio in the revision.