We thank Reviewer 1Jfq for the excellent review and the insightful, forward-looking questions about our work's practical deployment. We are encouraged that the reviewer recognized the novelty of our core ideas and the quality of our presentation.

Interaction with Speculative Decoding

Thank you for raising this highly relevant point. We view Spark Transformer and speculative decoding as highly complementary and synergistic technologies. Our method enhances both roles within the speculative decoding framework.

• As the Target Model: Spark Transformer directly accelerates the costly verification step. While verifying multiple tokens at once means more neurons are activated compared to single-token decoding, significant sparsity is still maintained for two reasons: 1) The number of speculative tokens is typically small (e.g., 2-4), and 2) Consecutive tokens often share activated neurons (a phenomenon termed "Aggregated Sparsity"). This allows Spark Transformer to reduce the latency of the primary bottleneck in the speculative decoding loop.
• As the Draft Model: A great draft model is both fast and accurate. Because Spark Transformer nearly matches the quality of its dense counterpart at a fraction of the inference cost, it serves as an ideal high-performance draft model, potentially leading to higher token acceptance rates and greater overall speedups.

We will add a detailed discussion of this synergy to the revised manuscript.

Effectiveness in Memory-Bound Cloud Environments

This is a critical point for practical deployment. While LLM inference can indeed be memory-bound, our approach is specifically designed to provide substantial benefits in this regime by reducing both computation and memory traffic.

Our innovation is not just a theoretical FLOP reduction. As detailed in Appendix B and illustrated in Figure 4, our custom sparse kernels are implemented to avoid loading the weights corresponding to zeroed-out activations from High-Bandwidth Memory (HBM). Therefore, our activation sparsity translates directly into reduced memory I/O.

This means our reported ~2.5x FLOP reduction is coupled with a significant reduction in memory access, directly addressing the memory bandwidth bottleneck. This dual benefit also leads to substantial energy savings, a vital metric for large-scale cloud deployments. We will clarify this crucial point more explicitly in the revised paper.

Interplay with Quantization

This is another excellent suggestion. We consider activation sparsity and weight quantization to be orthogonal optimization techniques that can be combined for even greater efficiency.

While we have not yet performed these experiments, we see no fundamental conflicts. Standard methods like post-training quantization (PTQ) or quantization-aware training (QAT) should be directly applicable to Spark Transformer. We hypothesize that because our model learns activation importance scores during training, it may even prove robust to the precision loss from quantization. Investigating this powerful combination is a promising avenue for future work, and we will add it to our discussion section.

We thank Reviewer PzjZ for their thorough review and for recognizing the novelty of our formulation, the strength of our theoretical backing, and the evidence of wall-time wins. We address the specific technical questions below.

Baseline Ambiguity in Figure 5

This is an excellent point regarding clarity. To provide a direct comparison, we have conducted new decoding speed evaluations. We benchmarked our optimized Spark Gemma-2 against the original dense Gemma-2 (on CPU, with ongoing GPU experiments), and also measured the overhead of our architecture before sparsity optimizations were applied.

The results confirm a significant net speedup. The slight overhead from our operators is far outweighed by the gains from sparsity.

Huber Regularization Ablation

Thank you for this question. As briefly mentioned (L209), we empirically investigated this. We found that a non-zero hyperparameter δ for the Huber loss provided no noticeable improvement in model quality or training stability over setting δ=0. Therefore, for simplicity and to avoid introducing an unnecessary hyperparameter, we proceeded with δ=0 in all experiments, which effectively reduces to the standard soft-thresholding operator.

Comparison to Predictor-Based Methods like Deja-Vu

This is a crucial point that distinguishes our work. Our approach targets a fundamentally different challenge than methods like Deja-Vu.

• Deja-Vu was designed for older architectures (e.g., OPT) with naturally sparse activations. Its predictor identifies this pre-existing sparsity.
• Our work targets modern LLMs with non-sparse gated activations (e.g., SwiGLU). Here, no natural sparsity exists. Our statistical operator is part of an architecture that induces structured sparsity from scratch while minimizing quality loss.

Attention Sparsity on Long-Range Reasoning Tasks

This is an important consideration. Our work focuses on architectural innovation within the context of the base model's capabilities. Since the Gemma-2 models we build upon are pre-trained for an 8k context length, they do not generalize well beyond that window. Benchmarking on long-context tasks would therefore not yield meaningful results for either the dense baseline or our Spark Transformer.

We agree that evaluating a version of Spark Transformer pre-trained for a long-context regime is a vital piece of future work, but it is beyond the scope of the current study.

Training Stability and Recipe Adjustments

One of the practical advantages of our method is its stability. The Spark components integrated seamlessly with the standard Gemma-2 training recipe, requiring no adjustments to the learning rate, optimizer, or gradient scaling. This highlights the robustness and ease of adoption of our approach.

We thank Reviewer 4xrP for their feedback. We appreciate the acknowledgment of our work's strengths, including the elegant end-to-end training, the novelty of the statistical top-k operator, and the application of sparsity to both FFN and Attention. We address the noted weaknesses below.

Empirical Evaluation using Gemma-2B only

We thank the reviewer for this suggestion. The decision to focus our study on Gemma-2B was driven by the substantial computational cost of pre-training. This approach allowed us to conduct a thorough and controlled investigation on a strong, representative open-weight model, a common practice in LLM pre-training research where resources are a key constraint.

While larger models like Gemma-2 9B are an important future direction, their pre-training cost was beyond the scope of this project. By concentrating our resources, we were able to provide what is, to our knowledge, one of the first complete pre-training demonstrations of activation sparsity's feasibility in a modern LLM. We believe this foundational evidence is a valuable contribution and provides a strong empirical basis for future explorations on larger-scale models.

Code availability

While proprietary constraints on the full training data and scripts prevent a complete code release, we are committed to ensuring our work is reproducible. To this end, we have provided the following key components to the community:

• A detailed architectural blueprint of our Spark Transformer (Section 2, Figure 2), enabling straightforward re-implementation.
• Complete hyperparameter specifications for all FFN and Attention layers (Section 4.1), allowing for a faithful replication of our results.
• Implementation details and pseudo-code for our custom sparse computation kernels (Appendix B), designed for direct integration into open-source frameworks like gemma.cpp.

Comparison with Alternatives (Table D.1) and FLOPS vs. Wall-Clock Time

Thank you for these points. Regarding comparisons with prior methods, our results in Figure 1(a) show that Spark achieves a greater FLOP reduction than methods like ReLU, ReLU^2, and Top-k, while maintaining comparable model quality.

On the topic of evaluation metrics, we agree that direct wall-clock comparisons are valuable. However, fairly implementing and optimizing each baseline within our specific hardware and software framework (gemma.cpp) would be a significant engineering effort beyond the scope of this work. Therefore, to ensure a standardized and hardware-agnostic comparison against a wide range of published methods, we adopted FLOPs as our primary metric. This is an established practice in the activation sparsity literature (e.g., "ReLU strikes back"), allowing for more direct and fair comparisons with prior art.

GPU Evaluation on T4 vs. A100

We thank the reviewer for the opportunity to clarify our hardware selection. Our choice of the NVIDIA T4 was strategic, aimed at demonstrating the practical benefits of the Spark Transformer beyond high-end data center accelerators. While prior work has validated activation sparsity on premium hardware like A100s and TPUs, our goal was to prove its effectiveness on more widely accessible GPUs.

Our strong results on the T4, together with our CPU performance, show a clear path toward making powerful LLMs more efficient on commodity and resource-constrained hardware. Therefore, our findings do not simply replicate prior work but instead complement it, by confirming that these efficiency gains are achievable across a broader and more common class of hardware.

We thank Reviewer hAy9 for their positive assessment and for recognizing our work as "technically solid" with "high impact." We have found the comments very helpful in strengthening the paper and have revised the manuscript accordingly.

Generalizability and Scaling (Evaluation on Gemma-2 2B only)

We agree with the reviewer that evaluating scalability to larger models and generalizability to instruction-tuning are valuable directions for future research. We have clarified this in the paper's conclusion.

Our primary goal in this work was to provide a rigorous and foundational study of the Spark Transformer's effectiveness. To this end, we deliberately focused our significant computational resources on the Gemma-2 2B model for two key reasons:

• Controlled and In-Depth Evaluation: Concentrating on a single, state-of-the-art open model allowed for a thorough and controlled study. This approach is a common and necessary practice in resource-intensive architecture research before undertaking even costlier scaling experiments.

• Strong, Representative Evidence: Demonstrating significant FLOP reduction and wall-time speedup on a strong baseline like Gemma-2 provides compelling evidence for our approach's viability, establishing the groundwork for future scaling law analysis.

Analysis of Statistical Top-k and Applicability to MoE

This is an excellent and insightful point. As detailed in the paper, the key advantages of our method are twofold:

• Advantages over Standard Top-k:

  • Efficiency: As shown in Figure 6, its linear-time complexity avoids costly sorting operations, yielding a >10x reduction in training overhead compared to optimized approximate top-k operators.
  • Differentiability: Our use of soft-thresholding (Theorem 3.2) provides a smooth gradient signal, which we hypothesize leads to the superior convergence and final model quality we observed.

• Application to MoE Models: This is a fantastic connection, and we thank the reviewer for suggesting it. Motivated by this comment, we investigated its applicability and found it to be a highly promising direction.

  • A recent paper [a] uses a similar soft-thresholding technique for MoE routing but still relies on a costly sorting operator. Our method's ability to find the threshold in linear time could therefore address a key computational bottleneck, especially as MoEs trend toward a larger number of experts [17, 32].
  • We believe this is an exciting avenue for future work and, to reflect the value of this insight, we will add a new discussion on this topic to the revised manuscript.

[a] REMOE: ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing, ICLR 2025