SPARQ: Outlier-free SpeechLM with Fast Adaptation and Robust Quantization

Reviewer's Comment: LoRA experiments: I would also suggest…

Response: Thank you for this valuable suggestion. We agree that separating the experiments in Sec. 4.2 and Table 2 to provide clearer insights into the individual and combined effects of LoRA and quantization. We have restructured our experiments accordingly:

(1). Impact of LoRA in FP 16:

We first evaluate LoRA's impact without quantization to isolate adaptation effects. As shown in the results below, SpARQ significantly improves stability during low-rank adaptation even in FP 16:

This demonstrates that SpARQ's outlier mitigation is inherently beneficial for adaptation stability, reducing performance degradation from 381.00% to 116.75% even before quantization is applied.

(2). Combined Effects with Quantization:

We then investigate how different quantization levels affect LoRA-adapted models:

The results show that SpARQ maintains its benefits under aggressive quantization, reducing performance degradation from 330.47% to 136.07% at 8-bit quantization and from 387.50% to 176.52% at 4-bit quantization. The performance degradation with SpARQ shows a more gradual pattern across quantization levels, suggesting better stability under increasingly aggressive compression. Furthermore, the confounding effects between LoRA and quantization are significantly mitigated by SpARQ, as evidenced by the smaller performance gaps between 8-bit and 4-bit configurations.

This restructured analysis better illuminates how SpARQ helps in both stabilizing low-rank adaptation independently and maintaining stability when combining adaptation with aggressive quantization. We have add the results onto our paper Appendix G.5 to reflect this clearer experimental organization and analysis. Thank you for this suggestion which has helped better demonstrate the distinct benefits of our approach.

Reviewer's Comment: Quantization experiments

Response: Thank you for this valuable suggestion about comparing with weight clipping approaches. We have conducted additional experiments combining our method with different clipping strategies (1% and 5% tail weight values) during quantization scale computation on the OPT-350m:

The results reveal several interesting findings about weight clipping in our multimodal setting. For 8-bit quantization, traditional weight clipping actually leads to significant performance degradation in both frameworks. The vanilla framework with 1% clipping shows a 49.18% performance drop, which worsens to 71.30% with 5% clipping. While SpARQ exhibits slightly better resilience to clipping (39.21% and 55.75% drops for 1% and 5% clipping respectively), it still performs best without clipping (0.36% drop).

This pattern becomes even more pronounced in 4-bit quantization, where SpARQ without clipping achieves a 130.71% performance drop, substantially outperforming both vanilla and clipped variants. The vanilla framework with clipping shows severe degradation (267.56% and 273.36% for 1% and 5% respectively), while SpARQ with clipping, though better than vanilla, still underperforms compared to SpARQ without clipping.

These results suggest that traditional weight clipping, while effective for single-modality models, may not be suitable for multimodal speech-text tasks. We hypothesize this is because weight clipping indiscriminately removes outliers that might be important for cross-modal interactions. In contrast, SpARQ's outlier-free layer approach preserves these critical cross-modal features while selectively managing outliers that could harm quantization stability.

We will include these results in the paper Appendix G.4 to provide a more comprehensive comparison with traditional outlier mitigation techniques.

Reviewer's Comment: where does the training efficiency…

Response: By reducing attention to outlier tokens, we avoid unnecessary computation. Our approach's efficiency is backed by formal theoretical analysis in Proposition 3.2 (Section 3.4). The proposition establishes that efficient training is achievable when the sequence length L and embedding dimension d satisfy d = O(log L), and proper normalization of input and model weights is maintained. Our max-shift normalization and stabilized outlier-free layer satisfy these conditions, enabling more stable gradient flow and better concentration of attention on informative tokens while reducing computational overhead from outlier token processing. These theoretical properties directly contribute to the observed 1.33x speedup in full model fine-tuning.

Reviewer's Comment: The primary contribution…

Response: While we acknowledge that our work builds upon the Outlier-Efficient Hopfield Layer, SpARQ's contribution extends beyond mere integration. Our key innovation lies in addressing the unique challenges of cross-modal adaptation in SpeechLMs through:

• (1). Scalability to Large Models: Our stabilization mechanism enables the practical application of outlier-free techniques to much larger models than previous work. It based on a recent work [1] which demonstrates that attention efficiency critically depends on bounding input magnitudes. The key insight is that max-shift normalization helps maintain input magnitudes within theoretically optimal bounds. While the original Outlier-Efficient Hopfield Layer work was limited to smaller models, our approach successfully scales to models with over 1.3B parameters, by adding a stabilized module, with theoretical guarantees suggesting extensibility to even larger scales.

• (2). Cross-Modal Adaptation: We address the unique challenges of speech-text fusion through: A novel stabilization mechanism specifically designed for multi-modal learning; Analysis of conditions for effective low-rank adaptation (Theorem C.1), providing insights into why our approach can help with cross-modal transfer; Empirical demonstration of improved quantization stability in multi-modal settings

• (3). Practical Improvements: 1.33x training speedup for large models; 41% improvement in cross-modal low-rank adaptation; 45% improvement in quantization performance

[1] Alman, J., & Song, Z. (2024). Fast attention requires bounded entries. Advances in Neural Information Processing Systems, 36.

Reviewer's Comment: Techniques like Clipped Softmax…

Response: Thank you for suggesting the comparison with Clipped Softmax and Gated Attention. We have conducted additional experiments comparing SpARQ with these alternative techniques on the OPT-350m:

The results demonstrate that at 8-bit quantization, all methods perform similarly well, with performance drops under 0.5%. This suggests that for moderate quantization, the choice of outlier mitigation strategy is less critical. However, the differences become pronounced at 4-bit quantization, where SpARQ achieves a 130.71% performance drop compared to 215.03% for Gated Attention, 223.72% for Clipped Softmax, and 233.36% for the vanilla approach.

SpARQ's superior performance under aggressive quantization (4-bit) can be attributed to its architectural approach to outlier mitigation, which addresses the issue at its source rather than merely limiting attention values. This is particularly important for speech-text cross-modal tasks where maintaining precise attention patterns is crucial for accurate modality fusion.

These results validate SpARQ's effectiveness compared to alternative attention modification approaches, particularly in challenging low-bit quantization scenarios. Thank you for suggesting these important baseline comparisons. We will add these results to Appendix G.1 for clarity.

Reviewer's Comment: It remains unclear how much…

Response: Thank you for raising this important question about the individual contributions of SpARQ's components. We have conducted an additional experiment to isolate the effect of max-shift stabilization by applying it to a standard Transformer architecture (OPT-1.3b):

The results show that applying max-shift stabilization alone to a standard Transformer provides minimal improvement (34.4% vs 34.3% validation accuracy). This suggests that the significant performance improvements we observe in SpARQ (as shown in Tables 1 and 2) are primarily attributable to the synergistic combination of the outlier-free Hopfield layer and max-shift stabilization, rather than either component alone.

The minimal improvement from stabilization alone is understandable given that max-shift stabilization was specifically designed to address numerical stability issues that arise when using the outlier-free layer, rather than to directly improve model performance. These results help clarify that SpARQ's effectiveness comes from the complementary nature of its components: the outlier-free layer provides the fundamental mechanism for outlier mitigation, while max-shift stabilization enables practical training with this architecture. We will add these results to Appendix G.2 for clarity.

Reviewer's Comment: The author states that outliers negatively…

Response:

Thank you for pointing out the observed performance of QLoRA versus LoRA, particularly with OPT-350M. We agree that clarification on this result is necessary to enhance understanding.

Outliers negatively impact both LoRA and quantization performance by skewing attention distributions and introducing numerical instability. However, QLoRA incorporates quantization-aware optimizations, such as more precise activation handling and selective dequantization during forward passes. These optimizations mitigate the effects of outliers, enabling QLoRA to maintain stable adaptation even in the presence of such challenges. Additionally, QLoRA often achieves faster convergence compared to standard LoRA due to its fine-grained parameter updates, which allow it to adapt more quickly to task-specific distributions.

In cases like OPT-350M, where the model size is moderate and quantization effects are more pronounced, QLoRA’s faster convergence and improved handling of quantized activations allow it to outperform LoRA, despite the presence of outliers. These advantages become more evident in scenarios where computational resources are limited or where achieving rapid adaptation is critical.

We appreciate your observation and have revised the discussion to emphasize QLoRA’s faster convergence and its impact on performance relative to LoRA. Thank you again for your valuable feedback.

Reviewer's Comment: Previous studies…

Response: Thank you for raising this important point about out-of-domain (OOD) evaluation. While work [1] highlights the potential benefits of outlier features for OOD robustness, it's important to note that our work addresses a fundamentally different challenge in the speech-text domain. SpARQ focuses specifically on improving the efficiency and stability of cross-modal adaptation and quantization in speech processing tasks, where the primary challenges are parameter-efficient adaptation between speech and text modalities; maintaining speech recognition and synthesis quality under aggressive quantization; and ensuring stable training across modalities with varying sequence lengths.

In speech processing tasks, particularly ASR and TTS, the concept of OOD generalization differs significantly from traditional classification tasks. The primary metrics of concern are Word Error Rate (WER) and perplexity across different speakers and acoustic conditions, which we have extensively evaluated in our experiments. Our results show that SpARQ maintains strong performance on these metrics even under challenging conditions of low-rank adaptation (41% improvement) and aggressive quantization (45% improvement).

While we acknowledge the importance of OOD evaluation in general machine learning contexts, our focus on speech-specific tasks and metrics provides a more relevant assessment of SpARQ's practical utility in real-world speech applications. The stability improvements we demonstrate are particularly crucial for maintaining reliable speech processing performance across different deployment scenarios.

[1]Crabbé, J., Rodríguez, P., Shankar, V., Zappella, L., & Blaas, A. (2023). Robust multimodal models have outlier features and encode more concepts. arXiv preprint arXiv:2310.13040.

Reviewer's Comment: Do Table 1 and Table 2 reflect …

Response: Thank you for this insightful question. You are correct that Tables 1 and 2 show separate experiments for PTQ and LoRA, respectively. To provide a complete picture of how SpARQ performs when combining these techniques, we conducted additional experiments applying post-training quantization (smoothquant) to LoRA-trained models (OPT-350m).

The results demonstrate that SpARQ maintains its advantages even when combining LoRA with smoothquant (SQ). When applying 8-bit quantization to LoRA-trained models, SpARQ exhibits a significantly lower performance drop (155.95%) compared to the vanilla framework (396.34%). This advantage persists even under more aggressive 4-bit quantization, where SpARQ shows a 186.66% performance drop versus 417.52% for the vanilla framework.

Most notably, SpARQ's ASR performance remains relatively stable under quantization (18.52% to 28.32% WER), while the vanilla framework shows severe degradation (93.91% to 97.24% WER). These results further validate SpARQ's effectiveness in handling outliers, as it maintains better performance not just in individual scenarios (as shown in Tables 1 and 2) but also when combining low-rank adaptation with quantization. This is particularly important for practical applications where both model compression techniques might need to be applied simultaneously.

We will update the paper to include these results in Appendix G.3, providing a more comprehensive evaluation of our method's robustness across different compression scenarios.

Reviewer's Comment: Testing Constraints

Response: Thank you for your valuable feedback. We acknowledge the computational limitations that restricted testing to OPT models up to 1.3B parameters. However, our theoretical analysis in Section C.2 provides expressiveness guarantees that should extend to larger models. Specifically, Theorem C.1 establishes that under non-singularity assumptions, our approach can exactly represent target models regardless of size, suggesting scalability to larger architectures. Additionally, our empirical results show consistent improvements as model size increases (from 125M to 1.3B), with relative performance gains of up to 45% in quantization for OPT-1.3B, indicating a positive scaling trend. As explicitly stated in our limitations section, we plan to extend SpARQ to test different larger decoder-based models in future work, including the 3B LLama2 and 6.7B OPT models. This future work will help validate our theoretical predictions about scalability and provide valuable insights about the framework's performance on state-of-the-art large language models."

Reviewer's Comment: Architectural Complexity

Response: Thank you for your insightful feedback. While our architecture does introduce additional complexity through the stabilized outlier-free layer, we've provided detailed implementation guidance in Section 3.2 and comprehensive theoretical justification in Section 3.4. The improved training efficiency (1.33x speedup for OPT-1.3B) and robust quantization performance justify this added complexity. The architecture's design prioritizes practical utility while maintaining compatibility with existing transformer implementations, making it accessible to practitioners with basic understanding of transformer models. Our codebase will further facilitate easy adoption of SpARQ in real-world applications. To support broader adoption, we are preparing detailed implementation guides, including modular codebases, to ease integration into existing workflows. We also plan to release pretrained models to mitigate entry barriers for practitioners without extensive technical expertise.

Reviewer's Comment: Risk of Bias Amplification

Response: We appreciate this important concern. While our framework focuses on computational efficiency, several aspects of our approach may actually help mitigate bias amplification. The outlier-free layer's ability to focus attention on significant tokens (demonstrated in Figure 4) means that important contextual information is preserved during adaptation, potentially helping maintain model fairness. However, we agree that further investigation of bias effects is warranted and suggest this as an important direction for future work.

Reviewer's Comment: Given the computational limitations…

Response: Based on our theoretical analysis and empirical observations, we expect SpARQ to maintain its benefits when scaled to larger models like OPT-6.7B for several reasons:

• (1). Our expressiveness guarantee (Theorem C.1) shows that SpARQ can exactly represent target models under mild conditions, independent of model size.

• (2). The relative performance improvements increase with model size in our experiments (from OPT-125M to OPT-1.3B).

• (3). The stabilization mechanism becomes more critical for larger models where gradient issues are more pronounced (demonstrated in Table 3).

Reviewer's Comment: Several stabilization methods are compared…

Response: Max-shift normalization was selected based on its superior ability to resolve gradient instability, as demonstrated in Table 3. Unlike L1 and mean-centering normalization, max-shift ensures a tighter control over the numerical stability of activations, especially in the presence of outliers.

There might be specific scenarios where alternative methods could be more appropriate:

• L1 Normalization might be suitable for cases for sparse attention distributions because it directly minimizes the sum of absolute values (∑|x|), which helps control the scale of outliers while preserving sparsity patterns.

• Mean-centering might work better for normally distributed attention scores as it centers the distribution around zero, making subsequent operations numerically stable. It helps maintain consistent variance across attention heads by removing systematic biases in attention scores.

Reviewer's Comment: Show in experiments explicitly that this indeed happens in practice also. Noting that this is different to showing general performance of the model in some downstream tasks.

Response: We appreciate the reviewer's insightful comment and acknowledge the importance of demonstrating that SpARQ's outlier-free architecture directly addresses outliers during cross-modal low-rank adaptation and post-training quantization, beyond just downstream task performance.

In response, we have added additional experimental results to explicitly validate SpARQ's effectiveness in mitigating outliers:

• Outlier Reduction in Cross-Modal Low-Rank Adaptation:

We conducted a detailed outlier analysis comparing SpARQ with the vanilla SpeechLM framework using maximum infinity norm and average kurtosis metrics across text, speech, and cross-modal inputs. As demonstrated in Figure 3 (Section 4.3), SpARQ significantly reduces both metrics, indicating fewer outliers during adaptation. Specifically, in ASR tasks with cross-modal inputs, SpARQ reduces the maximum infinity norm by approximately 39%, showcasing its ability to handle mixed speech and text inputs more effectively.

These results provide concrete evidence that SpARQ's outlier-free layer stabilizes the attention distribution, especially when adapting text-based LLMs to speech inputs.

• Visual Evidence of Outlier Reduction:

In Appendix A, we provide heatmaps visualizing the attention patterns for the vanilla and SpARQ models during fine-tuning. These visualizations clearly illustrate that SpARQ achieves a more concentrated distribution of attention weights, focusing on significant tokens while suppressing outlier attention. This directly correlates with reduced outlier effects during both adaptation and quantization stages.

• Outlier Mitigation During Post-Training Quantization:

In response to your concern, we conducted additional experiments using the OPT-1.3b model under various low-bit quantization settings (e.g., W4A4). As shown in Table 1 (Section 4.1), SpARQ achieves up to a 45% reduction in performance degradation relative to the vanilla framework when applying SmoothQuant, AffineQuant, and OmniQuant methods. This highlights SpARQ's robustness in reducing outlier effects that degrade performance during quantization. These findings align with those in [1], particularly Table 1 and Table 2, which demonstrate similar reductions in outliers during transformer adaptation. By leveraging a stabilized outlier-free layer, SpARQ shows a consistent improvement in reducing quantization-induced performance drops.

By explicitly including these analyses, we provide clear evidence that SpARQ's design can addresses the challenges posed by outliers during the critical stages of model adaptation and quantization.

[1] Hu, Jerry Yao-Chieh, et al. "Outlier-efficient hopfield layers for large transformer-based models." arXiv preprint arXiv:2404.03828 (2024).

[2] Evan Miller. Blog post: Attention is off by one, 2023. URL https://www.evanmiller.org/attention-is-off-by-one. html. Accessed: July 4, 2024.

Reviewer's Comment: Definition of the outlier in the context of the paper. Especially considering that classical definition of outlier in statistics and ML is that observation is not from the distribution that is being modeled.

Response: Thanks for your comments. The definition of the outlier is different from the classical definition. Outliers in SPARQ, as referred from [1], are defined as low-information tokens or activations that disproportionately affect the attention mechanism in transformer models, leading to inefficiencies and performance degradation. These outliers are not data points that are outside the modeled distribution but rather models that arise internally during the attention computation process.

**Reviewer's Comment: How does the equation 3.1 relate to outliers? Add theoretical analysis to show.

Response: We appreciate the opportunity to elaborate on the significance of Equation 3.1 and its relation to outliers in transformer-based models. In our work, we define outliers as low-information tokens or activations that disproportionately influence the attention mechanism, leading to inefficiencies and degraded performance. Equation 3.1 describes the core attention mechanism in transformers, where the output is composed of two key components: the residual connection and the attention calculation:

$$\text{Output} = \text{Residual} (X + \text{Attention}),$$

where:

$$\text{Attention} = \text{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V.$$

This structure highlights two critical aspects relevant to outliers:

1. Residual Connection: The residual connection ensures that the model retains the original input information $X$ in its output. However, if the attention calculation introduces large distortions caused by outliers, the residual connection alone cannot counteract these distortions.

2. Attention Calculation: The attention mechanism calculates attention weights using a softmax function applied to the scaled dot product of queries $Q$ and keys $K$. These weights are then used to compute a weighted sum of values $V$.

The use of the softmax function makes the attention calculation highly sensitive to the presence of outliers—tokens that have abnormally high or low attention scores.

Theoretical Analysis of Outliers in Equation 3.1

1. Outlier Behavior in Attention Mechanism

The softmax function inherently assigns non-zero probabilities to all tokens, as shown below:

$$\text{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)_i = \frac{\exp\left(\frac{q_i k_i^\top}{\sqrt{d}}\right)}{\sum_j \exp\left(\frac{q_j k_j^\top}{\sqrt{d}}\right)}.$$

• Sensitivity to Large Values: If $$q_i k_i^\top$$ is much larger than the other dot products in the sequence, the corresponding token $i$ will dominate the attention distribution, even if it is low-information (e.g., a delimiter or punctuation mark).

• Sensitivity to Noise: Even small numerical instabilities in $$q_j k_j^\top$$ can disproportionately impact the attention weights due to the exponential nature of softmax, causing irrelevant tokens to receive high attention.

If the attention input $X$ already holds sufficient information, the transformer should prevent it from making updates. In such cases, the output $$\text{Output}$$ should equal the input $X$. At this time, the attention output of the input $X$ should ideally be close to 0. However, the attention mechanism with softmax is a probability-based equation. When tokens with large attention values are assigned smaller probabilities, the remaining tokens—often those with small attention values—are given disproportionately higher probabilities.

Unfortunately, these small attention value inputs are often no-op tokens, such as delimiters, end-of-sequence (eos) tokens, or other low-information tokens. Ideally, these no-op tokens should also receive near-zero attention probabilities, as they do not contribute meaningful information. However, the standard softmax function fails to suppress them effectively, leading to inefficient attention distributions.

2. Residual and Attention Contribution

The output of Equation 3.1 can be decomposed into:

$$\text{Output} = X + \left(\text{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V\right).$$

• The residual connection $X$ ensures that the model retains the original input. However, when the attention term introduces outliers, it disrupts the balance between the residual and attention components.

• In the presence of outliers, the attention weights are skewed, amplifying irrelevant tokens and diminishing the contribution of meaningful tokens. This undermines the transformer’s ability to focus on important information.

---

Below, we also summarize the key points in our responses:

Key Points in Our Responses

Reviewer RFsE

• We addressed the concern about the lack of a clear definition of outliers by clarifying that outliers in SPARQ are defined as low-information tokens or activations that disproportionately affect the attention mechanism, diverging from the classical definition in statistics and ML.

• We clarified the connection between Equation 3.1 and outliers, emphasizing that the softmax function in the attention mechanism is the main reason to cause the outlier proposed in SPARQ.

• We provided additional theoretical justification for Equation 3.2, showing how the Softmax1 function modifies the denominator to dampen the influence of outliers. This analysis highlights its resistance to low-information tokens and supports the claim with limiting behavior comparisons.

• We classify our experiments results to show the outlier issue happen in practice.

Reviewer Pk8X

• We addressed the concern about testing constraints by highlighting our theoretical guarantees (Theorem C.1), which ensure scalability to larger models under non-singularity assumptions, and by providing empirical evidence of consistent performance improvements across increasing model sizes. Future work will extend testing to larger models, such as LLama2-3B and OPT-6.7B, to validate scalability further.

• We clarified the added complexity of our architectural design by demonstrating its benefits, including a 1.33x training speedup for OPT-1.3B and robust quantization performance. Implementation guidance in Section 3.2 and theoretical support in Section 3.4 make the design accessible. To encourage adoption, we plan to release modular codebases and pretrained models to simplify integration into existing workflows.

• We clarified how the outlier-free layer mitigates risk of bias amplification by focusing attention on significant tokens, as illustrated in Figure 4. This approach helps maintain important contextual information during adaptation. Further analysis of fairness impacts will be pursued in future work.

• We conducted additional experiments to compare SpARQ's effectiveness on larger models. These tests will include OPT-6.7B to confirm that our stabilization mechanisms, which become more critical with model size, maintain their benefits.

• We clarified the choice of max-shift normalization by explaining its superior performance in resolving gradient instability, computational efficiency, and robustness to outliers. Comparative insights on L1 normalization and mean-centering were also provided, highlighting their trade-offs relative to max-shift.

Reviewer H2gv

1. We addressed the concern about the effectiveness of SpARQ in quantization experiments by comparing it with weight clipping strategies (1% and 5% tail values) on OPT-350M. Results show that SpARQ significantly outperforms vanilla and clipped approaches, especially in 4-bit quantization, where SpARQ achieves a 130.71% performance drop compared to 199.02% and 212.89% for 1% and 5% clipping, respectively. These findings confirm SpARQ’s targeted approach is better suited for multimodal tasks.

2. We clarified the rationale behind the rank-128 configuration in LoRA experiments. For models like OPT-1.3B, LoRA parameters only account for 3.9% of the full model size, yet SpARQ maintains strong performance compared to vanilla frameworks, which suffer severe degradation due to unmitigated outliers. This highlights that SpARQ’s outlier-free design is the key factor in effective adaptation, not parameter count alone.

3. We clarified the separation of adaptation (FP16) and quantization effects in LoRA experiments. SpARQ reduces performance degradation during low-rank adaptation from 381.00% to 116.75% in FP16 and maintains benefits under quantization, achieving a 136.07% drop at 8-bit compared to 330.47% for vanilla. This separation highlights SpARQ's unique advantage in stabilizing both adaptation and compression stages.

4. We addressed the observed ASR WER patterns by showing that larger models amplify outlier effects, leading to severe performance degradation in vanilla frameworks. SpARQ mitigates these effects, allowing larger models like OPT-1.3B to leverage their capacity effectively, reducing WER from 46.92% (vanilla) to 8.25%. This demonstrates the importance of outlier management in scaling cross-modal tasks.

5. We clarified SpARQ’s training efficiency through theoretical analysis in Proposition 3.2. By bounding input magnitudes and reducing attention to outliers, SpARQ achieves stable gradient flow and avoids unnecessary computation, leading to a 1.33x training speedup in full model fine-tuning.