CrossSpectra: Exploiting Cross-Layer Smoothness for Parameter-Efficient Fine-Tuning

Q1: Add baseline

Thanks for the related work. We will cite ALL the suggested papers and discuss them in the revision.

We compare with MONA[1] and LORAND[2] as they are most closely related to our Work. We add the MONA and LORAND adapters before the layer normalization of ViT-B/32, following the same experimental setup as Table 4.

Image Classification Results (ViT-B/32)

Key Differences and Advantages:

The fundamental difference lies in the adaptation mechanism:

• CrossSpectra: Follows the LoRA-style approach where adaptation is achieved through direct weight modification: W = W₀ + ΔW. This enables seamless integration without altering the original architecture.

• MONA & LORAND: Inject additional trainable layers g(x) with parameters ΔW into the main computational branch, transforming the forward pass from f(x) to f(g(x)).

Advantages of Our Approach:

• Plug-and-Play Nature: CrossSpectra maintains the original architecture integrity, allowing easy deployment without structural modifications.
• Inference Efficiency: No additional computational overhead during inference since adaptations are merged into existing weights.
• Superior Performance: Achieves 0.74% and 0.94% higher average accuracy compared to MONA and LORAND, respectively.

Q2 The methodology is described in a somewhat complex manner. Present the core ideas and advantages of the new method more intuitively, possibly through diagrams. Additionally, the mathematical formulations could be simplified where necessary.

Thank you for this valuable suggestion. We will add more intuitive diagrams and simplify the mathematical formulations in the revision. Since we cannot provide the updated PDF in this rebuttal, we explain the core ideas and advantages more intuitively below:

2.1 Core Idea in Simple Terms: Think of weight adaptations across transformer layers like a smooth curve rather than independent points. Just as a smooth curve can be efficiently represented using only its low-frequency components, we can represent all layer adaptations using sparse spectral coefficients.

2.2 Three Key Steps:

1. Skip Connections Create Smoothness: Skip connections ($H_l = H_{l-1} + f(H_{l-1})$) make adjacent layers similar, producing smooth gradient changes during fine-tuning rather than abrupt variations.

2. Smoothness Concentrates Energy: This smoothness causes ~70% of adaptation energy to concentrate in low-frequency spectral components (Figure 1), with the strongest concentration across the layer dimension.

3. Sparse Representation Enables Efficiency: Since most energy resides in low frequencies, we only store these essential coefficients and use 3D inverse FFT to reconstruct the full adaptation tensor, achieving 275× parameter reduction compared to LoRA.

2.3 Key Advantage: Unlike LoRA, which treats each layer independently, CrossSpectra exploits the natural architectural properties of Transformers, enabling joint parameterization across all layers with dramatically fewer parameters while maintaining comparable or better performance.

2.4 Intuitive Analogy: Imagine compressing a smooth signal—you need far fewer frequency components than for a noisy signal. Similarly, the smooth cross-layer adaptations in Transformers can be compressed much more efficiently than independent layer-wise adaptations.

Q1 Add baseline FaCT[1] and VB-LoRA[2]

Thank you for the suggestion. We have included comparisons with these important baseline methods (our reproduced result).

1.1 Experiment result Commonsense Reasoning (LLaMA2-7B):

Image Classification (CLIP ViT-B/32):

1.2 Discussion on CrossSpectra and Tensor factorization methods

• Architectural Awareness: Unlike other tensor factorization methods, CrossSpectra explicitly leverages the cross-layer smoothness property induced by skip connections in modern Transformers, aligning adaptations with the model's inherent architectural characteristics.

• Spectral Structure Exploitation: While existing methods perform generic low-rank decomposition, CrossSpectra exploits the frequency-domain structure of weight adaptations, concentrating parameters in the most informative low-frequency components.

• Dual Benefits: CrossSpectra simultaneously reduces trainable parameters and preserves the network's inherent layer smoothness property, leading to both efficiency gains and performance improvements.

This architecture-aware approach explains why CrossSpectra consistently outperforms both VB-LoRA (+1.2% average) and FaCT (+1.44% average) while using comparable or fewer parameters.

Q2 Including metrics like wall-clock time per epoch or GPU memory usage versus LoRA would be appreciated.

Thank you for the question. We provide a comprehensive analysis of computational scaling across different model sizes and configurations. For the memory analysis, please see our response to Q4 from reviewer puJh, which details our approach to peak memory management.

Model Performance and Training Time Comparison

We measure computational speed based on one forward pass and average performance on commonsense reasoning tasks:

Key Observations:

• Training time scales sub-linearly with model size due to sparse parameterization
• LoRA shows faster training but lower performance, highlighting the efficiency-accuracy trade-off

FFT Runtime Benchmarking

Since our implementation relies on torch.fft.ifftn, we provide runtime analysis for different 3D tensor sizes:

Scaling Analysis:

• FFT computation scales approximately O(N log N) with tensor size, as expected
• The 3D FFT overhead remains negligible compared to the attention mechanism's O(n²d) complexity
• Modern GPU implementations ensure efficient parallel computation of frequency transformations

Q1: Show evidence that the approach holds for the model with more parameters. Does the low-frequency assumption still hold at that scale?

Thank you for this important question regarding scalability.

1.1 Scaling Analysis Our theoretical framework is scale-agnostic, as the spectral properties we exploit (cross-layer gradient smoothness from skip connections) are fundamental to transformer architectures regardless of size. We have validated this across diverse model scales and modalities:

• Vision: ViT-B/32 (86M parameters) [included in paper]
• Language: RoBERTa-large (355M), LLaMA2-7B (7B), [included in paper], LLaMA-13B (13B) [new]

1.2 Experimental Validation To directly address scalability concerns, we conducted additional experiments on LLaMA2-13B. Due to resource constraints, we were unable to test larger models, but the results strongly support our approach's scalability.

Spectral Energy Distribution Across Model Sizes:

Performance on LLaMA2-13B

Key Findings:

• The low-frequency spectral bias consistently holds across all model sizes, with 60-70% of adaptation energy concentrated in low frequencies.
• The slight reduction in spectral bias for larger models (61.2% vs. 69.7%) is expected due to vanishing gradients in deeper networks, but remains sufficient for effective parameter reduction.

These results demonstrate that our approach scales effectively to larger models while maintaining the fundamental spectral properties that enable dramatic parameter efficiency.

Q2： The motivation has not been clearly stated, for example, why this algorithm is applied, and the related preliminaries (e.g. FFT).

Motivation. Existing PEFT methods such as LoRA treat each layer independently, ignoring the architectural characteristics of Transformers and resulting in parameter counts that scale linearly with model depth. This layer-independent assumption fails to exploit the architectural properties that should be preserved during fine-tuning, resulting in parameter counts that scale linearly with model depth and limiting efficiency gains for increasingly deeper architectures.

Key Insight. Skip connections in Transformers ($H_l = H_{l-1} + f(H_{l-1})$) create smooth gradient fields across layers, causing weight adaptations to concentrate ~70% of energy in low-frequency spectral components (Figure 1).

Solution.: Since smooth functions have sparse frequency representations, we exploit this cross-layer spectral structure using 3D inverse FFT to:

Reconstruct full adaptation tensors from sparse low-frequency coefficients Enable joint parameterization across all layers rather than independent treatment Achieve 275× parameter reduction vs. LoRA while preserving architectural characteristics

Why FFT: The spectral concentration in low frequencies makes FFT ideal for efficient reconstruction with minimal parameters, leveraging modern optimized implementations for computational efficiency. This approach preserves Transformer characteristics during fine-tuning, achieving comparable or better performance with dramatically fewer parameters by exploiting architectural properties rather than ignoring them.

Q3: Single Run

We fix results by conducting five independent runs for each experiment with different random seeds and report the average results:

Q4: Reconstructing full updates via iFFT each step could may potentially use memory

4.1 Memory Complexity Analysis

We acknowledge that reconstructing full updates via iFFT could temporarily require the same memory as the full adaptation tensor. During the forward pass, CrossSpectra requires reconstructing the full adaptation tensor $T_{QKV}\in \mathbb{R}^{(d×d×3L)}$ via T_QKV ← torch.fft.ifftn(C_full), which temporarily uses $O(3Ld^2)$.

4.2 Chunked reconstruction strategy

However, we could apply the Chunked reconstruction strategy for controlling peak memory usage:

Memory-Efficient Forward Pass

1. Divide layers into n chunks: {1,...,L} → {C₁, C₂, ..., Cₙ}
2. For each chunk Cᵢ containing layers [l_start, l_end]:
   -  Extract spectral coefficients: C_chunk ← C[:,:,3l_start:3l_end]
   - Reconstruct chunk: T_chunk ← iFFT3D(C_chunk)  
   - Apply adaptations: W̃ˡᴹ ← Wˡᴹ + ΔWˡᴹ for l ∈ Cᵢ, M ∈ {Q,K,V}
   - Deallocate T_chunk

4.3 Theoretical Justification

This approach is theoretically sound because:

1. Spectral Structure Preservation: Our theoretical analysis (Theorem 3.3) establishes that adaptation patterns exhibit the strongest decay in the cross-layer dimension (β₃ > β₁, β₂). The chunked approach preserves this structure within each chunk.

2. Gradient Smoothness Continuity: The cross-layer gradient smoothness property (Theorem 3.2) ensures that adjacent layers have similar adaptations. Chunking maintains this local smoothness while reducing global memory requirements.

4.4 Complexity Analysis

• Parameter Storage: Remains O(k₁k₂) (unchanged)
• Peak Memory: Reduced from O(3Ld²) to O(3Ld²/n) where n is the number of chunks

Q5: Task range limited

5.1 Robustness across different domains Thanks for the question. We have evaluated our method on a wide range of tasks, demonstrating robustness across different settings. Specifically, our experimental evaluation (Section 5) focuses on four specific task categories:

• Image Classification (IC): CLIP ViT-B/32 on 7 datasets
• Natural Language Understanding (NLU): RoBERTa-large on GLUE benchmark
• Commonsense Reasoning (CR): LLaMA2-7B on 8 reasoning benchmarks
• Arithmetic Reasoning (AR): LLaMA2-7B on GSM8K, MAWPS, SVAMP, AQuA

5.2 Missing Evaluation: As noted, we did not evaluate on generation-heavy tasks such as:

• Text generation and completion
• Machine translation
• Dialogue systems
• Creative writing tasks

This is because our evaluation design follows established practices for objective, quantifiable assessment of parameter-efficient fine-tuning methods.

• Commonsense Reasoning: Yes/No and multiple-choice questions (BoolQ, PIQA, HellaSwag, etc.)
• Natural Language Understanding: Classification tasks with definitive labels (GLUE benchmark)
• Arithmetic Reasoning: Mathematical problems with exact numerical answers
• Image Classification: Single correct class labels

This task approach enables deterministic performance measurement and fair comparison across methods. in contrast, generation-heavy tasks may be problematic, as it have

• Subjective Assessment: Text generation quality relies on human judgment or complex automated metrics (BLEU, ROUGE) that may not reflect true quality differences between parameter-efficient methods.
• Multiple Valid Outputs: Unlike classification tasks where there's one correct answer, generation tasks have numerous valid responses, making it difficult to isolate the impact of the parameter-efficient method from other factors.

Q6: Hyperparameter sensitivity, (k_1, k_2)

6.1 Parameter Selection Strategy:

• Optimal k₁ and k₂ values are determined through cross-validation on each dataset to ensure robust performance across tasks.
• We recommend maintaining the layer sparsity ratio (k₁×k₂)/(d²×3L) around 2×10⁻³ for optimal parameter efficiency.
• The cross-layer frequency range should cover approximately 20% of the spectrum (k₂/L), consistent with our theoretical analysis showing low-frequency dominance in adaptation energy (Figure 2).

6.2 Grid Search Results on Image Classification Tasks:

Key Observations:

• Performance improves with k₁ up to 4000, demonstrating the benefits of capturing more within-layer frequency components.
• k₂ shows diminishing returns beyond 6, consistent with stronger spectral decay in the cross-layer dimension.
• Optimal configuration: k₁=4000, k₂=3 (81.5% accuracy) with minimal parameter overhead.

Q1: An empirical study on the scaling of computational load with model size would be appreciated.

Thank you for the question. We provide a comprehensive analysis of computational scaling across different model sizes and configurations. For the memory analysis, please kindly see our response to Q4 from reviewer puJh, which details our approach to peak memory management.

Model Performance and Training Time Comparison

We measure computational speed based on one forward pass and average performance on commonsense reasoning tasks:

Key Observations:

• Training time scales sub-linearly with model size due to sparse parameterization
• LoRA shows faster training but lower performance, highlighting the efficiency-accuracy trade-off

FFT Runtime Benchmarking

Since our implementation relies on torch.fft.ifftn, we provide runtime analysis for different 3D tensor sizes:

Scaling Analysis:

• FFT computation scales approximately O(N log N) with tensor size, as expected
• The 3D FFT overhead remains negligible compared to the attention mechanism's O(n²d) complexity
• Modern GPU implementations ensure efficient parallel computation of frequency transformations

Q2: While QKV are information-dense, it potentially misses optimization opportunities in other components like the feed-forward layers, which also contain a large number of parameters.

Thank you for this insightful question. While adapting QKV layers follows established conventions in the LoRA literature, we conducted additional experiments to explore the impact of including feed-forward (FF) layers in our adaptation strategy.

1.1 Experimental Results:

Commonsense Reasoning Tasks (LLaMA2-7B)

Natural Language Understanding Tasks (RoBERTa-large)

Image Classification Tasks (ViT-B/32)

1.2 Analysis: Our experiments reveal that including feed-forward layers consistently leads to performance degradation across all modalities. This counterintuitive result can be attributed to:

1. Overfitting Risk: Adding more adaptable parameters increases the likelihood of overfitting, particularly given our extremely sparse parameterization
2. Spectral Mismatch: Feed-forward layers may not exhibit the same cross-layer spectral properties as attention mechanisms, making them less suitable for our frequency-domain approach
3. Information Density: QKV matrices capture the core relational information in transformers, while FF layers primarily perform non-linear transformations that may not benefit from cross-layer frequency sharing

These findings validate our design choice to focus on attention mechanisms, where the cross-layer smoothness induced by skip connections is most pronounced and beneficial for spectral parameterization.

Q3: Hyperparameter Tuning for $k_1$ and $k_2$

3.1 Parameter Selection Strategy:

• Optimal k₁ and k₂ values are determined through cross-validation on each dataset to ensure robust performance across tasks.
• We recommend maintaining the layer sparsity ratio (k₁×k₂)/(d²×3L) around 2×10⁻³ for optimal parameter efficiency.
• The cross-layer frequency range should cover approximately 20% of the spectrum (k₂/L), consistent with our theoretical analysis showing low-frequency dominance in adaptation energy (Figure 2).

3.2 Grid Search Results on Image Classification Tasks:

Key Observations:

• Performance improves with k₁ up to 4000, demonstrating the benefits of capturing more within-layer frequency components.
• k₂ shows diminishing returns beyond 6, consistent with stronger spectral decay in the cross-layer dimension.
• Optimal configuration: k₁=4000, k₂=3 (81.5% accuracy) with minimal parameter overhead.