Parameter-Efficient Fine-Tuning of State Space Models

Thank you for your thoughtful feedback. We've addressed all your concerns below.

Q1: Lemma 2 Clarifications

• (i) Use $\odot$ or $\circ$ for Hadamard product?
• (ii) Clarify "same permutation"?
• (iii) (5) seems independent of $f_0$—is this correct?
• (iv) What does "all hidden dimensions are active" mean?
• (v) Is it the exact or an upper bound on tunable parameters?

• (i) Fixed.

• (ii) "Up to the same permutation" means that the same permutation matrix $P$ is applied to all three parameters of the initial model $f_0$. This yields a model that is functionally equivalent but with permuted hidden dimensions, giving $\Theta_0= \\{(P^\top \overline{A}_0 P, P^\top \overline{B}_0, C_0 P) : P \text{ is a permutation matrix}\\}.$ Here, $\overline{A}_0$, $\overline{B}_0$, and $C_0$ are the parameters of $f_0$ before fine-tuning.

• (iii) The quantity in (5) does, in fact, depend on the initial model $f_0$ through the constraint set $\Theta_0$. To prevent confusion, we rephrased Lemma 2 to highlight this constraint set.

• (iv) It means that (for target model $f_\star$), all hidden dimensions are non-zero.

• (v) When the assumption (iv) holds, it refers to the exact minimum.

Preview Revised Lemma 2 with improved clarity (https://anonymous.4open.science/r/ce7/l2.png). Notations are from Sec. 5.1.

Q2: Theorem 1

• (i) Define $L^\star$ and $H^\star$—are they from the target model?
• (ii) Does "selectively fine-tuning" imply a specific algorithm?
• (iii) Clarify "accurately represent."

• (i) $L^\star$ and $H^\star$ are indeed the number of layers and the hidden dimension of the target model $f_\star.$ We have updated statement for a better introduction of the target model and the corresponding notations.
• (ii) The statement focuses solely on model expressiveness, independent of training algorithms.
• (iii) It means "functionally equivalent," i.e., the fine-tuned model $f$ satisfies $f(x) = f_\star (x)$ for all input sequences $x$.

Preview Revised version of Theorem 1 with improved clarity (https://anonymous.4open.science/r/ce7/t1.png).

Q3: Lemma 1

• (i) Lemma 1 covers only expressiveness, not optimization—please comment.
• (ii) Lemma 1 suggests fine-tuning linear projections alone is sufficient; does this diminish your method’s significance?

• (i) New Exp. Lemma 1 analyzes only theoretical expressiveness. Empirical results (https://anonymous.4open.science/r/ce7/f1.png) confirm fine-tuning $W_{\text{in}}$ alone matches tuning $W_B$, $W_C$, $W_{\Delta, \uparrow}$ across three GLUE tasks.
• (ii) Great point. To clarify, in Lemma 1, $A$ and $W_{\Delta,\downarrow}$ are fixed. While $W_{\text{in}}$'s expressiveness includes that of $W_B$, $W_C$, and $W_{\Delta,\uparrow}$, it does not cover the expressiveness of $A$, which is essential for seq2seq operations. Thus, $W_{\text{in}}$ offers a portion of expressivity for SSM, yet it alone remains insufficient to attain optimal performance.

Preview  Updated Lemma 1 to emphasize fixed $A$ (https://anonymous.4open.science/r/ce7/l1.png) and will further smooth the Sec. 4–5 transition.

Q4: Be clear whether the model is S4 or S4D.

We specifically use S4D (diagonal $A$) and will clarify this in Sec. 1, 3, and 5.

Q5: Fine-tuning strategies lack mathematical details.

Thanks. We'll revise Sec. 3 ("Preliminaries") to include mathematical details on fine-tuning methods.

Q6: What is the fine-tuning time compared to LoRA?

New Exp. Based on your feedback, we conducted two additional experiments: (i: https://anonymous.4open.science/r/ce7/f2.png) a performance comparison between SDT and LoRA under varying time budgets in a synthetic setting; and (ii: https://anonymous.4open.science/r/ce7/f3.png) a runtime analysis of SDT and LoRA for training on a single batch using a pretrained model, across different sequence lengths. We observe that our method is slightly more efficient than LoRA, particularly as the sequence length increases, because LoRA introduces additional matrix multiplications, while SDT does not.

Q7: How were $\alpha$, $\beta$ in Table 3/4 chosen, and why is it a fair comparison to LoRA?

New Exp We fix $\beta = 0.99$ and sweep $\alpha \in \\{0.75, 0.90, 0.95\\}$. To ensure a fair comparison with LoRA, we use similar parameter budgets and hyperparameter sets of the same size for both methods: 45 configurations per method (15 learning rates × 3 method-specific settings). For LoRA, we additionally compare the three chosen configurations against alternative settings with similar parameter budgets (https://anonymous.4open.science/r/ce7/f4.png), showing that our selected configurations perform reasonably well.

Final Note: In addition, we’ll summarize all notations in the appendix. Since you said raising your score would be “an easy reach” once issues were addressed, we hope our response meet that bar. We’d appreciate an updated score and are happy to clarify anything further.

We thank the reviewer for the encouraging feedback, particularly for recognizing that (i) our method is creative and rational, (ii) our claims are supported by extensive experiments and theoretical analysis, (iii) our theoretical claims are sound, (iv) we advance SOTA results in fine-tuning SSM-based and hybrid models, and (v) our contributions are well-integrated with the broader scientific literature.

Q: Limited applicability beyond SSM-based and hybrid models.

Our proposed SDT algorithm is intentionally tailored for SSMs, with our theoretical analysis and experiments specifically focused on this model class.

Regarding your concern, while applying SDT to pure Transformers is beyond our current scope, we demonstrate its broader potential through experiments on Mamba-II, whose SSD module closely resembles attention [1]. When applied to SSD modules with LoRA on linear projection matrices, SDT consistently outperforms LoRA alone on Mamba-II across diverse tasks (Sec E.2).

Furthermore, we believe the theoretical insights and proof techniques from our analysis could prove valuable for developing PEFT methods for other architectures.

References:

[1] Dao, T. and Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning, 2024.

Final Note: Thank you again for your valuable comments. We are grateful that you appreciate our paper’s contributions in theory, methodology, and experimentation. If you have any remaining questions, please do not hesitate to let us know. Assuming our responses have addressed your concerns satisfactorily, we kindly ask you to consider raising your score and supporting our paper.

We thank the reviewer for acknowledging that (i) our paper is well written, (ii) it tackles the gap where PEFT studies favor Transformer over SSM, and (iii) it contains many experimental results to support the claims.

Q1: The proposed PEFT is incremental, such idea is mainly from the PEFT for transformer-based LLM.

We respectfully disagree. SDT is not just an incremental tweak of PEFT for Transformer-based LLMs—it’s built on a fresh theoretical analysis tailored for SSMs’ unique parameter structure. Beyond that, our paper delivers substantial contributions that stand firm, even if the method is viewed as incremental by the reviewer:

• (i) (One of the First) A systematic benchmark of PEFT methods on SSM models,
• (ii) (First) A theoretical analysis of PEFT in the SSM setting, and
• (iii) (One of the First) A new PEFT method designed specifically for SSMs, supported by both theory and strong empirical results.

Together, these contributions offer new insights into PEFT beyond what has been explored in Transformer-based LLMs.

Q2: The theoretical analysis is mainly for S4 or S6, how about S5 and Mamba2?

• New Theoretical Results S5: Although our original analysis primarily focused on S4 and S6, based on your comment, we have extended our theoretical results (Lemma 2 and Theorem 1) to include S5. Specifically, Lemma 2, which characterizes the minimal number of parameters required to update a frozen S4 model for functional equivalence to a target S4, has been extended to S5 with similar conclusions, accounting for multi-channel handling. Additionally, Theorem 1, regarding the expressive power of SDT-P with LoRA on simplified SSM-based models, also holds true for S5. We provide the detailed extension for S5 here: https://anonymous.4open.science/r/ce7/l9.png.

• Mamba-II: Although extending our theoretical analysis to Mamba-II is non-trivial, we successfully adapt our method to this architecture, as detailed in Sec. C.2 and E.2, and evaluate it on diverse tasks, including GLUE, DART, SAMSum, and Spider. Our method consistently outperforms LoRA alone on Mamba-II, highlighting its generalizability beyond the architectures explicitly covered in our theoretical analysis. We will acknowledge this theoretical limitation explicitly in the conclusion section, emphasizing that extending our analysis to Mamba-II is an important direction for future work.

Final Note: Thank you for sharing your concerns. While we understand your perspective, we kindly remind you of the substantial contributions our paper makes and the efforts we've undertaken to address your comments. We would greatly appreciate it if you could reconsider your evaluation score and support the acceptance of our paper.

We are delighted that the reviewer likes the paper, recognizing it as (i) well-executed, (ii) a systematic study of PEFT for emerging neural architectures (Mamba, Mamba 2, and hybrid variants), and (iii) providing strong and comprehensive empirical results. Thank you for your encouragement!