SAN: Hypothesizing Long-Term Synaptic Development and Neural Engram Mechanism in Scalable Model's Parameter-Efficient Fine-Tuning

Response to Reviewer 1yzx

Thank you for your thorough review and feedback. Your positive comments (such as acknowledging our clearly explained SAN and its connection to LTP/LTD) and suggestions have greatly helped us improve our paper. To address your concerns, we have carefully read your feedback and provided point-by-point responses below:

1. Minor typos

   a. missing period at the end of the abstract

   b. “(see Figure 3)” should be in front of the period on page 4 under “Expressiveness & self-regulation”.

2. Provide more discussion on the limitations of the linear transformation approximation and potential failure modes.

3. By explicitly propagating the scaling vectors, would it introduce other implicit regularizations besides the self-regularization mentioned, such as effects associated with reducing depth?

4. Provide a complexity analysis and more hyperparameter details.

Point-by-Point Response:

1. Regarding Typos

Thank you for your meticulous review. We will correct these in the revised version.

2. Regarding potential failure modes

We envision SAN as a plug-and-play method adaptable to current and future SOTA PEFT approaches across domains. This implies that SAN's failure mode mainly depends on the selected base method.

To stress-test pure linear transformations in large models, we evaluated SSF and SSF+SAN on LLaMA fine-tuning. The results (see Table below) indicate that for large models (7B+ parameters), pure linear transformations of features cannot effectively align pre-trained parameters to downstream tasks. This is reasonable, as although such methods achieve excellent results on vision foundation models (which typically have only hundreds of millions of parameters) by providing orthogonal fine-tuning-like [1] properties that preserve hypersphere energy and prevent catastrophic forgetting, their expressive power becomes constrained in larger language/multimodality models.

3. Regarding other implicit regularizations

Self-regularization emerges directly from the implicit $\gamma^2$ term during adjustment, which guides scaling factors to reduce extreme values and control variance. Besides, we haven't observed significant regularization effects beyond self-regularization.

To our knowledge, the depth-reducing phenomenon you mentioned appears in two types of literature: structural pruning (particularly layer pruning) like Layer Folding [2], which aim to improve model efficiency; and layer redundancy studies like stochastic depth [3], which prevent overfitting through random depth dropout during training. Our method doesn't directly relate to these works.

4. Regarding complexity and hyperparameters

We have provided basic hyperparameters in the paper, and more detailed parameter settings can be found in our supplementary code. As mentioned in Appendix B, we've recorded thousands of experimental setups and results on WandB, which we will make public.

For algorithm complexity, SAN, as a plug-and-play method, adds no additional parameters during training. The computational cost only appears in the decompose & propagate operations. Specifically:

• The decomposition cost doesn't involve matrix multiplication (using average pooling or direct reuse), making the additional computation negligible.
• For propagation, we multiply the decomposed $1×d$ vector with a $d×k$ posterior weight matrix. The complexity for a single propagation is $O(d×k)$, where $d$ is the input feature dimension and $k$ is the output dimension. If we have multiple layer groups $(1,2,...N)$ that need propagation, the total computational complexity would be the sum of each group's complexity: $O(∑(d_i×k_i)$ for $i$ from $1$ to $N$.

This minimal computational overhead makes SAN an efficient enhancement to existing PEFT methods. Practically speaking, throughout our experiment, we also did not see a significant slowdown by adding SAN.

Reference

[1] Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., ... & Schölkopf, B. (2023). Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36, 79320-79362.

[2] Dror, A. B., Zehngut, N., Raviv, A., Artyomov, E., Vitek, R., & Jevnisek, R. (2021). Layer folding: Neural network depth reduction using activation linearization. arXiv preprint arXiv:2106.09309.

[3] Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016). Deep networks with stochastic depth. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 (pp. 646-661). Springer International Publishing.

Response to Reviewer bAk4

Thank you for your thorough review and feedback. Your positive comments (such as acknowledging our solid experiment and idea) and suggestions have greatly helped us improve our paper. To address your concerns, we have carefully read your feedback and provided point-by-point responses below:

1. Challenging to read / parse / understand partly due to the abundance of acronyms and partly due to poor mathematical notation and some missing explanations:

   a. Minor Typos and overwhelming acronyms

   b. The $W_{down}$ in Equation 1 should be $\mathbb{R}^{r\times d}$ and $W_{up}$ should be $\mathbb{R}^{d\times r}$. What do $x,y$ represent?

   c. $W$ in Equation 2 should be $\mathbb{R}^{d\times d}$ and $γ$ is a vector of size $\mathbb{R}^{1\times d}$. Do the authors mean $(γ1^⊤)⊙W$, where 1 is the vector of all ones? In Equation 9, the $T(\cdot)$ function is overloaded.

   d. Why does the update stabilize or become implicitly regularized to be near 1 in Equation 12? Under what conditions do the assumptions in line 184 expect to hold?

2. Is there biological motivation for equation 6? Is it computed per example??

3. In the baselines, are all fine-tuning methods employing equal numbers of parameters or similar compute?

Point-by-Point Response:

1. Regarding mathematical notation and explanations

   a: Thank you for your meticulous review. We will correct these points in the revised version.

   b: In Equation 1, we denote $r\ll d$, where $r$ is the rank of the LoRA/Adapter. Accordingly, $W_{down}$ is defined as $\mathbb{R}^{d\times r}$ to compress the input $x$ from dimension $d$ to $r$, and $W_{up}$ is defined as $\mathbb{R}^{r\times d}$ to recover the original dimension. As noted in [Line 143-144], $x$, $y$, and $\theta$ represent the inputs, outputs, and the linear/non-linear function of the LoRA/Adapter, respectively.

   c: $W$ can also be shaped as $\mathbb{R}^{d\times k}$ (e.g., the FFN in transformers), as long as the scaling vector $γ$ of size $\mathbb{R}^{1\times d}$ is available for per-row scaling. We did not locate the expression $(γ1^⊤)⊙W$ mentioned in your comment. We assume your concern may relate to whether $γ$ is an all-one vector. Initially, $γ$ is indeed an all-one vector, but after training, it diverges (i.e., $γ$ is trainable or generated by trainable modules). For the function $T(\cdot)$, defined as $T(y_{l})=\gamma_{l}\odot y_{l}$, this represents element-wise scaling of $y_{l}$ prior to feeding it into layer $l+1$ layer. Specifically, we have: $y_{l+1}=W_{l+1}T(y_{l})+b_{l+1}=W_{l+1}\gamma_{l}\odot y_{l}+b_{l+1}=\gamma_{l}\odot W_{l+1}y_{l}+b_{l+1}=T(W_{l+1})y_{l}+b_{l+1}$, which is automatically managed by PyTorch’s broadcast mechanism.

   d: The implicit regularization effect arises from the explicit propagation, which introduces a quadratic effect ($(\gamma_l)^2$), which discourages extreme values and thus stabilizes the updates. This stabilization prevents overfitting by controlling the magnitude of divergence from the initial value 1. In a linear scenario, scaling the output is equivalent to scaling the weights in the next layer. Although ReLUs are not globally linear, they operate nearly linearly in their active regions (when the output is non-zero).

2. Regarding biological motivation for Equation 6

Equation 6 reduces both the batch and token dimensions (related to the size of data) while preserving the embedding dimension (related to the content of data), resulting in a 1D vector that can be used for scaling weights. It aligns with the principles of synaptic plasticity. Network signals are data-driven, and synaptic development adapts to these signals. For instance, high-frequency stimulation strengthens synaptic connections between neurons, and such strengthened connections maintain over a range of time, forming a specialized engram. Furthermore, while methods like SSF and DoRA apply scaling factors at the neuron level (using the same scaling for all parameters in each row of the weight), SAN operates at the synapse level. By introducing a propagation mechanism, we use the previous layer’s scaling as additional per-column scaling for the current layer. Resulting in more complex engram patterns, wherein each synapse within a neuron can have its scaling factor (see Figure 3). This design better aligns with recent neuroscience findings on neuronal engrams, such as the linking, buffer, and feature neuron engrams discussed by Choucry et al. (2024) [1].

3. Regarding trainable parameters of baselines

SAN itself doesn't introduce additional trainable parameters; our experimental parameter settings follow those established in SSF, SPT-LoRA, and DoRA, which are all published papers. We have also provided the parameter ratio in all tables.

References

[1] Choucry, A., Nomoto, M., & Inokuchi, K. (2024). Engram mechanisms of memory linking and identity. Nature Reviews Neuroscience, 25(6), 375-392.

Response to Reviewer WoLR

Thank you for your careful review and feedback. Your positive comments (such as praising our "sound evaluation criteria" and "easy to follow writing") have greatly support our paper. We have thoroughly reviewed your comments and address your key questions below:

1. For SAN+LoRA is the scaling factor propagated to the next layer that uses LoRA or just the immediate next layer of the model?

2. The claim regarding LTP/LTD seems a bit stretched.

Point-by-Point Response:

1. Regarding SAN+LoRA propagation mechanism

In our experiments, we followed the default LoRA placement settings used in DoRA [1] and SSF [2]:

• For LLaMA/LLaVA models: LoRA was added to ["q_proj", "k_proj", "v_proj", "up_proj", "down_proj"]

• For vision models: LoRA was added to all Linear layers (except the head)

This naturally led to scaling factor propagation to the immediate next layer. However, your question raises an interesting point: when LoRA modules are sparsely added (e.g., every N block), is it better to propagate the scaling factor to:

• The immediate next layer (even without LoRA)

• The next layer that uses LoRA (potentially skipping several blocks)

Due to time constraints, we conducted a simple test on ViT-CIFAR100 using only two LoRA modules (added to the qkv_attn layers in block 0 and block 11). The results are shown below:

We found that when LoRA modules are distantly placed, long-distance propagation does not yield superior results. This aligns with our findings in Section 4.5 where long-distance propagation occurs when randomly apply. Conversely, propagating to related adjacent layers shows benefits, even when those layers don't have LoRA modules. We hypothesize that this shifts the LoRA module's learning objective from modeling only the current layer's parameter updates to also accounting for subsequent layers' updates, which enhances LoRA's learning capacity.

2. Connection to LTP/LTD mechanisms

Our method's connection to Long-Term Potentiation/Depression (LTP/D) is straightforward: we explicitly introduce the propagation mechanism. Existing PEFT methods like LoRA, DoRA, and SSF (as shown in Figure 3) typically only model the current layer's updates. Even though DoRA and SSF decompose scaling factors, they don't leverage propagation.

In neuroscience, synaptic development is directly modulated by neuronal activation patterns. LTP and LTD exemplify this: high-frequency activation of presynaptic neurons induces LTP, strengthening the synaptic connection, while low-frequency stimulation induces LTD, weakening it [3] [4]. This aligns with Hebbian learning—the principle that “what fires together, wires together.”

In our SAN approach, the explicit propagation of scaling factors mimics this biological process. Rather than treating each layer’s update in isolation, SAN transfers the scaling factor learned from one layer to the next. This is analogous to how the potentiation (or depression) of a synapse in a neural circuit influences the subsequent neurons, ensuring that plastic changes are distributed in a coordinated manner. For example, if a layer’s output is amplified (akin to LTP), the following layer would receives this amplified signal as a prior guidance and adjusts its weights accordingly. Moreover, by propagating scaling factors, our method effectively connects trainable modules throughout the model:

• Locally: we shift the learning focus from modeling only single-layer updates to considering groups of related layers.

• Globally: All trainable modules throughout the model become "interlocked," simplifying the adjustment range each module needs to cover (Figure 2) and achieving better performance

This biologically inspired mechanism, therefore, not only provides a theoretical foundation for our approach but also delivers practical benefits in model transfer learning.

Reference

[1] Liu, S. Y., Wang, C. Y., Yin, H., Molchanov, P., Wang, Y. C. F., Cheng, K. T., & Chen, M. H. (2024, July). Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning.

[2] Lian, D., Zhou, D., Feng, J., & Wang, X. (2022). Scaling & shifting your features: A new baseline for efficient model tuning**.** Advances in Neural Information Processing Systems, 35, 109-123.

[3] Malenka, R. C., & Bear, M. F. (2004). LTP and LTD : An embarrassment of riches. Neuron**, 44(1), 5–21.**

[4] Tonegawa, S., Liu, X., Ramirez, S., & Redondo, R. (2015). Memory engram cells have come of age. Neuron, 87(5), 918–931.