Unlocking SVD-Space for Feedback Aligned Local Training

We appreciate the reviewer’s detailed feedback. We will address the queries below:

i) As SSA is essentially a low-rank approximation of DFA, I would expect a more extensive comparison with DFA, as the proposed method is built upon it.

Authors: We have added a detailed discussion of SSA versus DFA and its variants in updated Section 6.1 as present in Sanfiz et al. (2021), https://arxiv.org/pdf/2108.13446.

ii) Also, it would be expected that SSA at best matches DFA. On table 1 however it outperforms it by 13%. This is likely due to the alignment losses in SSA, and I believe that applying the same losses to DFA could provide a fairer baseline. It would also allow us to quantify the loss in accuracy caused by the low-rank factorization.

Authors: The custom loss operates on the SVD-components of the forward and feedback weights. We wouldn't be able to apply the same loss to normal weights. We begin our training on the full-rank of SVD-decomposed matrices itself, and then reduce the rank while training, applying different loss components to align the gradient direction and the weight-spaces between the feedback and forward signals. As we are training while reducing the rank, the loss due to rank-reduction gets adjusted in training.

iii) The method introduces 4 to 5 new hyperparameters, which currently have to be tuned. It would be interesting to measure the impact of these on the final accuracy.

Authors: We appreciate the reviewer’s concern about the impact of hyperparameters on the final accuracy. We have added a detailed explanation of our hyperparameter choices in Section 4 Experimental Setup. The loss-related hyperparameters (α,β,γ,δ,ϵ) were kept consistent across all experiments, with adjustments made only to the overall learning rate to account for differences between smaller datasets (e.g., CIFAR-10) and larger datasets (e.g., ImageNet).

As layerwise loss objectives are largely independent, tuning involves optimizing simpler, localized objectives. This allowed us to perform k=3-fold cross-validation on a small grid of candidate values, taking only 5–10% of the total training time. We verified the robustness of the hyperparameters under these conditions.

While this paper focuses on demonstrating the feasibility of SSA rather than exhaustive hyperparameter optimization, we acknowledge that further exploration of optimal tuning strategies is valuable and have noted this in Section 6.2 Limitations. We thank the reviewer for highlighting this area for potential future work.

iv) Eq. 6 is unclear to me. This quantity is the error (i.e. the gradient of the loss w.r.t the last hidden states), not the crossentropy loss. I find the notations very confusing. In addition, the layers do not receive the gradients of the cross-entropy loss but instead use DFA to approximate it, which is not reflected in equations 5 and 6.

Authors: We acknowledge the mistakes in the notations and have corrected them in the revised paper (Section 3.3). Additionally, we updated Fig. 1 to clarify the notations and illustrate the gradient input to the layers more effectively.

The local loss comprises cross-entropy loss, alignment loss, cosine similarity loss, a singular vector orthogonality regularizer, and the Hoyer regularizer. The gradient of this composite loss is provided as input to the layers, with the feedback error (from DFA) specifically corresponding to the gradient of the cross-entropy loss. These updates aim to improve the clarity and understanding of our method.

v) The theoretical analysis in the Appendix is wrong in many ways.

Authors: We appreciate the reviewer’s observation about the theoretical analysis. While the cosine similarity component or orthogonal regularizer component is not inherently Lipschitz smooth, we address this by applying constraints that ensure bounded gradients. Specifically, we normalize the norms of x and y to 1, effectively projecting the non-convex cosine similarity loss or orthogonal loss onto a convex sublevel set. This approach bounds the gradient differences, making the loss components quasi-convex and Lipschitz smooth under these conditions. These constraints are integrated into our custom loss function to maintain theoretical consistency. We will revise the part where we say loss will be always decreasing globally, in our theroretical analysis as pointed out by the reviewer.

I thank the authors for their answers, which addressed a few of my concerns. I also acknowledge that the updated paper has improved.

---

I still see a lot of issues in the theoretical analysis of Appendix A.1. In particular I see that errors that I mentioned in my previous message have not been addressed.

• The local losses are not independent: modifying the weights of layer $i$ will impact the input of a later layer $j$, as well as the error... Yes, if we were to optimize a single layer (with the other onesfrozen) then maybe it could converge, but if all layers are optimized at the same time there is no guarantee of convergence at all for any of the losses.
• The orthogonal regularization is NOT Lipschitz smooth
• There is no proof of convergence of DFA in the general case, so how can we get any theoretical insight of the convergence of the model after adding more losses?

The authors go through long and sketchy proofs to try to prove the convexity and Lipschitz smoothness of the local losses, which is still wrong in its current form. Moreover they use these results to assess the convergence of the global loss. This is equivalent to proving the convergence of an arbitrary neural network, which is not possible in the general case given that it is not convex not Lipschitz smooth.

I am starting to think that the proof has been written by an LLM.

---

The custom loss operates on the SVD-components of the forward and feedback weights. We wouldn't be able to apply the same loss to normal weights.

I beg to disagree, the cosine similarity loss does not rely on the SVD decomposition. Similarly, the alignment loss can very simply be adapted to the standard DFA (using $||W_i - B_i||^2$).

---

While I like the idea of the paper, the obvious errors I have seen in the proof and in the first version make the rest of the paper very unreliable. The poor quality of some answers to my questions make me wonder whether the authors know what they are doing. The paper is not in a state where it should be accepted, so I am downgrading my score.

That being said, the idea is indeed interesting and I encourage the authors to work on it and resubmit. I would advise dropping the proofs, which are not necessary to the rest of the paper, and focus more on the experimental part. I especially would like to see a fairer comparison with DFA, by either adding the alignment losses to DFA, or removing them from SSA. The current paper seems to try to do two things at once: i) making DFA more memory efficient using low-rank decomposition, and ii) improving DFA using new alignment losses. These are two very different contributions to the field, and should be made clearer -- it would also help selling the method better.

We thank the reviewer for taking the time to write a detailed feedback.

We answer the questions here:
i) Equation 6 issues. (Weaknesses pt 1 and 2):

We appreciate the feedback. We indeed made some mistakes in the notations, and have corrected in the revised paper (Section 3.3). We also updated Fig 1 for better understanding of the notations and gradient input to the layers.

The local loss is essentially a culmination of cross-entropy loss, alignment loss, cosine loss, singular vector orthogonality regularizer and hoyer regularizer. We input the derivative of this loss (gradient) to the layers, where the feedback error (DFA) is essentially the gradient of cross entropy loss.

ii) The composite loss comprises some terms that are not well motivated nor explained. While the provided ablation study is interesting, it clearly lacks details as for example one could ask if the cosine similarity loss indeed improves the cosine similarity. The behavior of each individual objective with respect to the others could also be interesting to study.

Authors: We have expanded on the role of individual loss components in Section 5.3 Ablation Study to provide greater clarity. The ablation results demonstrate that each loss term is critical for SSA's performance and efficiency. For example, Cross-Entropy Loss is essential for accuracy, while Cosine Similarity and Alignment Losses work together to preserve gradient direction in the absence of BP gradients. The Orthogonality Regularizer ensures meaningful angular transformations by maintaining the unitary properties of SVD components, and the Hoyer Regularizer sparsifies weights during rank reduction. While analyzing the interactions between loss components is beyond the current scope, we recognize its importance and have included it as future work.

iii) Some sentences sound like claims and are not motivated: eg l.254: what are the "stable and smooth updates" ensured by DFA? Is there a study of the "stability of the training process" (l.262)?

Authors: We have revised our paper accordingly to refine our motivations. We only show stability in our theoretical analysis and through convergence in our empirical results. Our loss components ensure that the gradient direction is maintained like BP (cosine-similarity), the unitary properties of SVD-components are maintained (orthogonality regularizer) so that any angular or linear transformation on those components will preserve the angles in the components. The alignment loss further tries to align the weight-space for better gradient direction updates. The feedback error essentially projects the global loss gradient into the layers to optimize global loss in a certain way. We account for the stability through the meaningful local loss optimization.

iv) The empirical results are somehow strange in many ways (Weaknesses pt 5)

Authors: We appreciate the reviewer’s observations and have addressed these points as follows:

SSA vs. DFA Comparison: We have added a detailed discussion of SSA versus DFA and its variants in Section 6.1. While DFA can be extended to deeper networks such as ResNet-56, it suffers from significant accuracy loss. This is what we intended by stating that DFA is not easily extensible to larger networks. Our aim was to highlight SSA’s ability to mitigate such issues through low-rank approximations and structured local updates.

PEPITA Results: The reported PEPITA results are sourced from the work by Apolinario et al. (2024): "LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronization" (arXiv:2405.15868). This study successfully trained PEPITA on networks deeper than those reported in the original paper by Dellaferrera and Kreiman (2022). We have cited this work in our revised manuscript for clarity.

ImageNet Results: The reviewer is correct in noting the inconsistency between Top-1 and Top-5 accuracy in Table 2. This was a typographical error, and the column titles were inadvertently swapped. We have corrected this error in the revised version of the paper.

v) The theoretical analysis provided in the appendix is unfortunately false as the composite loss function is not Lipschitz smooth as the cosine similarity component is not Lipschitz smooth. Linee 830-834 are false if x is very small.

Authors: We appreciate the reviewer’s observation regarding the theoretical analysis. While it is true that the cosine similarity component is not inherently Lipschitz smooth, we apply constraints to ensure bounded gradients. Specifically, we project the non-convex components, such as the cosine similarity loss, onto a convex sublevel set by normalizing the norms of x and y to 1. This ensures that the gradient differences are bounded, making the quasi-convex cosine similarity loss Lipschitz smooth under these constraints. We apply these constraints in our custom loss function as well.

We thank the reviewer for highlighting the incoherences in our work.

i) Regarding Table 1: The DFA results were mistakenly attributed to Akrout et al., 2019. This was a typographical error, and we sincerely apologize for the oversight.

ii) Regarding the Preprint: Upon contacting the authors of the preprint, they confirmed that PEPITA was not run on SmallConv, as reported. Instead, the results are based on the same architecture used in the original PEPITA paper.

iii) We have addressed our limitation of the hyperparameter search for the loss coefficients in the paper. We will further deep dive in the limitation in our future work.

We acknowledge these errors and will ensure they are corrected in the next iteration of our paper. Thank you for bringing these issues to our attention.

We thank the reviewer for highlighting the strengths of our paper. Below, we provide detailed answers to the reviewer's concerns.

i) A dynamic rank reduction strategy is used to reduce the rank of weight matrix progressively. How is it implemented and guranteed, especially considering that updates to SVD components are not inherently directed towards reducing rank.

Authors: We have updated the section 3.4 to include detailed explanation of how we reduce the rank of weight matrix progressively. Every ten epochs, we apply Hoyer regularizer as a part of our custom loss objective to increase the sparsity (ratio of L1 norm of singular values to L2 norm of singular values). This reduces rank. We also make threshold based checks (check if the rank of the weight matrix is below 95% of the matrix's energy) in the later part of the epochs, to stop aggressive reduction in rank, losing a part of the representational capacity. If we see that the rank reduction is not effective based on the threshold based check, we halt the epoch-based scheduling. In the initial epochs, we can reduce rank without worrying for reduction in representational capacity, as then the network can update itself for the change in the ranks in the next epochs.

ii) Beyond theoretical analysis, it would be beneficial to present empirical results that demonstrate the convergence rate and training stability of SSA in comparison to BP.

Authors: We provide error convergence graphs for a small DNN in Section 6.1 in the updated paper.

iii) The decomposition of a convolutional layer while retaining its hierarchical information is described in Appendix A.3, however it may be challenging for readers to grasp. Including a schematic diagram would enhance understanding by providing a visual aid.

Authors: We have put a detailed explanation of the convolutional layer decomposition in SVD-space in Section 3.1, as per the suggestions.

We hope these updates address the reviewer’s concerns and welcome any additional suggestions or clarifications.

We thank the reviewer for their detailed and valuable feedback, which led to huge improvements in the revised paper.

We address each of your comments below:

From Weaknesses and Questions:

i) Despite the main motivation of methodology/paper, only real-world compute/memory complexity results in paper (including appendix) are VGG-13/Figure 2. and What are the real-world compute/memory results for the other models in your work?

Authors: We have incorporated the compute and memory savings graphs for MobileNetV1 and ResNet-32, alongside VGG-13, as part of Figure 2 in Section 5.2, in our revised paper. We thank the reviewer for this insight.

ii) No variance/stddev in results.

Authors: We have updated Table 1 in Section 5.2 to include variance and standard deviation in the results in our revised paper.

iii) Although hard to evaluate due to the above, with the results as presented, there is not clear evidence that the results are significant. In many of the results AugLocal or SVD-BP appear to be very close or better than the results of SSA.

Authors: We appreciate the reviewer’s comment. BP and SVD-BP rely on finely tuned gradient-based optimization and a global loss signal, enabling strong layer-to-layer coordination. In contrast, local learning methods, driven by localized objectives, inherently lack such coordination. Achieving competitive results with BP using a local learning approach, as SSA does, is a significant milestone and remains an active area of research. Also, SVD-BP has to store gradients as BP and requires more memory and compute. AugLocal (layer-wise training) introduces additional overhead through auxiliary networks and classifiers (faring a bit better on deeper networks), SSA achieves competitive accuracy with significantly lower memory and compute requirements, making it a more efficient alternative.

iv) The work is not repeatable as presented, with a lack of experimental details, although there are some in appendix. and How exactly did you find your hyper parameters, and what are the hyper-parameters (and all other experimental details required to repeat the experiments?

Authors: We have made the following updates to address your concerns:

Hyperparameter Selection: We have added a detailed paragraph in Section 4: Experimental Setup to explain our hyperparameter choices. Specifically, we used a learning rate of $1 \times 10^{-4}$ for smaller datasets (e.g., CIFAR-10) and $5 \times 10^{-4}$ for larger datasets (e.g., ImageNet). The loss based hyperparameter values were kept consistent across experiments, with adjustments made only to the overall learning rate for layerwise updates. As layer objectives are largely independent, tuning coefficients involves optimizing simpler, localized objectives, allowing faster cross-validation with a narrower range of candidate values.

Rank Reduction Schedule: We expanded Section 3.4 to provide a clearer explanation of our progressive rank reduction strategy, highlighting how it balances memory and computational efficiency while maintaining accuracy.

Experimental Details for CIFAR-100 and ImageNet: Detailed training setups for CIFAR-100 and ImageNet, including preprocessing, hyperparameter settings, and hardware specifications, have been added to Appendix A.3 to ensure transparency and reproducibility.

v) How much do the five loss coefficients themselves have to be tuned to get good performance, i.e. how different are they for your different models/results? This is important as if they have to be found for each different experimental setup by sweeping training, it's hard to justify the method as speeding up training.

Authors: In Section 4 of our revised paper, we describe our hyperparameter selection process. Coefficients were kept consistent across all experiments, with only the overall learning rate adjusted for layerwise updates. Robustness of the coefficients was verified through a small k=3-fold cross-validation, taking 5–10% of training time. While layerwise tuning is simpler and faster than global loss tuning (due to simpler layerwise loss objectives), we acknowledge that this paper does not extensively explore optimal hyperparameter tuning, as we have mentioned in the paper in Section 6.2 limitations.

vi) Why did you choose to demonstrate improvements in training compute/memory usage not on a lightweight architecture, but instead VGG which is not used anymore in research/applications and widely recognized to be highly inefficient compared to e.g. ResNets, MobileNet, EfficientNet, etc.

Authors: There was no particular reason to choose VGG, we chose it due to the ease of visibility of the results in the layers.

Once again, thank you for your time and consideration, and for giving us such a detailed review with the opportunity to improve our paper.

I thank the reviewer for their concerns. We would address the concerns in our future work or the next iterations.

i) ImageNet/ResNet32 - ResNet 32 is an architecture built for CIFAR-10, not ImageNet. Not clear to me at all how the architecture is being changed to run with ImageNet/how this is a good model to be used with ImageNet.

Authors: We appreciate the reviewer’s observation regarding the use of ResNet-32 for ImageNet. We acknowledge that ResNet-32 is traditionally designed for CIFAR-10 and not directly optimized for ImageNet. In our experiments, we adapted ResNet-32 for ImageNet by increasing the input resolution, modifying the stride and pooling layers, and adjusting the number of filters to handle the larger image sizes and dataset complexity. As our model can't yet extend to ResNet-50 (due to linear separability as we had mentioned in limitations) , we had adapted ResNet-32 for ImageNet. Local training methods usually suffer for deeper network layers (beyond 10 layers). We are keeping the extension of our model to deeper networks for our future work.

For the other improvements suggested, we will include those improvements in our paper in the next iteration.

I thank the reviewer again for contributing to the improvisation of our paper.

We thank the reviewer for their time and consideration for reading our paper.

We answer your queries here:

i) The paper focuses on reducing memory and computational cost during training. However, to deploy models in resource-constrained environments, efficiency during post-training inferences is more important.

Authors: We have revised Section 3.5 to demonstrate both training and inference complexity. The computational complexity of inference is reduced to: $O(r \times n) + O(r \times r) + O(m \times r)$ where $r \ll \min(m, n)$. This decomposition not only reduces compute cost but also minimizes memory requirements, as only the SVD components ($U$, $S$, $V^T$) need to be stored: $O(m \times r) + O(r \times r) + O(r \times n).$ By dynamically reducing the rank $r$ during training, SSA further optimizes both compute and memory usage in the inference phase, making it highly suitable for deployment in resource-constrained environments such as edge devices.

ii) It's better to provide comparison between the proposed method and model compression techniques like quantization-aware training (QAT).

Authors: We provide the following tables highlighting the differences between SSA and QAT, and then compression percentages for QAT and SSA methods.

We provide the comparison of comparison methods obn VGG-11 reflecting inference-time memory savings.

Ref: Babak Rokh, Ali Azarpeyvand, and Alireza Khanteymoori. 2023. A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification. ACM Trans. Intell. Syst. Technol. 14, 6, Article 97 (December 2023), 50 pages. https://doi.org/10.1145/3623402

iii) The proposed method has many limitations such as hyperparameter sensitivity and inability to scale to larger models. Also, no evidence is provided to show the effectiveness of SSA on transformer-based models.

We have addressed hyperparameter settings in Section 4, Hyperparameter Selection. We state the hyperparameters used in that section based on the constraints of our theroretical analysis and verifying them in k=3-fold cross-validation. While our paper does not focus on exhaustive hyperparameter tuning, we acknowledge this as a potential limitation and have discussed it in the Limitations section. Future work may explore more sophisticated methods for tuning these parameters.

We can scale to ResNet-32 and convolutional layers, which is not the case for most feedback alignment-based local learning methods. Regarding transformer-based models, the scope of this work was primarily to validate a novel local learning rule in the SVD-space of DNN layers. While our experiments focus on CNNs, the design of SSA, particularly its operation in the SVD-space and optimization of local loss objectives, is architecture-agnostic. Future research will extend SSA to deeper networks and to transformer-based models, leveraging their modular self-attention mechanisms and natural compatibility with our rank-reduction strategy.

We thank the reviewer for prompting us to consider the efficiency of post-training inferences more deeply and to further explore the compression aspect of our lightweight model.