Pretraining with Random Noise for Fast and Robust Learning without Weight Transport

One is that the authors do not train until convergence, neither for the random pre-training nor for the task related data.

: Please find our response in Global Response, issue 1

We acknowledge the suggestion that comparisons should be made after ensuring full convergence during network training.

• We conducted additional experiments on various datasets beyond MNIST. In these experiments, we ensured that sufficient convergence was achieved before concluding training (Global Response, issue 1).
• Specifically, we ensured training accuracy reached saturation close to 1, or that validation accuracy no longer increased for 10 epochs, indicating the completion of learning.
• We will update all network training and analyses performed in the manuscript according to these criteria. Importantly, even after ensuring convergence, all trends and qualitative differences remained consistent.

It could be that having a too-low rank during pre-training is bad for learning later on

We found that even with extensive random noise pre-training, the rank reduction was not dramatic enough to impede the learning of rich representations.

• After random noise training, the effective rank decreased only slightly in most cases (Supple. C.3). This introduces a bias towards learning low-rank solutions in subsequent training but does not significantly hinder representation learning.
• Depending on the network structure (width and depth), certain layers experienced a decrease in effective rank during random noise pre-training, which was subsequently recovered during data training. This suggests that while initial rank may affect information representation, it can be compensated for in later stages of learning.

In additional experiments, we proposed an effective pre-training length, also addressing concerns about excessively long pre-training periods (Please find our response in Global Response, issue 2).

It could also be that at the end of training a network trained only on data performs better at the cost of more computation

We quantitatively compared accuracy after achieving sufficient convergence and confirmed that random noise training generally results in higher accuracy regardless of network depth or dataset type (Response Figs 1, 2, 3; Response Table 1).

I haven't seen the performances in numbers.

We provided numerical results for additional experiments (Response Table 1). We will include these numerical results in a separate appendix in the final manuscript.

it's still a good idea to pre-train for faster learning, even if that could potentially lead to slightly lower performances later on.

We appreciate your comments emphasizing the benefits of pre-training for faster learning, especially in biological systems. Our results indicate that our method achieves both faster learning and higher performance. Thus, our approach demonstrates that learning speed and accuracy need not be mutually exclusive but can be concurrently enhanced through random noise pre-training.

Minor issues (description for loss reduction, definition of alignment)

We will clarify that the loss refers to "training loss" for random noise.

We acknowledge that our current definition of “alignment” may not hold in some cases. We will revise the definition as follows: Given the forward weights W and backward weights B, we measured alignment using cosine similarity. We define W and B as aligned if \angle{W^T, B} is significantly smaller than 90 degrees.

A few related works:

We will discuss the importance of weight symmetry for high-performance deep learning and also mention that there are learning architectures that do not require weight symmetry. Relevant papers will be cited in this discussion.

I think it should be possible to show that the pretraining would reduce the effective rank.

We appreciate your theoretical insights, which bolster our empirical findings.

• Your detailed suggestions for theoretical proofs are insightful, particularly in understanding how random noise training reduces effective rank, especially in scenarios with limited readout capabilities.
• During this rebuttal period, we focused on conducting additional experiments to validate our results in broader contexts. Due to time constraints, we could not fully develop and present the theoretical proofs, despite your helpful guidance.
• We understand that including these proofs would significantly strengthen our findings, and we aim to incorporate them into the final manuscript.

I don't get what is the task agnostic section supposed to show.

We used "task-agnostic" to indicate that random noise training provides benefits in subsequent learning across various datasets. However, we acknowledge that this term might be unclear. We agree that "generally useful for various tasks" better conveys our intended meaning, as you suggested. We will replace the section title and descriptions accordingly in the manuscript.

I don't get what the fact that the architecture that they mention does not use weight transport implies that more experimental studies are needed.

We appreciate your perspective and detailed explanation regarding the biological implausibility of weight transport.

• Describing specific biological circuits, as you suggested, would provide more compelling evidence than simply stating that weight transport is biologically implausible. We will include this in Section 2, Preliminaries, where we discuss the weight transport issue and mention that in the brain, apical (top-down) and basal (bottom-up) projections are anatomically distinct.
• We will remove this description from the limitations section.

We extend our gratitude to Reviewer o5xh for recognizing the significance of our research. Your thoughtful review has encouraged us to conduct additional experiments during the rebuttal period, further validating the importance of our findings.

I am unclear as to what the Definition in lines 147-149 is trying to convey.

We acknowledge that the definition in Lines 147-149 regarding alignment between forward and backward weights is unclear.

• \mathbb{E} represents the expectation over the parallels between W and B across independent neurons.
• \angle refers to the angle between W and B, not each of W and B.

We will revise the definition of weight alignment as follows: Given the forward weights W and backward weights B, we used cosine similarity as a measure of alignment for individual neurons. We assert that W and B are aligned if \angle{W^T, B} is significantly smaller than 90 degrees.

However, the label of the X-axis says “No. of training inputs.”

We understand that the term "No. of training inputs" is not commonly used in machine learning and can be misleading. In the final version of the manuscript, we will correct it to "epoch."

• Initially, we used "No. of training inputs" due to the ambiguous definition of an epoch in random noise training.
• Random noise training samples input x and label y in a random distribution. This is why we did not use "epoch" on the x-axis, which typically refers to traversing the entire dataset once.
• To clarify, we will label the x-axis as "epoch" and explain in the text that in random noise training, samples in each epoch are newly sampled.

Referring to models without pre-training as “untrained” can be misleading.

In our paper, we distinguish four network states:

1. Untrained (no training performed)
2. Only random noise trained
3. Data trained with an untrained network
4. Data trained with a random noise trained network

• Most results cover states (3) and (4). In Figure 3a, 3b, and 3c, "untrained network" refers to a randomly initialized, untrained network (1), and "random trained" refers to a network trained only with random noise (2).
• We will organize the terms and colors to clearly distinguish these four network states in the final version of the manuscript.

Appendix sections C.4 and C.5 ...

As suggested, we will include results with backpropagation in Appendix C.4 and C.5.

• We already have these results and have confirmed that the network with random noise pretraining learns much closer to backpropagation than to baseline feedback alignment.
• This strengthens our argument that random noise training improves feedback alignment learning to levels comparable to backpropagation.

How much would considering the extent of pre-training close the gap in training speed between these two strategies?

: Please find our response in Global Response, Issue 2

As you pointed out, comparison of convergence speed in Figure 2b does not account for the "time" spent on random noise training and only considers subsequent data training. We agree that it is important to investigate whether random noise pretraining ensures fast convergence even when considering the time spent on pretraining. This factor can further emphasize the significance of our research.

• We conducted additional analyses and measured the time to converge in real data learning after random noise pretraining with various lengths (Response Fig 4).
• Our analyses showed that networks with random noise pretraining can learn data quickly, even when including the time spent on pretraining in most conditions. Additionally, this analysis allowed us to estimate the optimal pretraining length for resource-efficient learning.

What is the overall contribution this paper wishes to make?

Our goal is to improve current strategies in deep learning by drawing insights from the brain.

• There exists a significant accuracy gap between backpropagation and biologically plausible learning strategies without weight transport. While backpropagation is effective, it is computationally intensive, requiring dynamic memory access due to weight transport (access W in memory to compute backward updates).
• As pointed out, the results of pretraining with random noise and aligning forward-backward weights resemble traditional backpropagation properties, but our approach achieves this without weight transport.
• Therefore, our interest lies not in achieving comparable performance to backpropagation but in achieving it without weight transport. Hence, our results provide insights into achieving learning efficiency levels comparable to backpropagation without using weight transport.
• It is noteworthy that feedback alignment (and random noise pretraining) does not require weight transport and relies solely on local information. In the context of energy-efficient neuromorphic chip engineering, feedback alignment is sometimes used for learning, albeit with some performance trade-offs. Our results demonstrate that such sacrifices are minimal in our model.

We briefly discussed these arguments in the broader impact section, but we realize that our intentions were not explicitly stated. This will be clearly articulated in the final manuscript's discussion.

On the other hand, we also discovered that pretraining with random noise benefits learning with backpropagation, although we did not include these results as they are not the main focus of the study.

• For instance, results related to weight alignment (Figure 1) are specific to the feedback alignment algorithm. However, findings related to the low-rank bias of forward weights and its resulting better generalization (Figure 3, 4) are also observed in backpropagation.
• Please note that we have identified additional benefits of random noise pretraining in backpropagation and are preparing this as part of a follow-up study.

We sincerely appreciate Reviewer 5zAn's careful consideration in improving the organization and clarity of our manuscript. We particularly value the opportunity to articulate how our research aims to contribute to the field and to present additional results that highlight the strengths of our work. We hope that our rebuttal and additional experimental findings will be convincing.

The bulk of the experiments only studied one-hidden neural network trained on MNIST

: Please find our response in Global Response, Issue 1

We expanded our experiments by increasing the depth of the network and confirmed that even with deeper architectures, the accuracy of the randomly noise-pretrained network consistently surpasses that of the baseline FA (Response Figs 1, 3 and Response Table 1).

Furthermore, beyond MNIST, we assessed the impact of random noise training on fMNIST, CIFAR10, CIFAR100, and STL10, consistently achieving higher accuracy (see Response Figs 2, 3 and Response Table 1).

Please tune baselines.

We are confident that our results were not obtained from different baselines as you expressed concerns.

• Please note that our results are based on hyperparameters typically chosen for effective learning in both FA and BP. We conducted and compared baseline FA (FA w/o), FA with random noise pretraining (FA w/), and BP under carefully controlled conditions.
• We controlled hyperparameters such as batch size and learning rate, as well as initial forward weights, to isolate differences attributable to different learning algorithms. Specifically, in a single trial, we duplicated one randomly initialized network into three, starting with identical weights. These networks were then trained using the same hyperparameters but different learning methods and algorithms (FA w/o, FA w/, BP).
• Additionally, both baseline feedback alignment and backpropagation showed significant performance in benchmark tests. We have presented these results numerically (Response Table 1), which we believe further supports that our findings were obtained under appropriate training conditions.
• Regarding weight initialization, we primarily utilized He initialization since we employed ReLU as the non-linear function in the network. As backpropagation utilizes W for backward computation, we ensured that B in feedback alignment followed the same statistical characteristics as W.

Through further experiments, we demonstrated that varying hyperparameters (batch size, learning rate) and initialization within a reasonable range consistently reproduced our results (Response Figure 5). Due to the 1-page PDF limit for the global response, we were unable to include it, but we confirmed that our results are reproducible when using tanh as the non-linear function.

Please study the following crucial baseline: Initialize the feedback weights B with W transpose.

As you suggested, we initialized B to match the transpose of W and conducted learning (Response Fig 6). This approach resulted in gradients and alignment fairly similar to backpropagation during training. Thus, we confirmed that this setup does not hinder learning and instead demonstrates efficiency comparable to backpropagation.

• During the learning process (especially at the beginning), weight alignment may slightly loosen but remains mostly valid.
• This baseline example illustrates that achieving initially aligned forward and backward weights allows for learning comparable to backpropagation, even without enforced synchronization of backward weights during training.
• It's a specific instance highlighting the validity of random noise training in aligning forward and backward weights in a biologically plausible manner before data-driven learning.

How can you overcome the limitations of depth?

Our additional experiments confirmed that random noise pretraining scales effectively to deeper networks and more challenging tasks, despite the limitations of baseline feedback alignment.

Indeed, previous studies on biologically plausible backpropagation without weight transport often focus on simple network structures and easy datasets [Lillicrap et al., Nat. Comm., 2016; Dellaferrera and Kreiman, ICML, 2022; Toosi and Issa, NeurIPS, 2023], and this limitation results in lower learning capacity and presents challenges in scaling up due to biological constraints that preclude weight transport.

Through our additional experiments, we have shown that random noise pretraining consistently outperforms baseline feedback alignment, even in deeper networks. However, we also observed that the performance gap with backpropagation widens as the number of layers increases.

• This challenge partly arises from the difficulty of achieving precise weight alignment in early layers using feedback alignment, without weight transport.
• The concept of "layerwise local loss" may provide ideas to address this issue, potentially enabling us to scale our results more effectively and achieve performance comparable to backpropagation in very deep networks. Specifically, this approach involves optimizing consecutive 2-layer blocks independently within a deep network. We are actively exploring several strategies to narrow the performance gap between feedback alignment and backpropagation using this approach, which we are preparing as part of our follow-up project.

We sincerely appreciate Reviewer kKCk's insightful comments. We acknowledge that in our initial work, we may not have sufficiently demonstrated the reproducibility and generalizability of our results, potentially leading to concerns about overestimation. Thanks to your feedback, we have conducted extensive additional experiments, which we believe have strengthened our arguments. We are grateful for your constructive criticism and we hope that our additional findings are convincing.