DeeperForward: Enhanced Forward-Forward Training for Deeper and Better Performance

Thank you for your valuable and detailed review. The latest revised version has been modified and uploaded based on the reviewers' comments.

Response to concerns in weakness:

1. Thank you for your suggestions and corrections regarding the writing of our paper. We have incorporated the necessary changes in the revised version as follows: (1) a brief description of model parallel strategy in abstract. (2) The format in Section 3.2 has been adjusted to ensure consistency.

2. Figure 1 in our paper primarily highlights the distinction between our approach and other bio-plausible training methods. In bio-plausible training research, the differences in training frameworks are a key characteristic, not just the performance disparities. To provide a more comprehensive comparison, we include comparisons with other bio-plausible methods in Figure 1, not just FF methods. Although we use a single forward pass, we still adhere to the FF training framework, and the use of a single forward pass is more efficient.

3. The decoupling of goodness: The FF method uses goodness as a classification metric at each layer, as seen in the original FF, which uses the sum of squared outputs as goodness. If the output is directly used as the input for the next layer, it retains the classification information from the previous layer. For instance, in the case of a positive class, the input elements are relatively large, and for other classes, they are smaller. This allows the next layer to make new predictions based on the predictions carried over from the previous layer (e.g. an identity transformation), without the need for further image feature learning. Therefore, FF proposes the decoupling of goodness and uses L2 norm to normalize the input goodness information to a consistent value of 1. The issue in the CwComp paper is that they did not decouple goodness, allowing subsequent layers to rely on classification information from the previous layers, which leads to overfitting in deep networks.

   While L2 norm can decouple goodness, it cannot control the input within a reasonable range of mean and variance, which introduces Feature Scaling issues and lowers performance. To address this, we propose using Layer Normalization and Mean Goodness, which simultaneously solve the problems of decoupling goodness and feature scaling.

4. As explained in the Introduction, we use Mean Goodness and LayerNorm as alternatives to address the challenges in FF design, specifically the issues of feature scaling and deactivated neurons. The square goodness focuses more on the few outlier outputs, which leads to many outputs being deactivated, resulting in lack of features for the subsequent layers. Our mean goodness provides a more balanced focus on all elements, ensuring that more neurons are activated. Furthermore, as mentioned in the Method section, in the case of square goodness, the weights corresponding to the deactivated neurons are not updated. However, with mean goodness, the weights of all neurons can still be updated.

   To provide a more comprehensive explanation of this motivation, we have added Appendix I to compare the deactivated neurons in both cases and to confirm that our method results in more activated neurons.

5. As mentioned in the Conclusion under Limitations, tasks with large-scale datasets like ImageNet, which involve 1000 classes, remain a challenge for our method. This is because the number of channels per layer must be a multiple of the number of classes, and this multiplier is preferably greater than 10 to extract sufficient information. This requirement leads to impractical memory and time consumption for real-world applications. We experimented with a ResNet17 architecture where the channel numbers for the 4 stages were modified to [1000, 1000, 2000, 2000] and trained for 20 epochs. However, the Top-5 accuracy was below 10%, due to the inadequacy of channels.

   Currently, pure FF-based methods are predominantly evaluated on benchmarks like CIFAR10, MNIST, and FMNIST to validate feasibility, making our comparisons fair and reasonable. Moreover, our method significantly improves the performance of FF-based approaches on these benchmarks. Nonetheless, applying FF-based methods to larger datasets remains an important direction for future exploration.

6. The model parallel strategy is primarily compared with BP's data parallel method to evaluate our parallel performance. Other FF-based methods are typically limited to small networks (within 4 layers) and rely on progressive learning, training layer by layer rather than the entire network simultaneously. These methods lack parallel acceleration strategies and are therefore not directly comparable. As a result, we focus on comparing our approach with the commonly used DDP method on BP.

Response to concerns in question:

1. ( We have added experiments on deactivated neurons in Appendix I to make the paper more convincing. ) The term "model size" refers to the fact that prior FF methods have only been implemented in shallow networks. The issues of bio-plausibility in BP and model size in FF are distinct and address different concerns.

• First, bio-plausibility is a challenge associated with backpropagation, as described in lines 27–37 of the introduction. These challenges include weight transport, non-local updates, freezing activity, and update locking. Our method addresses these issues by leveraging the characteristics of the FF framework. Specifically, by setting independent objectives for each layer, we eliminate the need for error backpropagation, thereby resolving bio-plausibility concerns.
• Second, the limitations of FF concerning model size refer to the observation that current FF-based methods are only effective in shallow networks. Deeper networks fail to improve performance or suffer from overfitting. In the introduction, we identified two main causes for this limitation: Feature Scaling and Deactivated Neurons.
  • Feature Scaling: The original FF approach relies on L2 normalization, which cannot adequately control feature values within a reasonable mean and variance range. This results in large feature discrepancies, leading to overfitting. In this work, we address this issue by designing the network with Layer Normalization.
  • Deactivated Neurons: The original FF method uses square goodness, which disproportionately emphasizes large outliers, resulting in a significant number of deactivated neurons. We resolve this problem by introducing mean goodness, which provides a more balanced approach. In the new vision of the paper, we have added the experiment on deactivated neurons on Appendix I.

These improvements allow our method to effectively scale to deeper networks while addressing the bio-plausibility challenges inherent to backpropagation.

2. ( We have rewritten the SIP part to make it clearer. ) In other FF-based methods, it is common practice to simply sum the goodness values of the last one or several layers as the final output. Our SIP strategy, however, seeks the optimal combination of layers whose cumulative goodness achieves the best results. Specifically, we evaluate various combinations by calculating accuracy on untrained data and selecting the best-performing combination as our final choice. Given L layers, there are 2^L possible combinations, making an exhaustive search computationally prohibitive. To reduce complexity, we restrict the combinations to only contiguous layers—summing the goodness values from a specified start layer to an end layer—thus reducing the search space to (L+1)L/2 combinations.

3. Xr serves as the input to the shortcut in the residual structure and is also the output from a preceding layer. Figure 3 illustrates the structure of a residual connection, where the output features are fused with the output Xr from a previous layer to generate new features. Xr can be the output of any preceding layer, depending on the network design. As stated in the paper, we adopted the residual connection strategy based on the ResNet18 architecture. The detailed design of the entire network model is provided in Appendix A.

4. (The results of HPFF has been added in Table1.) Thank you for providing the two papers: Loco and HPFF. Both papers incorporate auxiliary networks in the layers to perform block-wise local backpropagation training. Loco is based on contrastive learning, while HPFF uses supervised learning. Both methods still rely on backpropagation's powerful learning capabilities, but they face issues related to bio-plausibility, such as the fact that neurons do not perform both forward and backward transmissions simultaneously, as in the brain, but rather in a one-directional manner.

   The Forward-Forward (FF) method, which is brain-inspired, aims to replace backpropagation by exploring a training approach that more closely aligns with the mechanisms of the brain, specifically by updating weights during forward computation. In our method, we compute the weight updates simultaneously with the output at each layer, following the FF approach without relying on auxiliary networks. While the performance still lags behind BP-based methods to some extent, our approach significantly enhances the capability of FF-based methods, bringing us closer to a more brain-like training process.

Thank you for the insightful recommendation!

5. ( We have modified Appendix C as your suggestion. ) Thanks for your suggestion. We have revised our experimental setup to achieve better convergence in Appendix C. We used 1000 epochs and applied StepLR to reduce the learning rate by half every 100 epochs, with an initial learning rate of 0.08. The final results on CIFAR10 are as follows:

   no augmentation	2 augmentations	4 augmentations
   87.16	87.47	88.72

   As shown in the table, standard augmentation has no significant impact on performance, while enhanced augmentation improves performance from 87.16% to 88.72%. However, compared to the performance gains from data augmentation in traditional BP, the improvement with our method is less pronounced. This result demonstrates that DeeperForward is not sensitive to data augmentation, compared to BP.

Thank you for the insightful recommendation!

Thank you for your valuable and detailed review. The latest revised version has been modified and uploaded based on the reviewers' comments.

1. As mentioned in the Conclusion under Limitations, tasks with large-scale datasets like ImageNet, which involve 1000 classes, remain a challenge for our method. This is because the number of channels per layer must be a multiple of the number of classes, and this multiplier is preferably greater than 10 to extract sufficient information. This requirement leads to impractical memory and time consumption for real-world applications. We experimented with a ResNet17 architecture where the channel numbers for the 4 stages were modified to [1000, 1000, 2000, 2000] and trained for 20 epochs. However, the Top-5 accuracy was below 10%, due to the inadequacy of channels.

   Currently, pure FF-based methods are predominantly evaluated on benchmarks like CIFAR10, MNIST, and FMNIST to validate feasibility, making our comparisons fair and reasonable. Moreover, our method significantly improves the performance of FF-based approaches on these benchmarks. Nonetheless, applying FF-based methods to larger datasets remains an important direction for future exploration.

2. ( We have added the experiment on deactivated neurons in Appendix I to analyze the relationship between the deactivated neuron ratio and the number of layers. ) As discussed in Appendix D of the paper, we conducted experiments on architecture depth, including ResNet models with 33 and 100 layers. The results show that both the 33-layer and 100-layer ResNet models perform worse than the 17-layer model.

   ResNet17	ResNet33	ResNet100
   86.22	85.43	83.06

   Upon analysis, we found that the 33-layer model suffers from slower channel expansion as the depth increases, leading to weaker feature extraction capabilities in the shallow layers compared to the 17-layer model. On the other hand, the 100-layer model uses a bottleneck structure, which limits the local feature extraction capacity and makes it less compatible with our method. Additionally, the FF algorithm is inherently greedy, aiming to extract higher-level features from the previous layer. However, it becomes more difficult to extract even higher-level features from already high-level features in deeper layers.

   In deeper networks, convolutional layers focus on matching high-level features, which results in only a small number of neurons being activated. This, in turn, makes it more challenging for subsequent layers to learn new information. To illustrate this, we have added Appendix I, which provides a detailed analysis of the relationship between the deactivated neuron ratio and the number of layers.

   In summary, the 17-layer architecture delivers better performance, and our method marks a significant improvement over other FF-based approaches.

3. (The results of mentioned papers have been added in Table 1.) Thank you for providing the two block-wise local update papers. Both leverage auxiliary networks to perform local backpropagation updates and even surpass BP in performance. However, FF research focuses on brain-inspired training, aiming to address BP's bio-implausibility issues, such as the one-directional nature of neurons. In FF, weights are updated during the forward process, eliminating the need for distinct forward and backward phases. In our original paper, non-FF methods were categorized as brain-inspired algorithms. We have added the papers you provided to Table 1 for a more comprehensive comparison. Furthermore, combining FF algorithms with block-wise BP represents a promising research direction that warrants further exploration.

4. (Section 4.5 has been modified based on your suggestion.) Thanks for your suggestion. To test the performance on more GPUs, we used another machine with 4 GPUs (TITAN X). The test results are as follows:

   METHOD	1 GPU	2 GPUs	4 GPUs
   BP (DDP)	51.98s (1.0x)	32.70s (1.59x)	19.92s (2.61x)
   Ours	36.38s(1.0x)	20.77s (1.75x)	14.68s (2.48x)

   In the extended experiments, we observed that the acceleration rate on 4 GPUs was inferior to DDP. By monitoring GPU utilization, we found that GPU usage initially reached 100% at the start of training but subsequently dropped to around 50% on each GPU. This issue arises due to varying training speeds across layers, causing queue saturation and resulting in bottlenecks. Pipeline programs can only achieve their maximum advantage when computational loads across layers are balanced. In this paper, we demonstrate the feasibility and potential of model parallelism using a simple multi-threading approach. Further optimization can involve integrating advanced pipeline programming techniques, which would be a promising direction for future research.

5. ( We have rewritten the Memory-saving strategy part to make it clearer. ) The memory-saving strategy is a layer-by-layer update strategy, which consists of the following steps: (1) Perform computation for the current layer. (2) Update weights in the current layer. (3) Pass the output to the next layer. (4) Release all intermediate memory used by the current layer. (5) Repeat (1)-(4) to update subsequent layers. Since there is no limitation from freezing activities, this strategy achieves memory savings by promptly releasing memory after each layer's computation, thus minimizing memory usage during training.

6. Figure 4 illustrates our model parallel strategy, where different layers are assigned to specific GPUs (the same GPU can host multiple layers) and parallel training is performed using a pipeline program. In the pipeline program, each layer does not need to wait for the entire network to complete an update before proceeding with the next round of training. For example, in our experiments, we assigned layers 1-8 to GPU 0 and the remaining layers to GPU 1 for training. When using 4 GPUs, we can allocate 4-5 layers to each GPU to balance the computational load. If the number of GPUs exceeds the number of layers, we can assign one GPU per layer, leaving the extra GPUs unused. This approach helps to optimize GPU utilization and alleviate the computational burden on each GPU.

Thank you for the insightful recommendation!

Thank you for your valuable and detailed review. The latest revised version has been modified and uploaded based on the reviewers' comments.

1. As mentioned in the introduction, BP methods suffer from bio-implausibility issues. Therefore, the research on Forward-Forward (FF) training primarily focuses on finding suitable bio-plausible methods to replace BP. Bio-plausible training updates weights during the forward computation, without relying on error backpropagation, which aligns with the unidirectional nature of neurons. Currently, bio-plausible training lags behind BP in terms of performance, including FF-based methods. Our research has identified several reasons for the underperformance of FF methods and has addressed them. Compared to other FF-based approaches, we have achieved a significant performance improvement, narrowing the gap with BP and highlighting the potential of FF-based methods.

2. ( We have rewritten the SIP part to make it clearer. ) In other FF-based methods, it is common practice to simply sum the goodness values of the last one or several layers as the final output. Our SIP strategy, however, seeks the optimal combination of layers whose cumulative goodness achieves the best results. Specifically, we evaluate various combinations by calculating accuracy on untrained data and selecting the best-performing combination as our final choice. Given L layers, there are 2^L possible combinations, making an exhaustive search computationally prohibitive. To reduce complexity, we restrict the combinations to only contiguous layers—summing the goodness values from a specified start layer to an end layer—thus reducing the search space to (L+1)L/2 combinations.

3. ( We have added the experiment on deactivated neurons in Appendix I to analyze the relationship between the deactivated neuron ratio and the number of layers. ) The study on deeper architectures is discussed in detail in Appendix D, where we analyze and explore the performance of ResNet33 and ResNet100. These architectures are derived from ResNet34 and ResNet101, respectively, by removing the fully connected layers and adjusting the number of channels. The following table presents the performance comparison:

   	ResNet17	ResNet33	ResNet100
   CIFAR10 Acc.	86.22	85.43	83.06

   The results show that the 17-layer model performs the best. We attribute this to several factors. Unlike BP, FF can only perceive inputs from the previous layer and greedily outputs higher-level features. In deeper networks, convolutional layers focus on matching high-level features, which results in only a small number of neurons being activated. This, in turn, makes it more challenging for subsequent layers to learn new information. To illustrate this, we have added Appendix I, which provides a detailed analysis of the relationship between the deactivated neuron ratio and the number of layers. Additionally, Section 4.2 and Table 2 demonstrate that the larger the number of channels, the better the feature extraction performance. In the ResNet100 model, the bottleneck structure significantly reduces the number of channels to minimize parameters, which in turn degrades performance. Our method does not rely on error propagation, so it does not encounter the vanishing gradient problem.

4. This experiment primarily aims to demonstrate the parallel acceleration performance of the model parallel strategy, comparing it with the commonly used distributed data-parallel (DDP) method. Although other non-FF methods adopt different training frameworks, they all rely on DDP for multi-GPU parallel training. Therefore, we use BP as the baseline for comparing the parallel acceleration of the model parallel strategy and DDP.

   In this study, we measured the computational cost of DeeperForward, particularly its performance on the CIFAR-10 classification task. On 2 GPUs (TITAN X 24GB), DeeperForward completed 150 epochs of training in 51 minutes 55 seconds, achieving a speedup of 1.75x compared to the single-GPU case. In contrast, BP using DDP completed 150 epochs of training in 81 minutes 45 seconds, achieving a speedup of 1.59x compared to the single-GPU case.

Thank you for your valuable and detailed review. The latest revised version has been modified and uploaded based on the reviewers' comments.

1. As mentioned in the Conclusion under Limitations, tasks with large-scale datasets like ImageNet, which involve 1000 classes, remain a challenge for our method. This is because the number of channels per layer must be a multiple of the number of classes, and this multiplier is preferably greater than 10 to extract sufficient information. This requirement leads to impractical memory and time consumption for real-world applications. We experimented with a ResNet17 architecture where the channel numbers for the 4 stages were modified to [1000, 1000, 2000, 2000] and trained for 20 epochs. However, the Top-5 accuracy was below 10%, due to the inadequency of channels.

   Currently, pure FF-based methods are predominantly evaluated on benchmarks like CIFAR10, MNIST, and FMNIST to validate feasibility, making our comparisons fair and reasonable. Moreover, our method significantly improves the performance of FF-based approaches on these benchmarks. Nonetheless, applying FF-based methods to larger datasets remains an important direction for future exploration.

2. ( We have added the experiment on deactivated neurons in Appendix I to make the paper more convincing. ) Thank you for your feedback, which has significantly enhanced the comprehensiveness of our paper. We tested the average deactivated neurons ratio for each layer in the trained 17-layer ResNet on the CIFAR-10 test set, comparing our mean goodness design with the original square goodness design. The results are presented in the following table:

   The results show that our mean goodness design indeed significantly reduces the deactivated neurons ratio. Specifically, in the shallow layers, square goodness deactivates a large number of neurons by focusing on outliers. Moreover, we observed that the mean goodness deactivated neurons ratio increases as the depth of the network increases. This phenomenon aligns with our observation that high-level feature matching becomes sparser in deeper layers, which also explains why deeper networks (such as ResNet33 and ResNet100) do not show improved performance — extracting higher-level features from high-level features becomes increasingly difficult.

   Your suggestion has been critical for broadening the analysis in our paper, and we have added a new section in Appendix I to describe and analyze the effect of the deactivated neurons ratio.

Table: Deactivated neurons ratio (%) for each layer in 17-layer ResNet on CIFAR10.

Thank you for the insightful recommendation!