Enabling Fine-Tuning of Direct Feedback Alignment via Feedback-Weight Matching

Thank you for your valuable comments on our work. We appreciate your detailed feedback and questions regarding the shortcomings. Below, we provide our answers to your review.

---

Response to Q1, W1, W2:

We acknowledge this issue. Since our method relies on weights trained with back-propagation, we recognize its limitation in achieving lower performance compared to back-propagation.

First, it is difficult for the DFA method to surpass the performance of back-propagation.

This is a fundamental limitation of DFA, not our method. Unlike back-propagation, which tracks gradient changes layer by layer, DFA transmits a single error to all layers using random feedback matrices to approximate the gradient. This inevitably leads to performance differences. In fact, in existing papers as well, it is rare for DFA to surpass back-propagation in terms of performance.

Therefore, rather than aiming to surpass back-propagation, we focused on improving the performance of DFA. Specifically, we aimed to address the issue where fine-tuning with back-propagation weights does not work well in DFA.

Fine-tuning pre-trained weights with random feedback matrices often results in poor performance, as demonstrated in our experiments. In some cases, it even underperforms compared to training from random weights. In the case of training the BERT-Small model on the MRPC dataset, the accuracy is 75.2% for training from scratch, 70.9% for fine-tuning from back-propagation weights, and 92.5% for our method.

This is because the random feedback matrices are entirely unrelated to the pre-trained weights. In DFA, it is the feedback matrices—not the weights—that transmits errors to each layer during the backward pass, making its role crucial.

Our contribution lies in improving fine-tuning performance in DFA by considering the feedback matrix that incorporates high-performing pre-trained weights into DFA.

Although not mentioned in the paper, we believe that our method would be effective in scenarios where DFA is utilized more efficiently than back-propagation, as suggested by Launay, Julien, et al. in "Hardware beyond backpropagation: a photonic co-processor for direct feedback alignment" (arXiv preprint arXiv:2012.06373 2020).

Dear Reviewer 49ZY,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission6057

Thank you for your efforts to provide valuable comments on our work. We appreciate the reviewer's comments on what we need to consider more from an experimental point of view. We will add the part of the experiment as soon as possible.

---

Response to Q1:

Are we correct in understanding that your question is asking whether our method can be applied not only to models with fully connected layers but also to models like CNNs?

Unfortunately, our method is currently only applicable to fully connected layers.

As of now, the DFA method does not perform well with convolution layers. According to Crafton, Brian, et al. in "Direct feedback alignment with sparse connections for local learning" (Frontiers in Neuroscience 13, 2019), when DFA is applied to convolution layers, spatial information is not effectively learned, which hinders proper feature extraction. As a result, when fine-tuning a CNN model, the accuracy on ImageNet is 6.2%, while transferring learning with only fully connected layers yields an accuracy of 65.3%.

We believe this may be because DFA does not fully capture the characteristic behavior of CNNs, where different features are extracted at each depth of the model while the same error is transmitted.

Additionally, in the case of convolution layers with DFA, the feedback matrix transmits the error by unfolding the weights with dimensions (kernel_size, kernel_size, in_dim, out_dim) into a 2D form. Another challenge is that, in the feedback-weight matching method, one of the conditions for matching $W_l = F_{l-1} F_l$ is that $F_l^\top F_l = I$. Finding a feedback matrix that satisfies both $W_l = F_{l-1} F_l$ and $F_l^\top F_l = I$ for the 2D matrix $W_l$ is not straightforward.

However, by developing additional methods to address these challenges, we hope to extend our approach to other vision models in the future.

---

Response to W1:

It seems that we missed it because other papers rarely trained DFA in transformers. Thank you for your opinion. Our method will be effective in vision transformers with fully connected layers. We will do additional experiments and reply quickly.

---

Response to W2:

The reason we did not use a large dataset is that our method does not utilize convolution layers. As mentioned earlier, since fully connected layers are used in DFA, the performance on large datasets like ImageNet tends to be low, making accurate evaluation difficult. This is why other DFA-related papers also rarely evaluate their methods on ImageNet.

For this reason, we chose not to include results on large datasets. However, if our approach proves effective in vision transformers, it could certainly be extended to larger datasets in the future

---

Response to W3:
We also recognize that it might be difficult to understand. We will try to make it a little easier to follow. If you could point out which parts are unclear, we will take that into account and revise accordingly.

We would like to share the experimental results conducted on the ViT model. This experiment provided many valuable insights, and we appreciate your feedback and suggestions.

The table below shows the results of training on the CIFAR-10, CIFAR-100, STL-10, and Imagenette datasets using ImageNet-21k pre-trained weights. All images were resized to 224x224 resolution. The values to the left and right of each '/' represent the accuracy at 5 epochs and 10 epochs, respectively, using the ViT model. "DFA ours" refers to feedback-weight matching, while "DFA fine" refers to fine-tuning from back-propagation weights.

Table 1 : Performance of ViT models

Note: Bold indicates better result.

In the ViT model, our method outperformed not only DFA from scratch but also fine-tuned DFA from back-propagation weights. This demonstrates that our approach is effective in the Vision Transformer model as well. However, it still underperforms compared to back-propagation. As the DFA results indicate, DFA currently appears inadequate for application to Vision Transformers.

We hypothesize that delivering the same error across layers may impede the extraction of features with varying spatial characteristics, as seen in models like CNNs. This could explain the observed limitations. However, this assumption remains speculative due to a lack of supporting studies, highlighting the need for further research.

We will consider addressing these challenges in CNN and vision transformer models as one of our key objectives in future research.

Dear Reviewer tMBJ,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission6057

Thank you for taking the time to review our paper. We greatly appreciate your thoughtful and insightful feedback, which has been invaluable to us. We will revise the paper based on your suggestions. Below, we have provided detailed answers to your questions.

---

Response to Q1:

We understand your opinion, and we apologize for presenting this in a way that makes it difficult to understand.

The expression we used, “between $W_{1<l<L}^t$ and $F_{l-1}^\top$,” is represented as an equality in Equation (3), but our text states that there is no relationship, which seems to have made it even more difficult to understand.

What we intended to explain is that in weak alignment, the relationship between $W_l$ and $F_l$ cannot be determined. This is because, in weak alignment, the alignment matrix term $A_l$prevents a clear definition of the relationship between $W_l$ and $F_l$. The alignment matrix contains a combination of inputs and errors, which makes it challenging to establish a clear connection.

We will revise the original text by removing the $A_l$ term from the explanation and explicitly state that no relationship can be identified between $W_l$ and $F_l$. Thank you for pointing out this overlooked issue.

“Equation (3) is referred to as weak weight alignment (WA) (Refinetti et al., 2021), representing the state where no particular relationship exists between $W_{1<l<L}^t$ and $F_lF_{l-1}^\top$ and between $W_L^t$ and $F_{L-1}^\top$”

---

Response to Q2 and W2:

We understand that you interpreted the results to mean that among the DFA methods, DFA from scratch achieves the highest performance, and in Table 1, using CIFAR-100, our method achieves the highest performance, which seems contradictory and inconsistent. Is our understanding correct?

To address this, it seems there might have been a slight misunderstanding in interpreting the data.

While there are cases where DFA from scratch achieves high performance, in most cases—except for training BERT-Small on the MNLI and RTE datasets—our method outperforms DFA from scratch. So, we believe that both tables show consistent results that demonstrate the superiority of our method.

We feel that this misunderstanding could be related to the lack of detail in the table description.

We will add a statement to the table description indicating that bold values represent the high-performance results among the fine-tuning methods using DFA.

---

Response to Q3:

We also recognize that dataset STL has lower performance. We believe that it is due to the following reasons.

• The STL dataset has 10 times smaller training data than CIFAR-10. STL10 has only 500 samples per class, and CIFAR100 has 5,000 samples per class.
• We resized the original 96x96 images to 32x32 for training. The reason for resizing to 32x32 was to evaluate fine-tuning performance using the same pre-trained weights; changing the input size would alter the weight dimensions of the fully connected layer. We think it affected the lower result because we used data with smaller size than the original data.

---

Response to W2:

The proof is short, and we intended for it to be understood while reading the paper, but it seems to have caused confusion. To make it clearer, we will move the proof to the appendix and revise the paper.

Thank you for your feedback. It will certainly help make the paper better.

Dear Reviewer R6RF,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission6057