Training Binary Neural Networks in a Binary Weight Space

We thank the reviewer for suggesting an experimental design to better evaluate our method. We have answered each of the reviewer's questions below.

The experimental results in Tab.3 seem extremely bad on medium-sized datasets such as CIFAR-10 and Tiny-ImageNet, for both SGD+STE method and the proposed method. This raises the question of whether it is reasonable to abandon the real-valued weights when training BNNs.

The underperformance of BNNs trained on CIFAR-10 and Tiny-ImageNet (shown in Table 3 of the submitted version) is mainly due to the limited generalization ability of the 4-layer MLP, not the proposed training algorithm. To show that our method (or STE) has a capability to achieve satisfactory performance on CIFAR-10 and the larger CIFAR-100, we conducted additional experiments using AdamBNN (MobileNetV1 architecture) and ReActNet (ResNet-18 architecture), which are detailed in the global response. In summary of the global response, our method achieved a top-1 accuracy of 91.94% on CIFAR-10 and 74.64% on CIFAR-100, demonstrating that our binary-space approach performs sufficiently on these medium-sized datasets.

The author claims that this training strategy is useful for edge devices. However, people seldom train models on edge devices from scratch. Thus, it is more reasonable for the author to conduct experiments on fine-tuning a pre-trained BNN model with binary weights only.

We thank the reviewer for suggesting a more practical experimental setting. We conducted experiments where binary models are pre-trained with ImageNet-1k on a central server and are fine-tuned on a medium-sized dataset obtained on an edge device. The detailed settings and results have been described in the global response.

Since the results given in the paper are far from satisfactory, it is hard to judge the effectiveness of the proposed method.

The underfittings were mainly due to the limited generalization performance of the 4-layer MLP, so we experimented with CNN-based BNNs with more expressive power, and have reported the results in the global response. In the experiments, our method achieved a top-1 accuracy of 91.94% on CIFAR-10 and this accuracy is close to the state-of-the-art in BNN training algorithms.

You still have real-valued gradients during training, how could you reduce memory usage by 33x Besides, memory consumption should be listed in the table.

The fact that our method can achieve a memory reduction of 33x has been described in detail in Section 4. Simply stated, the proportionality coefficients of the maximum memory usage for binary- and real-space training w.r.t. the number of layers L are (D_{hid} + 17B)D_{hid} and (33D_{hid} + 17B)D_{hid}, and the memory usage that is not proportional to L can be ignored when L is sufficiently large. Thus, if L is large and D_{hid}>>B, our binary-space training can achieve 33x memory reduction. However, as already mentioned in Section 4, this is not a realistic setting, and in training 4-layer MLPs on MNIST, the memory reduction is 2x due to the real-valued gradient bottleneck. In order not to confuse the reader, we have revised the introduction and Section 4 to clarify the 33x memory reduction is asymptotic analysis. Additionally, in response to the points raised, we have reported the maximum memory consumption in the additional experiments in the global response. In the training of AdamBNN, which is a middle-sized binary CNN, the memory reduction was >3.6x.

We hope these answers and the global response have addressed all your queries. Should you have further questions, we are ready to provide additional clarifications promptly.

Dear Reviewer 2PTe,

With less than 24 hours remaining before the deadline, we kindly request any feedback you might have. We wish to ensure that all your concerns have been adequately addressed in our response letter.

We appreciate your dedication and effort.

We have addressed the new concerns the reviewer 832g raised (especially, the use of Adam optimizer). As the deadline is approaching, we kindly request a re-evaluation if all concerns have been resolved. We are truly grateful for the dedicated advice provided in this rebuttal process.

We thank the reviewer for suggesting an experimental design to better evaluate our method. We have answered each of the reviewer's questions below.

the experimental section leaves a lot of questions as it is not at the scale of most well-received computer vision or even BNN evaluations.

Particularly, the error rates on CIFAR10 and Tiny-ImageNet are very high, regardless of whether STE training or the proposed EMP mask training is used. Comparing such underfitted (in my opinion) models does not paint a clear picture.

As the reviewer mentioned, the results for CIFAR-10 and Tiny-ImageNet in Table 3 show underfitting and this is mainly due to the limited expressive power of the 4-layer MLP. To adequately evaluate the training performance of our method (and STE) on the middle-sized datasets, we conducted additional experiments using AdamBNN (MobileNetV1 architecture) and ReActNet (ResNet-18 architecture), and have reported the detail in the global response. In summary of the global response, our method achieved a (top-1) accuracy of 91.94% on CIFAR-10 and 74.64% on CIFAR-100 (we utilized instead of Tiny-ImageNet), which were comparable to STE, demonstrating that our binary-space approach performs sufficiently on the middle-sized datasets.

The experimental setup is limited to fully-connected layers. While these layers are certainly well utilized, most modern DNNs and BNNs employ either convolutional or transformer layers, which questions the practical applicability of the method.

What obstacles do the authors forsee in applying the proposed method to BNNs using convolution layers?

We agree with the reviewer that many modern BNNs are CNN-based, and the training performance on these architectures is important for the realistic evaluation of training algorithms. Therefore, as described above, we evaluated the binary-space training using the popular ResNet-18 and memory-efficient MobileNetV1, and have reported the results in the global response. In the original manuscript, the evaluations of the proposed method were limited to MLP, not because our method is unable to train CNN-based BNNs, but to provide a pure comparison of real- and binary-space training in the most basic structure. In fact, as the additional experimental results (Ex.2 in global response) show, our method has been experimentally demonstrated to achieve comparable performance to STE in training convolutional layers.

We hope these answers and the global response have addressed all your queries. Should you have further questions, we are ready to provide additional clarifications promptly.

Dear Reviewer 832g,

With less than 24 hours remaining before the deadline, we kindly request any feedback you might have. We wish to ensure that all your concerns have been adequately addressed in our response letter.

We appreciate your dedication and effort.

Thank you for taking the time for us. We are grateful for the further advices. However, it seems there is a bit of a misunderstanding regarding underfitting, so let us explain in more detail.

In Ex. 1, the lower accuracy of ReActNet are not indicative of our method's lack of training capability, but rather caused by the limited feature extraction capability of ReActNet. In fact, the following reasons support this:

1. In the fine-tuning experiments of Ex. 1, we used a fixed backbone to extract features from CIFAR-10/100, and this backbone employs officially provided pre-trained weights trained on ImageNet-1k.
2. We conducted a further additional experiment; training the real-valued header by Linear Probing with ReActNet backbone, and the performance was 49.15%±0.09 (on CIFAR-10). The low accuracy of the LP indicates that the features of ReActNet themselves are inseparable. Note that our method outperformed the LP.
3. Our method achieved a comparable performance to baseline (SoTA) real-space training methods, including ReSTE.
4. In AdamBNN in Ex. 2, the performance exceeds 91.9% on CIFAR-10 (which cannot be considered underfitting), indicating our method performs well with separable features (, or in full-model fine-tuning).

As already mentioned in the global response, while ReActNet's results indeed seems to be underfitting, we showed the results of ReActNet to demonstrate that our method achieves comparable performance with other real-space training methods, regardless of the type of backbone used. The performance in non-underfitting situations has already been demonstrated with AdamBNN, proving sufficient training capability of our method through experiments.

Also, regarding the 'respective paper,' no results of CIFAR-10 fine-tuning experiments were found in this paper. Therefore, there is no basis to conclude that these results are lower than those reported in the paper (in fact, as mentioned above, the performance is higher than linear probing, indicating that the underfitting is a problem with ReActNet's feature extraction).

Also, I noticed the authors used SGD when training other BNNs but it is well-known that Adam outperforms SGD, particularly for BNN training

Thank you for your valuable advice to improve our experiments. Following the reviewer's suggestion, we have conducted a new experiment using the Adam optimizer.

We conducted the header fine-tuning experiment in Ex.1 using Adam as the optimizer. The experimental setup was the same as in Ex.1, and we used AdamBNN trained on ImageNet-1k as the backbone. The results are shown in Table 3. As shown in Table 3, the performance of ReSTE improved by using Adam, but our binary-space training still achieved the comparable (within the error range) performance to ReSTE, the SoTA real-space training method.

Our method can reduce the memory consumption proportional to the number of layers by immediately discarding gradients after propagating them to the layer below; however it is also possible to combine our binary-space training with Adam. When using Adam, there is an increase in memory consumption in exchange for performance. Note that, in Ex.2, where the full models are fine-tuned, an optimizer like Adam that retains real gradients of all layers consumes considerable memory footprint, which may not be affordable by low-end devices.

These results and discussion will be added in the appendix.

We have also addressed the new concerns the reviewer raised. As the deadline is approaching, we kindly request a re-evaluation if all concerns have been resolved. We are truly grateful for the dedicated advice provided in this rebuttal process.

In addition, if there are any additional comments regarding the following statement made by the reviewer, please let us know. We have not been able to confirm this as a fact.

lower than what was reported in their respective paper

We thank the reviewer for the careful reading of our paper and constructive comments in detail. We have addressed each of the reviewer's questions below.

The analogy from Figure 1(a) to 1(b) is more like binarizing the gradients in a real-valued space.

Let us confirm your question. Do you mean, "In Figure 1 (b), it appears that w is being updated in the real-valued space"? This confusion might have stemmed from the placement of omega in Figure 1(a) and w in Figure 1(b) at identical positions; therefore, we have updated the figure in the revised paper to alter the positioning of the weights.

It is not obvious to the reader that wt* is the ideal target weights in the binary space since the loss landscape may have been changed a lot after binarization. It needs more discussion or justification.

The target weight merely represents the direction, toward which a point that reduces the loss is found. Here, we use the word 'merely', since such a point may not be necessarily located in the binary coordinates and w* might increase the loss. We added a footnote that describes the word choice of 'target weight' so that readers won't experience the same confusion you had.

In addition, previous studies like XNOR-Net and AdamBNN have shown that reflecting changes in the loss landscape of BNNs in the optimizer is crucial for enhancing BNN training performance. Our study mainly focuses on emulating the real-space SGD for BNNs, and the loss landscape problem is not directly addressed; however, the combining these methods with our binary-space training is not difficult. For instance, by using the gradient scaling of the XNOR-Net in our binary-space trainining, we can partially alleviate the change in the loss landscape, and in fact, we used this combination in Ex. 1 in the global response.

It seems that the proposed update rule cannot be applied to propagating the gradients to 1-bit activations, which means STE is still needed when training a BNN.

We do use STE in our method, but the real-valued gradients are computed strictly at the points in the binary coordinates. It means we do not hold real weights at all during training.

• For your interest, our proposed random mask relies solely on the direction of the gradient, not its magnitude. As demonstrated in Table 4 of our original paper, this hypermask performs nearly as well as the EMP mask in larger networks. This indicates that our method is effective even when the gradient is binary. Therefore, in future work, if gradients can be computed entirely within the binary space, our random mask could enable complete binary-space training of BNNs, both in terms of weights and gradients. This is a particularly intriguing possibility for the field.

The experiments need to be done using at least some popular models, such as ResNet.

Acknowledging the reviewer's feedback, we conducted further experiments with ReActNet, which is based on the well-known ResNet-18 structure, and AdamBNN, which is based on the memory-efficient MobileNetV1 structure. The results of these additional evaluations have been detailed in the global response. Notably, in the fine-tuning experiments (Ex. 2), our method attained a top-1 accuracy of 91.94% on CIFAR-10 and 74.64% on CIFAR-100.

The paper does not take BOP [1] as a baseline. BOP should be a closely related work that aims at not using latent weights for training BNNs.

We thank the reviewer for suggesting a proper baseline. We have added BOP to the baseline methods in the additional experiment (Ex. 1) shown in the global response and the revised paper.

• Just for the record, BOP by Helwegen et al. is a study that proposed to reinterpret the latent real-valued weight in the BNN as a real-valued accumulator of the gradient. In contrast, our binary-space training eliminates the real-valued accumulator because the l-th gradient can be deleted after computing the (l-1)-th gradient and there is no need to maintain gradients for all layers at the same time. Thus, our binary-space training and BOP are essentially different optimizations.

We hope these answers and the global response have addressed all your queries. Should you have further questions, we are ready to provide additional clarifications promptly.

Dear Reviewer Wfp6,

With less than 24 hours remaining before the deadline, we kindly request any feedback you might have. We wish to ensure that all your concerns have been adequately addressed in our response letter.

We appreciate your dedication and effort.

We have addressed the new concerns the reviewer 832g raised (especially, the use of Adam optimizer). As the deadline is approaching, we kindly request a re-evaluation if all concerns have been resolved. We are truly grateful for the dedicated advice provided in this rebuttal process.

References
[1] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In ECCV, 2020.
[2] Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng. How do adam and training strategies help bnns optimization? In ICML, 2021.
[3] Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang-Ting Cheng, and Roeland Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. In NeurIPS, 2019.
[4] Xiao-Ming Wu, Dian Zheng, Zuhao Liu, and Wei-Shi Zheng. Estimator meets equilibrium perspective: A rectified straight through estimator for binary neural networks training. In ICCV, 2023.