Signed-Binarization: Unlocking Efficiency Through Repetition-Sparsity Trade-Off

We thank the reviewer for the detailed evaluation and positive comments. The following clarifications are inline with the updated version of the paper.

---

Summary1 “paper proposes a new quantization method…”

Thank you for review. Please refer to common response 1. We believe the confusion lies in thinking Signed Binary is just {0,±1}. The paper proposed the idea of repetition-sparsity trade-off (Section3) and performs quantization-system co-design (Figures1 and 8) to create signed-binarization to address the trade-off (Abstract). SB = {Signed Binary Quantization + Signed Binary EDE + Signed Binary Kernel}.

Summary2 Signed Binary exploits the repetition-sparsity tradeoff

We agree with the reviewer. However, the causality is flipped. Repetition-Sparsity tradeoff causes Signed-Binary to be faster. Repetition-Sparsity tradeoff is the key insight, signed-binary is a tool to prove it.

---

Q1 Diff between Signed Binary w/o Sparsity support and Binary {1,-1}?

Thank you for this question. We want to reiterate that Signed-Binary is a quant-system co-design approach. For a conceptual difference, contrast Figures 3 & 8. For quantitative difference, ablations in Tables 1&7 and Supplementary G show that both have competitive accuracy wrt total parameters and signed-binary is more accurate wrt effectual parameters

Q2 Diff between Signed Binary Quantization Functions {0,±1} and Binary {1,0} w/o Sparsity support?

Thank you for this question. The difference is the accuracy. We trained ResNet18-ImageNet using the two quant schemes in an ablation. {0,±1} was 6% more accurate than {1,0}.

Q3 Paper is incremental work over prior binary research

We request the reviewer to please cite prior work so that we can highlight our contributions better. We believe that our contributions are substantial and are re-iterated in Common Responses 3, 4, and 5.

Q4 Confusion about primary contribution

We thank the reviewer for the question. Please refer to common responses 3 and 4. We hope this addresses all the concerns about the primary contributions.

Q5 performance is limited as signed-binary is 1.26x faster than binary

We understand the reviewer’s perspective. However, as shown in Figure 2, we report gains on Energy, Throughput, Latency, Density, and Accuracy wrt Effectual Params. Our gains are not individualistic but combined because of quantization-system co-design. Also, please refer to Common Response 5.

If you meant to ask why Signed binary is 1.26x faster than binary and 1.75x faster than ternary while giving significant improvements in every other domain – please refer to Section 6.2 Para 2 for a detailed explanation. Sparsity support for general-purpose devices is an active area of research and our speedup is comparable with recent work. To put this in perspective, [1] demonstrates the real speedup of 1.28x-1.64x on general-purpose devices due to unstructured sparsity. Prior works, like ternary quant works (ProxQuant ICLR), simply do not deploy their quantization schemes on general-purpose devices. To our knowledge, this is because the first system to support ternary on CPUs was published in 2021! To our knowledge, we are the first ML work to deploy the proposed method on real hardware. This is a strength of this work.

Please note: To our knowledge, the first work to deploy ternary quant works

Q6 Tables

Table 2-5: ResNet 20 + CIFAR10

Figure 5-6: ResNet 18 + ImageNet

Figure 4: [ResNet18 + ImageNet, ResNet34 + ImageNet, ResNet20 + CIFAR10, ResNet32 + CIFAR10]. Thank you for pointing this out. We have reported the numbers for all baselines in whichever dataset they were available. We have changed Figure 4 and text to highlight this clearly. For example: ProxQuant (ICLR’19) only reported results on CIFAR10 and subsequent literature reviews don’t report ProxQuant on ImageNet.

Q7 Signed Binary Training and Implementation

Thank you for this question. This is mentioned in supplementary F. However, we will clarify this here in the text: Filters are assigned to each function before training and are predetermined. Additionally, Tables 2 & 4 provide ablations quantization function assignments during finding the optimal design.

signed-binary quantization functions should not be swapped during training as it can influence the results. Please refer to Section 5.2 Para 2 & Figure 10. This is because the distributions of {1,0} and {-1,0} are neither zero-mean nor laplacian! They need to be combined to create Laplacian & Gaussian-like distributions for effectual and ineffectual params respectively. We stabilize these distributions using Signed-Binary-EDE.

---

[1] Gong, Zhangxiaowen, et al. "Save: Sparsity-aware vector engine for accelerating dnn training and inference on cpus." In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO),

As this phase of the discussion period concludes, we wanted to reach out to see if there are any remaining questions or clarifications we can provide!

The key concerns that Reviewer pHUD pointed out were (1) clarification about models used; (2) figures and tables; and (3) Clarification on the speedup achieved against the landscape of SOTA inference systems. We have updated figures and tables and marked the corresponding models used in each one. Additionally, figures 1 and 8 help provide more information about the intention of this work. We have also provided explanations with references to demonstrate that the speedup is meaningful. We believe the remaining inline responses, extra experiments, and revised draft (uploaded) also work to address any remaining concerns.

We're excited to see how our reviewers view our paper in light of these changes. Again, if there are any remaining questions, please let us know as we will stay available. Thanks so much!

Q1 Efficient matmul is the primary reason for efficiency and not reduced memory accesses due to repetition

Thank you for the question. We feel that there has been a misunderstanding: the term “weight repetition” is the equivalent of weight quantization during the processing of DNNs on hardware (see B3). The major factor in efficient inference is achieved by reducing data movement from memory and not by simpler matmuls (see B4). To that end, reducing the bit-width and weight repetition (albeit used interchangeably in [3]) have the same effect: reducing data movement from DRAM. Prior work (see B1) shows that repetition reduces memory access up to 20x is plays a central role in SOTA system design [5].

Q2 Explain Claim: Binary chooses to maximize repetition of weights

Binary does not need constraints or regularizations on the objective function. Since binary has two unique weights, repetition happens due to the pigeonhole principle (B3, B2, Page 2 Line 1 of [3]). Repetition has always existed but was missed by prior works. Signed-Binary identifies this and addresses our newly identified trade-off between repetition and sparsity. See Common response C1

Q3 Provide evidence for Binary learning repetitive weights

Carrying forward with the previous answer, consistent with prior work (See B3 above), we observe a similar trend in Figure 11 of this work, where we achieve up to a 15x reduction in operations due to weight repetition for ResNet.

Q4 Show Repetition-Sparsity Trade-Off

Thank you for the question. Signed binary requires fewer operations than binary (Figure6). This can also be shown a higher reduction in operations (Figure11) leading to faster inference (Figure7). This makes SB more efficient than B (Figure2). The table below demonstrates that improving one compromises the other. This makes signed-binary most efficient:

Q5 "In the context of paper, a model with zero weights is extremely sparse and has extremely repetitive filters"

Please name the specific model of your concern.

Let’s visualize this using a toy example for [3,3,256,256] conv layer. This layer has 256 x 256 = 65.5K 3x3 filters. Assuming a uniform distribution of filters, pigeonhole principles will give us:

Q6 Differentiate between naive {0,1} quant and signed-binary-quant functions {0, ±1}. Demonstrate overhead with analysis

Thank you for improving our manuscript. We have added Figures 8 & 9 to visualize inference with signed-binary-quant & naive {0,1} quant. Overhead is only loading one additional unique weight {y} per filter in Signed-Binary. Toy example from Fig 9:

Input: {a,b,c,d,e,f}

Signed-Binary Filters: {x,0,x,0,x,0} {y,y,y,0,0,0}

Naive {1,0} Filters: {x,0,x,0,x,0} {x,x,x,0,0,0}

Inference Dataflow graph Signed-Binary:

PS0 = (a+c)

Output1 = (PS0 + e)*x

Output1 = (PS0 + b)*y

Inference Dataflow graph {1,0}:

PS0 = (a+c)

Output1 = (PS0 + e)*x

Output1 = (PS0 + b)*x

Analysis with ResNet18/ImageNet: Signed Binary is 6% more accurate from naive {0,1} while both methods have comparable ops and latency.

Q7 Diff between Signed Binary Co-design and [7,8] Binary Model Arch Exploration

This work explores quantization-system co-design to prove the existence of repetition-sparsity trade-off. This is orthogonal to [7,8] which proposes a custom Conv+Atten-based DNN architecture for binary quantization. While a special Conv+Atten architecture could be designed for signed-binary, our experiments are sufficient to prove the repetition-sparsity trade-off. Finding the best architecture for signed binary is outside the scope of this work. We have added a discussion about [7,8] in Section 7 of the updated PDF of this work.

We thank the reviewer for the detailed evaluation and comments. We thank the reviewer for giving his perspective. We would like to thank the review as this review led us to the improvement of the manuscript as we have added Figure 6, Figure 9, and Figure 11. Additionally, we have added Supplementary Sections B, C, and J.

Before answering the raised concerns, we would be thankful if the reviewer gives us a chance to first establish common terminologies. Recent fundamental breakthroughs [2,4,5] in the inference of these models have changed the underlying landscape of efficient processing of DNNs. We use the following background information in answering concerns raised by the reviewer.

---

---

Extended Background (Supplementary C) :

---

---

B1 What is Weight Repetition and highlight its impact on inference (used in explaining B2)

Lightning Talk: https://youtu.be/7r9zGOHYf2g?si=wEKq17eTTt7Cda_7

B2 Filter Repetition vs Weight Repetition (used in answering Q2)

Filter repetition is one technique for exploiting weight repetition, which is a broader term. Recent methods [4,5] care about the number of unique weights.

B3 Quantization enforces Weight Repetition (used in answering Q3)

B4 Comparing energy consumed for ADD/Multiply and data movement (used in answering Q1)

The reduction of data movement from the DRAM is the biggest bottleneck (for latency, energy, and throughput) for efficient processing of DNNs [3]. To put this in perspective, loading two numbers from DRAM is 6400x (206x) more expensive than adding (multiplying) them [3]. Thus the goal of efficient inference is summarized as reducing data movement [6]. Quantization makes things efficient because it reduces the amount of data that needs to be read (Chapter 7 of [3]).

---

[1] Binarized Neural Networks, NeurIPS 2016 (https://arxiv.org/abs/1602.02830)

[2] UCNN, ISCA 2018 (https://dl.acm.org/doi/10.1109/ISCA.2018.00062)

[3] Vivienne Sze, Joel S. Emer - Efficient Processing of Deep Neural Networks-Morgan and Company (2020) ISBN: 9781681738314

[4] SumMerge ICS 2021 (https://dl.acm.org/doi/pdf/10.1145/3447818.3460375)

[5] Q-Gym PACT’22 (https://dl.acm.org/doi/10.1145/3559009.3569673)

[6] Eyeriss Tutorial (https://eyeriss.mit.edu/tutorial.html)

[7] N. Guo, arxiv’22.

[8] PokeBNN CVPR’22.

---

---

Rebuttal Starting

---

Using the aforementioned fundamentals, we would be thankful if the reviewer gave us a chance to provide clarifications.

---

Summary1 “The paper proposes a new binarization scheme"

Thank you for your review. Please refer to CommonResponse 1. This work is not a new conventional binarization scheme. Instead, we propose the repetition-sparsity trade-off along with Signed-Binarization, a quantization-system co-design framework to address this trade-off.

Summary2 “It aims to improve the accuracy-model size Pareto frontier”

We feel that there has been a misunderstanding. We talk about the accuracy-effectual parameter Pareto front and never discuss the accuracy-model size Pareto front in the paper. Please refer to Section 5.1

We thank the reviewer for the detailed evaluation and the comments from the reader’s perspective. The following clarifications are in line with the updated version of the paper.

---

Q1 Fixed Figure 1

We thank the reviewer for pointing out the issues in Figure 1. We have now updated Figures 1 & 8 to show that the proposed method is a quantization-system co-design framework to prove the repetition-sparsity trade-off (and the objective is not to create a new quantization scheme). We hope it now becomes clearer that the paper is about re-visiting design principles, i.e., Signed-Binary = {Signed-Binary Quant + Signed-Binary EDE + Signed Binary Kernel} and not just {0,±1} quantization functions.

Q2 Method extension to linear layer

We thank the reviewer for this extensibility idea. Yes, we believe it is possible to extend it to linear layers and have added Figures 6 & 9 to help the reader visualize the same. Repetition-Sparsity trade-off is independent of the model architecture. Prior work shows that ternary gets worse (read less efficient wrt binary) as models get bigger. We believe this is an exciting extension and implore it to the community to take this up as future work.

Q3 Ineffectual parameter and multiplication

We agree with the reviewer and carry forward the terminology of “effectual” and “ineffectual” parameters from previous works [1]. Since zero weights need not be loaded in from the memory they result in an inherent I/O speedup for DNNs. That is why, the difference between “effectual” and “ineffectual” parameters plays an important role in understanding our work.

Ineffectual parameters are the characteristic of the model being processed that signifies how many parameters (read zeros) need not be loaded from the memory. Ineffectual multiplications are characteristic of the underlying system and signify how well the system can take a sparse model and reduce the number of operations due to zero weights.

Q4 What’s new? Key insight?

We thank the reviewer for this question. Proposing a new quantization schema, Signed Binary Quant is a subset of our contribution. The other subset of contribution is how it translates to reduction of I/O calls in hardware, this is where the repletion-sparsity tradeoff comes in. Figure 6 shows us that the weight is multiplied after adding all the corresponding activations it should be multiplied with, i.e., x*(PS_0+e). This results in {1,0} quantization schema and {±1,0} quantization schema having near identical inference kernels. The only difference is the non-zero weight value which is being loaded per filter. To the best of our knowledge, no one has put all of these together to provide significant gains in efficiency.

Please refer to common responses 3, 4, and 5 for findings and impact.

[1] Sparseloop MICRO 2022 (https://sparseloop.mit.edu/)