BiDRN: Binarized 3D Whole-body Human Mesh Recovery

Q4-1: For the results shown in Table 1, please consider including more recent BNN-based methods. Currently, the latest work in the table is from 2021, which is three years old. Including more recent work would make the comparison more reasonable, as the BNN domain has advanced in recent years (as stated in the related works).

A4-1: Thank you for your valuable suggestion. Unfortunately, to the best of our knowledge, more recent BNN-based methods focus on specific domains and are not designed as general-purpose solutions. For instance, BBCU [ref1] is tailored for image restoration, while BiLLM [ref2] targets large language models. Nevertheless, your suggestion is valuable, and we attempted to adapt BBCU to the HMR task by applying its Basic Binary Conv Unit to the backbone. The results demonstrate that our BiDRN significantly outperforms it.

[ref1] Basic Binary Convolution Unit For Binarized Image Restoration Network, ICLR, 2023.

[ref2] BiLLM: Pushing the Limit of Post-Training Quantization for LLMs, ICML, 2024.

Q4-2: As BNNs significantly reduce computational requirements, it would be helpful to check the FPS rate for a video for the BNN network compared to the original network, which could better illustrate whether the BNN-based model can achieve real-time HMR.

A4-2: Thank you for your suggestion. We have provided the inference time comparison in Table 7 of the Appendix, where our BiDRN achieves impressive acceleration compared to Hand4Whole. The deployment to support video inference requires more engineering work, and we will incorporate this into our future work.

Q4-3: If authors are able to provide the FPS rate for the reconstruction for a video, along with the resolution of the frames and computational resources, it can be very helpful to further validate how fast BNN can achieve for HMR task.

A4-3: Thank you for your suggestion. Adapting the method for video reconstruction involves significant engineering effort. Due to time and hardware constraints, we plan to address this as part of our future work.

Q3-1: This paper introduces a binarization model based on the state-of-the-art multi-stage method, Hand4Whole. The experiments and binarization techniques are tailored specifically to optimize this baseline. However, it remains unclear whether the proposed binarization approach is compatible with more efficient one-stage methods, such as MultiHMR or AiOS, which may offer greater potential for faster performance on mobile devices. If the proposed binarization method cannot be applied to other whole-body HMR methods, its novelty and contribution to the field would be limited.

A3-1: Thank you for your suggestion. We add additional explanations and experiments below.

1. Our BiDRN is compatible with one-stage methods as well. Since the majority of parameters in these methods reside in the backbone, the LCR and BR techniques, along with the four redesigned modules introduced by BiDRN, can be adapted for use in their backbones.
2. Although our method is compatible with both MultiHMR and AiOS, resource constraints (as AiOS requires substantial training resources, including at least 16 V100 GPUs) prevented us from conducting experiments on the AiOS model. Nevertheless, we followed your suggestion and applied our BiDRN to MultiHMR. For a fair comparison, we replaced the backbone of MultiHMR from ViT to the convolution-based ConvMixer, as all the binarization methods discussed in this paper are based on convolution. Due to time constraints, we have not yet completed all experiments. At this stage, we present the results obtained after 70,000 iterations and will include a detailed discussion in our paper once the training is finalized.

Q3-2: The challenges are described as engineer problems, such as the quality of backbone features or dimension mismatching. The motivation behind all the binarization designs hasn’t been discussed in an obvious way.

A3-2: Thank you for your feedback. We provide additional insights into our method to better clarify its motivation:

1. The binarization process often results in significant information loss, making the full-precision input particularly valuable under such conditions. To address this, we introduced the Local Convolution Residual (LCR) and Block Residual (BR) mechanisms, designed to bypass and retain full-precision information. This approach is also inspired by residual connections in full-precision networks, which facilitate training and enhance image feature representation.
2. Dimension mismatches are almost unavoidable, as many networks include operations that alter the shape of hidden features. These mismatches hinder the flow of full-precision information. To overcome this limitation, we redesigned these modules to ensure compatibility with LCR, enabling seamless propagation of full-precision information from input to output.

Q3-3: The scientific value of the proposed method might be determined by whether the proposed method can be generalized to the other HMR methods, especially the one-stage methods, AiOS / MultiHMR, which adopts a very different architecture.

A3-3: Thank you for your feedback. The proposed BiDRN can be extended to one-stage HMR methods, as these methods also rely on a feature extraction backbone. Consequently, the LCR, BR, and the four redesigned modules can be seamlessly integrated into their backbone architecture. We have responded to another similar question, Q3-1. Please refer to A3-1 for more details.

Q2-6: How is the prediction head of SMPLX parameters binarized?

A2-6: Thank you for your question. Following prior work in the binarization field (e.g., retaining full precision in the prediction head for large language models), we maintain the prediction head in full precision. This ensures that the output range will not be restricted by the effects of binarization.

Q2-7: A visual architecture comparison with other binarized methods could help to capture the gist and novelty of the proposed design, especially when previous methods have similar components, e.g., Bi-real also adopts a piecewise polynomial function and shortcuts.

A2-7: Thank you for your suggestion. We have added architectural comparisons in Figure 2 and Figure 3 of the Appendix to highlight the differences between BNN, Bi-Real, and our BiDRN. Specifically:

1. In the Identity Layer, BiDRN introduces RPReLU to redistribute the binarized output before applying the residual connection.
2. In the Down Sample Layer (and other convolutional modules redesigned by BiDRN), Bi-Real cannot apply a residual connection due to the dimension mismatch issue. In contrast, BiDRN redesigns the Down Sample module to integrate the residual connection, preserving as much full-precision information as possible.

Q2-8: Typos: "e.g.the" -> "e.g., the", "Hands regions" -> "Hand regions".

A2-8: Thanks for pointing out the typos, we have refined them in the new version.

Q2-1: The proposed method is not clarified clearly. In the down sample residual block (Figure 4, left), why the dual binarized convolutions with the same inputs, kernel sizes, and strides are needed? How does it differ from a grouped binarized convolution (i.e., with 2 groups)? As for the FullPrecision convolution in the block residual, its kernel size and stride are 1 and 2, respectively. Will this result in checkboard artifacts?

A2-1: Thank you for your question. We add more explanations below.

1. As shown in the left part of Figure 4, the outputs of dual binarized convolutions are concatenated because the Down Sample doubles the channel dimension. To integrate it with LCR, we must ensure that the input and output dimensions of the binarized convolutions remain consistent. Therefore, we use two binarized convolutions, each incorporating LCR, and concatenate their outputs along the channel dimension.
2. A grouped binarized convolution with two groups cannot bypass the full-precision input. However, the Down Sample Residual effectively integrates with LCR, thereby preserving as much full-precision information as possible.
3. Checkerboard artifacts typically arise during upsampling operations, such as transposed convolutions. However, the convolution in the block residual performs downsampling. Furthermore, no checkerboard artifacts were observed during our experiments.

Q2-2: None of the competitors is designed for human mesh recovery. I wonder whether the proposed method still has significant advantages in efficiency compared with those tailored for this task (in terms of GPU memory and FPS), such as [1-2]. [1] Dou, Zhiyang, et al. Tore: Token reduction for efficient human mesh recovery with transformer. ICCV 2023. [2] Zheng, Ce, et al. Potter: Pooling attention transformer for efficient human mesh recovery. CVPR 2023.

A2-2: Thank you for your suggestion. We explain it below.

1. Unfortunately, both [1] and [2] are designed only for the SMPL model, whereas the baseline Hand4Whole and our BiDRN are tailored for the more advanced SMPL-X model. Since [1] and [2] cannot be adapted to HMR based on the SMPL-X model, we believe it would not be a fair comparison, as the SMPL-X model requires estimating more parameters for HMR tasks.
2. To ensure a fair comparison, we applied other efficient methods to Hand4Whole, including the lightweight backbone and pruning techniques outlined in Table 4 of the main paper. The results demonstrate that our BiDRN achieves superior performance while utilizing fewer parameters and operations.

Q2-3: The qualitative results show that some meshes predicted by the proposed method do not align with images well (e.g., hands and faces in Figure 7).

A2-3: Thank you for your feedback. We add more explanations below.

1. Considering that BiDRN uses only 22.1% parameters and 14.8% operations of Hand4Whole, the prediction cannot match the full-precision method but is quite well. Compared to other BNNs, our BiDRN aligns the body quite better. Other BNNs might totally fail with unmatched human mesh.
2. As shown in Figure 7, we add a column for Hand4Whole. Interestingly, even the full-precision method occasionally struggles to align accurately with the images, as seen in the second row of Figure 7.

Q2-4: It would be better to include the results of the baseline (Hand4whole) in the visual examples.

A2-4: Thank you for your valuable suggestion. In the refined version, we have added the results of Hand4Whole in visual comparison.

Q2-5: Why binarized deconvolution and linear layers even yield better performance (L345-346)?

A2-5: Thank you for your question. Since BoxNet is designed to predict hand and face boxes, which require only a small number of parameters, we hypothesize that using a dense network in this context might be redundant and could potentially result in overfitting. We hope this observation proves valuable for future research.

Q1-1: The novelty and contribution of the proposed binarization is limited as most of the design is quite straightforward. The block residual is something new to me.

A1-1: Thank you for your feedback. We hope that our BiDRN is easy to follow with these straightforward designs. Experiments show that they are quite effective and provide a large improvement compared to other BNNs. Guided by simple and effective principles, our BiDRN could serve as a foundation of binarized HMR.

Q1-2: The designed residual block is quite specific for ResNet. This limits the scope of the proposed binarized dual residual block.

A1-2: Thank you for your feedback. Although the residual block replaces the ResNet in main experiments, it can be applied to broader CNN-based backbones. For example, we have also migrated the BiDRB to MobileNet, as shown in Table 6 of the Appendix.

We binarized MobileNet using BNN and our proposed BiDRN, and evaluated the models on a 2D keypoint prediction task. The results highlight the effectiveness and versatility of our BiDRN approach.

Q1-3: The experiment is conducted with comparison to other standard binarization methods, which are designed for general networks. This comparison is kind of unfair as the proposed binarization only works for this particular model, or maybe a broader ResNet-like architecture. A comparison to other binarization/quantization methods for networks in 3D human reconstruction or ResNet-like models is needed.

A1-3: Thank you for your suggestion. We add more explanations and experiments below.

1. The compared BNNs are actually designed for CNN, and most of them are applied to ResNet only in their experiments.
2. Our BiDRN is not restricted to ResNet-like architecture, it can be adapted to any CNN backbone. Considering that the compared BNNs are proposed for CNNs, we consider it a fair comparison.
3. To the best of our knowledge, BiDRN is the first work for binarized 3D human reconstruction and no other binarization methods target this region. Yet, we also follow your suggestion to compare with one more binarization method BBCU. Although it is not designed for 3D human reconstruction, it is also applied in ResNet-like models (SRResnet).

The results show that our BiDRN also largely outperforms BBCU with fewer parameters and operations.

Q1-4: It is not very clear why the proposed binarization method target at the Hand4Whole method. There are other (latest) methods (with better performance) in the field.

A1-4: Thank you for your feedback. At the beginning of this project, Hand4Whole is the SOTA method. Yet, our BiDRN can be adapted to other latest methods. We added an experiment by adapting our BiDRN to the latest method MultiHMR [ref1]. Due to the time and resource limit, we haven't finished the experiments. Yet, after training for the same number of iterations and evaluation on EHF, our BiDRN largely surpasses BNN, showing that it can be generalized to other methods in the field.

[ref1] Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot, ECCV, 2024.

Q1-5: Latency. While the OPs is provided to have a rough idea of the number of operations, It is also necessary to see the change of speed with the proposed binarization.

A1-5: Thank you for your suggestion. We have provided the latency comparison in Table 7 of the Appendix, where our BiDRN achieves impressive acceleration compared to the full-precision method.