PBADet: A One-Stage Anchor-Free Approach for Part-Body Association

Dear Reviewer iAuQ,

Thank you for your detailed review and for raising critical points about our paper. Your feedback is essential for refining our work, and we have addressed each of your concerns below.

1. Originality of Part-to-Body Offsets:

A: Thank you for highlighting the paper [1] that discusses a concept similar to our part-to-body center offset. However, it is crucial to note the distinct focus of our work on the part-body association task, as opposed to the multi-person pose estimation task emphasized in [1]. This distinction is elaborated in Section 2 "Related Work" of our paper.

The part-body association task, exemplified by models like BodyHands, Hier R-CNN, and BPJDet, involves detecting the bounding boxes of both parts and bodies and establishing their association relationships. In contrast, multi-person pose estimation primarily focuses on detecting individual keypoints of each person.

To illustrate the practical application of our approach, consider hand gesture recognition in a scan room setting. Here, it is essential for the system to respond exclusively to the technician's gestures while ignoring the patient’s hands to prevent unintended interactions. This scenario requires not just the detection of hands (parts) but also the identification of their associated body, necessitating bounding boxes for both hand and body for effective gesture recognition and person classification.

As discussed in [4], while multi-person pose estimation can be adapted for the part-body association task, it requires additional models for detecting part and body bounding boxes. Using pose estimation for association, i.e., determining whether a keypoint lies within a part's bounding box, introduces a two-stage heuristic process that can add complexity and potentially reduce efficiency. Our method addresses this gap by providing a more streamlined, single-stage solution for the part-body association, enhancing both the efficiency and applicability of the approach in practical scenarios.

[4] Who are raising their hands? Hand-raiser seeking based on object detection and pose estimation

2. Definition of 'Single-Stage' Method:

A: In addressing your point about the classification of our method as 'single-stage', particularly in the context of multi-person pose estimation, we acknowledge that there could be a perspective from which it might be seen as a two-stage method. However, our definition of 'stages' is centered around the process of association prediction. We follow the definition of stages in other hand-body association works such as BPJDet, which is considered to be single-stage.

In earlier works like [4] and BodyHands [5], the task of association prediction typically involves two distinct stages. The first stage is the detection of part and body bounding boxes, and the second stage encompasses either employing pose estimation with a heuristic strategy or using a learned association network to link the part and body. This two-stage process, though effective, introduces additional complexity and potential inefficiencies.

Contrasting this, both our method and BPJDet streamline the association task. These models are designed to simultaneously output the part and body bounding boxes along with their association vectors. This integration significantly simplifies the process, effectively making it a single-stage approach in terms of association prediction. By directly outputting both detection and association information, our model avoids the complexities inherent in traditional two-stage methods, thereby enhancing efficiency and reducing computational overhead.

[5] Whose hands are these? Hand detection and hand-body association in the wild

Continue in the next comment

Dear reviewer vvPE:

Thank you for your thorough review and constructive feedback. We appreciate your emphasis on clarity in presentation and the importance of detailed documentation for reproducibility. Please find below our responses to your comments and queries.

1. Improving Clarity in Writing:

A: We acknowledge your concerns regarding the clarity of presentation in our paper, particularly in the Methods and Related Work sections. We understand the importance of clearly articulating the conceptual differences between our method and previous methods. We will undertake a thorough revision of these sections to enhance clarity and ensure that our method and its unique aspects are presented in a more comprehensible and detailed manner.

2. Revising Figures for Enhanced Clarity:

A: We appreciate your suggestions regarding Figure 2 and Figure 1. In the revised version of the paper, we will highlight the parts in Figure 2 where our approach introduces design changes. We will also address the clarity of the arrows in Figure 1 and enhance the differentiation in qualitative results to ensure that the unique aspects of our approach are more evident.

3. Availability of Code and Re-Implementation Details:

A: We have outlined the primary details under Section 4.1 "EVALUATION PROTOCOLS" in the main paper. Regarding the availability of our code, we are in the process of finalizing its release, pending approval. We are actively working towards this goal and plan to update the revised paper accordingly. Furthermore, we are committed to providing even more detailed implementation to ensure that other researchers and practitioners can reliably reproduce our results.

We are grateful for your feedback, as it plays a crucial role in refining our work. We are dedicated to addressing these points comprehensively in the revised version of our paper.

Dear Reviewer TS1Z:

Thank you for your detailed review and insightful queries regarding our paper. Your feedback is invaluable in improving our work. Below, we address each of your points to clarify our methodology and findings.

1. Handling of Occluded Parts:

A: Our method does not explicitly address occluded parts. For instance, in hand-body associations, if a hand is not visible (occluded), it is not associated with a body. This approach demonstrates an advantage over methods like BPJDet, which outputs a constant number of part center offsets regardless of visibility, making the association decoding more complex. Our part-to-body center offset definition ensures that no offset is attached to undetected (invisible) parts, enhancing the method's relevance and efficiency.

2. Dense Per-Pixel Prediction and Computational Efficiency:

A: The one-stage anchor-free object detection approach that we employ, with its dense per-pixel prediction, is established for real-time performance on many platforms. It is more efficient than many two-stage proposal-based object detection methods and traditional anchor-based models, which require multiple predefined anchor boxes for each prediction. Please refer to our response to Reviewer AkLB: "3. Model Capacity and Computational Complexity".

3. Sensitivity to Loss Function Hyper-Parameters:

A: We have calibrated the association loss weight (λ_assoc=0.2) to ensure that all losses are of the same magnitude. Other hyper-parameters are set as defaults in established object detection models. Note that, we maintain these parameters consistently across different datasets to demonstrate the generalizability of our method. For NMS thresholds, we select optimal values for each dataset considering their specific levels of crowdedness, which is in line with protocols followed in previous methods, such as BPJDet, ensuring a fair and effective comparison.

4. Comparative Analysis with Anchor-Based Models:

A: We acknowledge the importance of an in-depth comparative analysis with traditional anchor-based models. BPJDet, a state-of-the-art anchor-based method, has already been compared in our paper. Additionally, we have introduced a new ablation study in our revised paper to compare our model with the traditional anchor-based YOLOv7 model, using the same association definition of part-to-body center offset as in our approach. This comparison has been instrumental in highlighting the advantages of the anchor-free design, especially in the context of part-body association tasks. Our findings show that the anchor-free version of YOLOv7 outperforms its anchor-based counterpart, validating the effectiveness of our method. Furthermore, we compared the traditional anchor-based YOLOv7 model with BPJDet, which also adopts an anchor-based approach but utilizes a body-to-part association definition. This comparison demonstrates that our part-to-body association definition significantly enhances performance, offering clear evidence of the superiority of our approach over previous methods.

5. Applicability Across Various Domains:

A: We have expanded our research to include experiments on the AnimalPosedataset [1] [2], demonstrating our model's scalability and performance in domains beyond human part-to-body associations. The qualitative results of these experiments are included in the revised paper.

[1] Cross-Domain Adaptation for Animal Pose Estimation

[2] AP-10K: A Benchmark for Animal Pose Estimation in the Wild

6. Performance in Scenarios with Overlapping or Closely Situated Objects:

A: We have conducted experiments on the COCOHumanParts dataset, which includes scenarios where people overlap and are closely situated, as illustrated in Figure 4. Additionally, our Appendix includes experiments on the CrowdHuman dataset, featuring more densely overlapping situations.

We hope these responses address your concerns and provide a clearer understanding of our approach and its capabilities. We are committed to making the necessary revisions to reflect these clarifications and enhance the overall quality of our paper.

Dear Reviewer NDUe,

Thank you for your constructive feedback and insightful questions regarding our paper. We appreciate the opportunity to clarify the aspects you have highlighted. Please find our responses to your queries below.

1. AP for Small Objects:

A: We acknowledge your observation about the absence of AP for small objects in Table 2. It is important to note that in the original Hier R-CNN paper, small objects are further categorized into small and tiny objects, and AP for small objects is not explicitly provided. Similarly, another comparison method, BPJDet, also does not offer AP details for small objects. We will include the APs for small objects in our revised paper. We emphasize that Table 2 demonstrates our method's competitive object detection accuracy relative to state-of-the-art approaches. However, the critical metric for our task is association accuracy, where, as shown in Table 3, our method achieves superior performance. This distinction underscores our method's effectiveness in part-body association, which is central to our research objective.

2. Ablation Study on COCOHumanParts Dataset:

A: We have not conducted ablation studies on the COCOHumanParts dataset in our original submission. We acknowledge the importance of such analysis for a comprehensive understanding of our method's performance across different object sizes (small, medium, and large). We would like to highlight that our ablation studies were conducted on the BodyHands dataset, a large-scale dataset with unconstrained images and annotations for hand and body locations and correspondences. This dataset comprises 18,861 training images and 1,629 test images. The results from the BodyHands dataset should provide convincing evidence of our method's efficacy. Nevertheless, we understand the value of additional ablation studies on the COCOHumanParts dataset and, time permitting, we aim to conduct this analysis and share the findings.

Your feedback has been invaluable in helping us improve the clarity and comprehensiveness of our research. We are committed to addressing these points and enhancing the overall quality of our paper.

Dear Reviewer BLsk,

Thank you for your valuable feedback and insightful comments on our paper. We appreciate the opportunity to clarify and address your concerns.

1. Handling Multiple Parts Pointing to the Same Body:

A: We acknowledge your concern regarding how our method handles situations where multiple parts of the same category point to the same body. As detailed in Section 3.4 "DECODING PART-TO-BODY ASSOCIATIONS", our network outputs the center offset of a part, pointing to the body to which the part belongs. We use Euclidean distance between the estimated body center (part center plus the center offset) and the centers of unassigned bodies that enclose the part. The body with the smallest distance (i.e., the distance of the estimated body center and body center) is then assigned as the corresponding body for the given part. This approach ensures accurate and efficient part-to-body association.

2. Typographical Error in Figure 2:

A: We are grateful for your observation regarding the labeling error in Figure 2, where 'P3' and 'P5' in the 'Multi-scale features' were incorrectly labeled. We have corrected this typo in the revised version of the paper to ensure clarity and accuracy in our visual representations.

3. Clarification on the Hyper-parameter K in L_{assoc}:

A: Regarding the hyper-parameter K used in L_{assoc}, we have chosen K=13, aligning with the default value used in established anchor-free object detection methods such as YOLOv7 and YOLOv8. This choice is substantiated in our ablation study (Table 4), where it is shown that performance degrades when K equals the number of anchor points (i.e., w/o Task-Align).

4. Model Training from Scratch or Pretrained YOLO weights:

A: We appreciate your inquiry about whether our model was trained from scratch or using pretrained YOLO weights. To clarify, we followed the same protocol established in the BPJDet paper [1] that uses pretrained YOLO weights. This implementation detail will be explicitly stated in the revised manuscript to avoid any ambiguity.

[1] Body-Part Joint Detection and Association via Extended Object Representation

Your feedback has been instrumental in enhancing the clarity and accuracy of our work. We are committed to making the necessary revisions to address these points thoroughly.

Dear Reviewer AkLB,

We greatly appreciate your thorough review and constructive feedback on our paper. Your points have helped us identify areas for clarification and improvement. Please find our responses to your queries below.

1. Number of Scales Used:

A: In our experiments, the multi-scale feature maps align with the backbone feature representations. Specifically, the YOLOv7 model generates P3-P5 outputs (i.e., three scales), and the YOLOv5l6 model provides P3-P6 outputs (i.e., four scales). This consistency with the backbone ensures optimal utilization of the multi-scale feature capabilities.

2. Clarification on Body Center / Part-to-Body Center Offset:

A: In our model, the body center is determined as the center of the bounding box, following standard practices in bounding box detection. Specifically, the center is calculated using the formula ((r+l)/2, (b+t)/2) for a bounding box defined by coordinates (r, l, t, b). Our ablation study (Table 4) demonstrates that incorporating association loss (for detecting part-body center offset) does not adversely affect hand detection accuracy, showcasing the robustness of our approach.

3. Model Capacity and Computational Complexity:

A: We thank you for highlighting the importance of discussing model capacity and computational complexity. Our paper provides detailed information on model parameter sizes and input image sizes in Tables 1 and 3. To further emphasize the efficiency of our model, we have included a comprehensive analysis of inference speed.

Specifically, we have assessed the frames per second (FPS) of PBADet using YOLOv7 and YOLOv5l6 detectors on a GPU V100. With an input size of 1280x1280, PBADet achieves approximately 84 FPS with YOLOv7 and 63 FPS with YOLOv5l6. These numbers indicate that PBADet is well-suited for real-time applications, demonstrating significant efficiency in processing.

For context, we compared this with the ResNet-101-FPN-based detector Mask R-CNN, which is utilized in methods such as BodyHands [1] and Hier R-CNN [2]. The official FPS for Mask R-CNN is about 10 to 20 per image on an NVIDIA V100 GPU & NVLink [3] [4]. When considering the additional computational load due to the association network in BodyHands [1] and Hier R-CNN [2], it is evident that these methods lag significantly behind PBADet in terms of speed. Note that our method has an input size of 1024x1024 while BodyHands [1] and Hier R-CNN [2] use an input size of 1536x1536 (note that the original Mask-CNN uses 800x800), which further enlarges the FPS gap. This disparity in processing efficiency highlights the advanced capability of PBADet, particularly when it comes to real-time processing demands.

[1] Whose Hands Are These? Hand Detection and Hand-Body Association in the Wild, CVPR 2022

[2] Hier R-CNN: Instance-Level Human Parts Detection and A New Benchmark, TIP 2020

[3] Mask R-CNN

[4] https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md

4. Impact of Multi-Scale Module:

A: Our ablation study in Table 4 indicates that interpreting the center offset as a normalized absolute distance without multi-scale considerations leads to a reduction in hand prediction accuracy and joint AP. This is because each point in the multi-scale dense feature predicts a center offset, and it is challenging to achieve convergence for different scales to output the same absolute distance. The multi-scale module thus plays a critical role in enhancing performance.

5. Optimization of Loss Weights and NMS Thresholds:

A: The loss weights during training remain the same across different datasets. However, the NMS thresholds are not loss weights and are post-processing parameters in object detection. NMS thresholds are optimally selected for each dataset since different datasets present varying levels of crowdedness. This approach is in line with protocols followed in previous methods, such as BPJDet, ensuring a fair and effective comparison.

We hope these clarifications address your concerns and contribute to a better understanding of our work. We are committed to making the necessary revisions in our paper to reflect these clarifications and enhance its overall quality.