Referring Human Pose and Mask Estimation In the Wild

We sincerely thank Reviewer DJJG for their comments and appreciation of our work. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:

Does a point prompt contain only one point, while a scribble contains 12 points? This paper should provide a formal definition for both point and scribble.

Yes, the point prompt contains only one point, while the scribble prompt contains 12 points uniformly sampled from the curve. We will add the following formal definitions as suggested:

Point prompt: A single point $\mathbf{p}$=$(x,y)$ at any position in the target area, where $x$ and $y$ are the horizontal and vertical coordinates.

Scribble prompt: A scribble can be a continuous, free-form curve represented by an ordered set of $n$ points {$(x_{1},y_{1}), (x_{2},y_{2}), ..., (x_{n},y_{n})$} anywhere in the target area. In this work, we discretize the curve by uniformly sampling 12 points to form the scribble prompt $\mathbf{s}$ = {$(x_{\left\lfloor kn/12 \right\rfloor},y_{\left\lfloor kn/12 \right\rfloor}) \mid k = 1, 2, \ldots, 12$}.

Dagger denotes to adopt all images from MSCOCO into training? How UniHPD can utilize the rest 40K MS COCO images without ref annotation?

Yes, dagger means that all images from MS COCO have been used in training, consistent with previous pose estimation works. Besides the images in RefHuman, we generated point and scribble prompt annotations for the remaining MS COCO images to support the training of UniPHD$^{\dagger}$. We will release these annotations as well.

I think * (namely choose the top-1) should be the default evaluation configuration. What is the evaluation configuration without * and why adopt it as default configuration?

We agree with the reviewer and will adopt $^*$ (top-1 ranked query group) as the default evaluation configuration in the final paper, as it reflects practical deployment where a model outputs a single result per input (L289). Some results are shown in Table A, our method still demonstrates top-tier performance under the suggested PCKh@0.5 and oIoU metrics.

Our method without $^*$ uses 20 query groups for Pose AP and Mask AP evaluation (oIoU always uses only top-1), consistent with previous query-based pose estimation methods. For instance, GroupPose and ED-Pose use over 100 queries to enhance recall, despite most MSCOCO images contain fewer than three people.

Table A: Comparison on RefHuman using the top-1 query group ($^*$).

How about adopting multiple types of prompts to perform R-HPM? For example, using both text and point simultaneously to refer a specific person.

Interesting idea! We sequentially cross-attend visual features with point and text prompts to enhance multimodal representations for R-HPM, achieving significantly improved results due to the complementary information from both prompts, as shown in Table B. Future research could use our introduced RefHuman dataset to develop more advanced models.

Table B: Ablation of using both text and point prompts.

PCKh@0.5 and IoU are more suitable as R-HPM metrics.

Thank you for this insightful advice. We have reported the overall IoU for segmentation evaluation, consistent with the referring segmentation task, and will adopt PCKh@0.5 as the primary pose estimation metric in the revised paper. Please kindly refer to Table A in the previous response. Our model still demonstrates top-tier performance under PCKh@0.5 and oIoU metrics.

Re-implement some referring segmentation methods to test their performance on RefHuman.

Thanks for the suggestion. We trained popular open-sourced referring segmentation models on our dataset, with results in Table C. Using only text prompt to train our model, we achieve top-tier performance compared to competitors like SgMg and GRES. These results will be included in the revised paper.

Table C: Comparison to language-conditioned segmentation models.

[1] LAVT: Language-Aware Vision Transformer for Referring Image Segmentation, CVPR, 2022.

[2] Contrastive Grouping with Transformer for Referring Image Segmentation, CVPR, 2023.

[3] GRES: Generalized Referring Expression Segmentation, CVPR, 2023.

[4] Spectrum-guided Multi-granularity Referring Video Object Segmentation, ICCV, 2023.

Dear Reviewer DJJG,

Thank you for your diligent review of our submission. We have carefully addressed each of your concerns and provided our responses. We would greatly appreciate any additional comments, as your feedback is crucial in strengthening our work.

Your time and consideration are invaluable to us.

We sincerely thank Reviewer TbW6 for their comments and appreciation of our work. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:

What is the relationship between $\mathbf{F}^{vl}$ and $\mathbf{P}^{'}$, and how do you enhance the template based on $\mathbf{P}^{'}$?

Thank you for the comments. We extract the multimodal embedding $\mathbf{F}^{vl}_{\mathbf{c}}$ at the highest-scoring position $\mathbf{c}$ (evaluated using linear layers) to estimate a set of $k$=17 keypoint positions $\mathbf{P}^{'}$ for better query initialization. We enhance the keypoint query template $\mathbf{Q}^{P} \in{\mathbb{R} ^{k \times D}}$ by adding semantically rich multimodal embeddings $\mathbf{F}^{vl} _{\mathbf{P}^{'}}$ extracted at positions $\mathbf{P}^{'}$, which also serve as reference points for deformable attention in the decoder. We will explain this in more detail in the revised paper.

Why do "the keypoint queries struggle to ....."? It seems that these keypoint queries can interact with each other in existing decoders.

Perhaps there is some misunderstanding which can be removed by rephrasing the sentence as "However, after local detail aggregation, the keypoint queries struggle to perceive .......". We first update each query embedding separately by capturing local details, which lacks query interactions. Therefore, we mention this in L222 to introduce our global dependency modeling, which captures keypoint-to-keypoint, keypoint-to-instance, and instance-to-keypoint relations to enhance target awareness and instance coherence. We did not claim that other works' decoders cannot perform keypoint query interactions.

Comparisons of UniPHD with its variants, such as UniPHD without the mask head and UniPHD without the pose head.

Thanks for the valuable suggestion. We performed the comparisons in Table A, which reveal that removing either head adversely affects the performance. This confirms the effectiveness of our synergistic decoder, which facilitates keypoint-instance interactions, enabling bidirectional information flow and enhancing both predictions. We will add this ablation study to the paper.

Table A: Ablation of prediction heads.

Provide an ablation study of the proposed query initialization.

Table B shows the ablation study on query initialization by removing the query enhancement discussed in our response to the first comment. Results show a minor performance decrease due to the absence of dynamic spatial priors.

Table B: Ablation of query initialization.

Compare model parameters and computational complexity.

Thank you for this suggestion. Table C compares the model parameters and FPS (measured on a single RTX4090 GPU). Our method demonstrates better performance at comparable FPS while using slightly more parameters. We will include this comparison in the revised paper.

Table C: Comparison with competitors using the scribble prompt for inference.

Dear Reviewer TbW6,

Thank you for your diligent review of our submission. We have carefully addressed each of your concerns and provided our responses. We would greatly appreciate any additional comments, as your feedback is crucial in strengthening our work.

Your time and consideration are invaluable to us.

We sincerely thank Reviewer apkP for their comments and appreciation of our work. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:

The motivation for obtaining pose in a "referring" manner is unclear. It might be more valuable to focus on text-based human detection, which can open up many avenues.

Our task predicts both mask and pose of the referred person simultaneously using common prompts to provide detailed, identity-aware human representations that benefit human-AI interaction. This has the following advantages over text-based human detection:

(1) Our segmentation provides finer-grained, less noisy information than detection, crucial for robotics and precision tasks.

(2) Our pose estimation adds essential semantics for behavior understanding, complementing mask/box information.

(3) Support for diverse prompts (text, point, scribble), broaden the scope of human-AI interaction.

(4) Our proposed dataset supports various settings, including text-based human detection mentioned by the reviewer.

Reviewer DJJG also thinks the motivation of this work is pretty good.

It might be helpful to increase its capacity.

Thanks for this valuable suggestion. We enhanced the capacity of the multimodal encoder by adding three layers or increasing feature dimensions from 256 to 384. As shown in Table A, both strategies slightly improved overall performance, demonstrating that our current settings are highly effective. We will incorporate the results in the revised paper.

Table A: Increasing the capacity of the multimodal encoder.

The method seems to be two-stage, according to L205-214.

To address the reviewer's concern, we will remove the term "one-stage" since it does not affect our contribution. Previous papers like ED-Pose described "two-stage" methods that first perform detection, then pose estimation on cropped single-human images, or use heuristic grouping to process numerous detected instance-agnostic keypoints. Our method uses linear layers to identify a high-scoring point $\mathbf{c}$ on multimodal features for initializing keypoint queries (L205-214), which directly regress target keypoint positions on the entire image. Our query initialization is similar to the human query selection in ED-Pose, which claims to be one stage. Hence, we call our method one stage because it is end-to-end and avoids cropping, pose estimation on cropped single-human images, or heuristic grouping.

Moreover, the initialization on L205-214 does not even play a significant role in our method, evidenced by an ablation study using only learnable keypoint queries. As shown in Table B, this results in only a minor performance decrease, demonstrating the effectiveness of our method. We will include this ablation study in the paper.

Table B: Ablation of query initialization.

Table 1 does not require last column and typo “Scibble" —> “Scribble”.

Thank you for pointing this out. We will remove the last column from Table 1, correct the pointed typo and carefully check the complete paper.

How does Eq. 4 fit into mask prediction task? Is that applied only to the pose prediction branch?

Our graph attention, including edge construction (Eq. 4), is applied concurrently to both mask and pose prediction branches. Our query set $\mathbf{Q}^{IP} \in{\mathbb{R} ^{(k+1) \times D}}$ comprises one instance (prompt) query and $k$=17 keypoint queries. The graph attention treats all queries in $\mathbf{Q}^{IP}$ as nodes and models keypoint-to-keypoint, keypoint-to-instance, and instance-to-keypoint relations, enhancing all queries simultaneously. Ultimately, the instance query generates dynamic filters for mask prediction, while the keypoint queries estimate keypoint positions.

Dear Reviewer apkP,

Thank you for your diligent review of our submission. We have carefully addressed each of your concerns and provided our responses. We would greatly appreciate any additional comments, as your feedback is crucial in strengthening our work.

Your time and consideration are invaluable to us.