UniPose: Detecting Any Keypoints

1. The most important one weakness is that there is no clear explanation for the textual prompts of keypoints. For examples, clothing or table could have ambiguous keypoint names, how to employ textual prompts to detect their keypoints?

During the construction of our UniKPT dataset, one of the great efforts is to address the issues of ambiguous keypoint names in existing datasets. We merge and unify all the keypoints from 13 datasets into 338 keypoints in the UniKPT dataset. And we take great care in annotating each keypoint with comprehensive textual descriptions, incorporating rich directional information such as 'upper,' 'lower,' 'left,' and 'right,' and assigning each keypoint a unique name to add the fine-grained semantic meaning and prevent any conflicts. We will provide a comprehensive list of 338 textual prompts for keypoints in the Appendix of the revised version.

For instance, in the DeepFashion2 dataset, keypoints are identified solely by numerical labels (e.g., 1, 2, 3, 4) for clothing items. In contrast, our annotation process involves the detailed labeling of specific keypoints. For example, when annotating the neckline, we distinguish between six distinct keypoints: 'upper center neckline,' 'upper right neckline,' 'lower right neckline,' 'lower center neckline,' 'lower left neckline,' and 'upper left neckline.' This enables our model to achieve fine-grained text-to-keypoint alignment and utilize corresponding keypoint prompts for precise detection.

2. Another important issue is that all the visualizations are shown without prompts and thus difficult to analyse. It is critical for understanding the effects of various prompts.

Thanks for the suggestions. We will supplement the corresponding prompts along with visualizations in the revised version. For in-the-wild images presented in Figure 2, we employ textual prompts during testing, where these textual prompts are user-friendly and are particularly suitable for large-scale image testing. We default to using the keypoints description defined by AP-10K for animal categories. For the human category, we opt for the keypoints description defined by COCO. The results show that UniPose demonstrates exceptional fine-grained localization capabilities and its ability to generalize effectively across different image styles, instances, and poses.

3. In Section 4.1, why not apply UniPose in the setting of class-agnostic pose estimation as in CapeFormer? The comparison with adapted CapeFormer is unfair and unclear for indicating the performance of UniPose on detecting unseen objects.

In Table. 2, we indeed have conducted a fair comparison with CapeFormer and existing methods in the same class-agnostic pose estimation setting using the MP-100 dataset for training and testing. CapeFormer is a top-down-based method that utilizes ground-truth bounding boxes to crop images into individual object images for performing class-agnostic pose estimation. However, UniPose is an end-to-end method with a significant advantage in its ability to generalize to different objects and detect new ones. To maintain consistency with CapeFormer's settings, we also use ground-truth bounding boxes, foregoing object generalization. The results show that UniPose, as an end-to-end framework, could surpass existing top-down-based approaches with a superior inference time.

Furthermore, it is essential to highlight UniPose's competence in conducting detection and pose estimation in multi-object, multi-class scenarios—an achievement CapeFormer cannot attain. As demonstrated in Table. 3 and 12, we present results without ground-truth bounding boxes and test CapeFormer in single-/multi-object scenarios. These results underscore a notable performance decrease for CapeFormer in single-object scenarios (where disguising the background and object is challenging), and it demonstrates an inability to effectively handle scenes with multiple objects.

4. Lacking analysis for Section 4.3. The performance gaps are trivial, considering the differences on data and tasks.

Existing general models focus on training models to handle multiple vision tasks. However, in the field of pose estimation, their performance is not optimal due to the gap involved in handling multiple tasks, even when they employ large models. Also, due to the limited scope for pose estimation, they all tend to focus solely on tasks like person keypoint detection (e.g., COCO) or animal keypoint detection (e.g., AP-10K).

In comparison, our focus is to unify the keypoint detection tasks and target a keypoint generalist that achieves effectiveness and generality in keypoint detection across any articulated (e.g., human, animal, and detailed face and fingers), rigid (e.g., vehicle), and soft objects (e.g., clothing). In section 4.3, we compare UniPose with existing general models to demonstrate the effectiveness of UniPose as a powerful generalist model in the field of pose estimation.

1. Reading through the paper, it lets reviewer feel that the writing requires significant improvement and the math symbols are in mess.

Thanks for the suggestions. Our main article has aligned mathematical symbols with our implementation details. We will carefully refine this version and improve the notation in the revised edition. If you have specific examples in mind, we would be grateful if you could share them, and we can focus on those for better revisions.

2. There are already many works in literature using visual prompts such few-shot keypoint detection [1,2,3,4], and also the works regarding textual prompts such as [5,6]. The paper should fully discuss these two types of related works. From the technical perspective, it shows little advance compared to existing works. The simple combination of both types of prompts make the contribution and novelty of this work weak.

First and foremost, we would like to emphasize that UniPose is the first end-to-end framework that can generalize to unseen objects and keypoints in multi-object, multi-class scenes. In comparison, all the methods listed above [1-6] are top-down approaches, as they rely on ground-truth bounding boxes to crop a multi-object image into several single-object images and subsequently employ visual or textual prompts for the following single-object keypoint detection. Thus, these methods are unable to address multi-class multi-object scenarios without known instance-level object detection, particularly situations where an image contains numerous objects of different categories with varying keypoint definitions.

Secondly, UniPose is the first multi-modal prompt-based pose estimator. However, the methods [1-6] only support a single modal prompt. Only supporting visual prompts makes the user interaction unfriendly and inefficient, while only supporting textual prompts lack fine-grained low-level visual information and make it hard to localize the indescribable positions. UniPose jointly leverages visual and textual prompts for training via cross-modality contrastive learning to boost the generalization and effectiveness of any keypoint detection. We have demonstrated the mutual benefit between the two kinds of prompts in Table. 10.

Thirdly, UniPose targets a keypoint generalist that achieves effectiveness and generality in keypoint detection across any articulated (e.g., human, animal, and detailed face and fingers), rigid (e.g., vehicle), and soft objects (e.g., clothing), which is trained on the proposed UniKPT dataset. In contrast, references [1] or [2] primarily focus on single-object keypoint detection, specifically within the clothing or animal super-categories. Reference [3,4], despite handling more objects, exhibits limitations in terms of generalizability and effectiveness due to relatively small-scale training data. As for reference [5], as detailed in our related work section, CLAMP concentrates solely on single-object animal keypoint detection. It leverages CLIP with language guidance to prompt animal keypoints containing a fixed keypoint set (e.g., 20) and doesn't consider keypoint-level open vocabulary. The primary emphasis of CLAMP lies in cross-species generalization within a predefined skeleton structure, a domain where UniPose has made substantial advancements. Regarding reference [6], it's noteworthy that this work was submitted to Arxiv in October 2023, coinciding with our work. Also, it focuses on single objects, with a specific focus on animal body and facial parts.

3. Some claims may not be true. For example, ``Xu et al. (2022a) first proposed the task of category-agnostic...". Work [3] may not be the first work as [1-2] are earlier.

In fact, we follow the definition of the category-agnostic pose estimation (CAPE) task by [3]. This task requires the pose estimator to detect keypoints of arbitrary categories, such as animals, clothing, furniture, and persons, given the keypoint definitions.

[1] and [2] primarily concentrate on a single super category like clothing or animals. We will clarify this distinction and explain it in the revised version.