We sincerely thank the reviewer for the positive feedback and constructive suggestions. Below, we provide detailed responses to address each of the concerns raised.

Q1: To enable fair comparisons between methods, it is necessary to specify the number of labeled/unlabeled training samples and classes used for each method.

A1: Below, we firstly specify the number of base/novel classes, and then specify the number of labeled/unlabeled training samples.

The number of classes is determined by the dataset split. For the standard splits (i.e., MP-100 Split1-Split5), we follow [71,57,9] and employ 70/20 for base/novel classes, as introduced in Line 212-214. For the cross super-class splits in Table 4 of Appendix, we follow [71,57,9] and employ 99/1, 98/2, 95/5 and 97/3 for base/novel classes in Body, Face, Fur. and Veh. splits. For the Novel Bird split in Table 3, we employ 92/8 for base/novel classes.

The numbers of samples and labels for base/novel classes are determined by the learning paradigm. Specifically, all methods in weak-shot learning (e.g., all methods in Table 1 and Ours in Table 2 and Table 3) employ 3/30 labeled/unlabeled training samples for each base/novel class, as introduced in Line 214-217. For the few-shot methods (with 1-shot) in Table 2, we employ 3/1 labeled/labeled training samples for each base/novel class. For the unsupervised methods in Table 3, we employ 30 unlabeled training samples for each novel class.

Thanks again for the beneficial comments, and we will add the specifications in our revision.

Q2: Additionally, the authors should elaborate on the degree of similarity between the base and novel classes; while Appendix A4 provides some information, this discussion should be extended and included in the main text.

A2: We measure the similarity via the GT keypoints and a ResNet-50 backbone pre-trained on ImageNet. Specifically, we firstly extract an image feature vector for each image, by averaging the feature vectors of GT keypoints on the backbone feature map. Then, we extract a class feature vector, by averaging the image feature vectors belonging to the same class. After that, for each novel class, we compute its similarity to base classes by averaging its similarities to all base classes using class feature vectors. Finally, we obtain the similarity from novel classes to base classes by averaging the similarities of all novel classes.

In the table below, we summarize the similarities for all dataset splits. We could see that, the similarities of standard splits (i.e., Split1-Split5) are robust and intermediate due to random splitting. The similarities of cross super-class splits are significantly lower than the standard splits.

Q3: Table 1 in the supplementary material indicates that shuffling the keypoint order significantly degrades the performance of Hedlin et al., despite this method being unsupervised. What factors might account for this observation?

A3: Firstly, we adapt related methods (e.g., Hedlin et al. [21]) into weak-shot learning paradigm for fair comparison, as stated in Line 243-251. Secondly, our default learning paradigm is weak-shot learning, including the experiments in Table 1 of Supplementary. Therefore, in Table 1 of Supplementary, Hedlin et al. is an adapted weak-shot method, which significantly degrades due to overfitting unpractical relation, as explained in Line 218-223. The comparison to original unsupervised methods (e.g., Hedlin et al. [21]) is presented in Table 3 of main text, which are inferior due to the various keypoint numbers of multiple classes. Thanks again, and we will introduce distinguishing markers to differentiate the original method from its adapted version.

Q4: How computationally expensive is the computation of the correspondence map, considering that the approach requires calculating the cosine similarity between every pair of pixels?

A4: Overall, we follow the implementation of [44] to compute correspondences. In practice, we only need to compute the similarity between all keypoints on image1 and all pixels on image2. In this way, the computation of such correspondence is relatively efficient and affordable, i.e., 0.66 ms and 0.37 GFLOPs. Specifically, the implementations with tensor shapes annotated are shown below:

img1_feature_map = normalize_feats(img1_feature_map) # shape: (B, K, C)
img2_feature_map = normalize_feats(img2_feature_map) # shape: (B, W*H, C)
correspondence_map = torch.matmul(img1_feature_map, img2_feature_map.permute((0, 2, 1))) # shape: (B, K, W*H)

def normalize_feats(feats):
    feats = feats / torch.linalg.norm(feats, dim=-1)[:, :, None]
    return feats

Specifically, for the time cost, the spent time is 0.66 ms, which is averaged over 500 training iterations. For the theoretical cost, the GFLOPs is 0.37, which is measured by the fvcore package through aggregating all floating-point operations of feature normalization to unit vectors and matrix multiplication.

Thanks again for the beneficial comments, and we will add more specifications in our revision.

Q5: Is the "reference image" truly a reference in the conventional sense? According to my understanding, it simply refers to another image belonging to the same class. If so, this terminology may be misleading. In Equation 5, $M_i^*$ does not denote ground truth, whereas the same symbol is used to represent the ground-truth keyness map in Equation 4. This inconsistency could lead to reader confusion.

A5: Our "reference image" just means another image belonging to the same class, and we may employ "paired image" for better understanding. In Equation 5, we regard the transferred keyness map as pseudo ground truth, and we will remove the marker * for better consistency and readability.

We sincerely thank the reviewer for the insightful comments and constructive feedback. Below, we address each concern and question raised.

Q1: It's possible that this issue is coming from a terminology difference between different fields, but as a person from the 6D pose estimation field, the term "Pose Estimation" is somewhat misleading. A better term to be used is "Keypoint Estimation". The paper is mainly focused on keypoint estimation, and no where to find actual "pose" estimation.

A1: We intuitively follow the related works Category-Agnostic Pose Estimation [71,57,9] to use the term of "Pose Estimation", and we will revise it to "Keypoint Estimation" for better understanding.

Q2: Found one typo in line 130. I assume it's "single-class" instead of "sing-class"?

A2: Sincerely thanks for the meticulous comments, and we will correct the typo and revise the paper more carefully.

Q3: Estimating keypoints is not necessarily a pose estimation. It would be great to avoid using the term "pose estimation" too many times in the paper.

A3: Thanks for the insightful comments, and we will employ the precise term, i.e., keypoint estimation.

Q4: It would be great to have actual name on each loss term when they are introduced, for example in line 153: correspondence via XXX, XXX, XXX loss in base class ($\mathcal{L}\_{BP}$, $\mathcal{L}\_{BK}$, and $\mathcal{L}\_{BC}$), and line 154: keypoints via XXX, XXX, XXX loss in novel class ($\mathcal{L}\_{LG}$, $\mathcal{L}\_{NC}$, and $\mathcal{L}_{NU}$)

A4: Thanks for the beneficial comments, and we will add the actual name in the descriptions to improve the readability.

I do not find any significant reason to change my rating. Thanks to the authors for the response.

We sincerely thank the reviewer for the positive evaluation of our work and for acknowledging the insights of our method. Below, we provide detailed responses to address each of the concerns raised.

Q1: Definition of "class". It is not clear to me whether new class means new category (cat to dog) or new instance (cat 1 to cat 2). The paper seems to focus on a new instance. Otherwise, the reuse of the pretrained prompt tokens on different categories may introduce severe bias towards the new category.

Can authors provide a clearer definition of "new class"?

A1: We will add explicit definition, i.e., new class means new category, which follows the definition in related works [71,57,9]. Exactly, the cross-category bias is one of the major issues to address in our task. We will further explain our contributions to this issue in the following Q3-A3.

Q2: Task definition. The paper seems to estimate a set of keypoints that can be matched across different instances. However, the reviewer thinks this work may not belong to the pose estimation task, which aims to detect the predefined set of semantic keypoints, and can benefit the downstream tasks like action recognition, skeleton-driven animation, etc. If the reviewer does not misunderstand, the detected keypoints on the new class in this work are only "semantic-rich", but don't have specific (known) semantic meaning. Therefore, it should be a semantic keypoint detection work instead of a pose estimation work.

A2: We intuitively follow the related works Category-Agnostic Pose Estimation [71,57,9] to use the term of "Pose Estimation", and we will revise our task name to more precise one, i.e., Keypoint Detection or Keypoint Estimation.

As for the downstream tasks, our estimated keypoints have similar properties as the ones in unsupervised keypoint detection [21,19,83], which can also benefit the downstream tasks such as action recognition. In particular, optical flow is semantic-free, but is beneficial for action recognition. Our estimated keypoints may be regarded as sparse optical flow, and the keypoint trajectories across video frames could depict the motion pattern. Besides, as in the regression-based evaluation [21,19,83] described in Line 237-241, the estimated keypoints can be regressed to known semantic meaning.

Discovering keypoints with known semantic meaning is an interesting yet challenging problem, and we will explore more in future works. Our primary focus in this paper is on discovering keypoints for multiple classes, which also has received relatively limited exploration.

Q3: Given that the prompt tokens are pretrained exclusively on the base categories, could using them during the unlabeled training phase for novel categories inadvertently introduce bias?

A3: Indeed, and addressing this cross-category domain gap (bias) is one of the major issues. Preliminarily, reusing the prompt tokens for novel categories is effective due to the keypoint sharing between base categories and novel categories. For example, "bird" and "lion" both have eye keypoints. Moreover, we propose to employ transferred keyness and correspondence to further bridge the domain gap (bias). As explained in Line 41-46, both keyness and correspondence belong to comparing entities(or relations), which are highly transferable. By above designs, our model outperforms existing methods dramatically in learning keypoints for multiple classes without labels.

We sincerely appreciate the reviewer’s positive feedback and constructive suggestions. Below, we provide point-by-point responses to address each concern raised.

Q1: How is the keyness map $M$ used to identify the corresponding keypoint for each pixel?

A1: Actually, the keyness map $M\in \mathbb{R}^{2\times H\times W}$ presents the directions towards more key positions, and we employ it to guide the estimated keypoints to move to more key coordinates. For example, given coordinates $p=[x,y]$, the 2D offset vector $M[p]=[\Delta x,\Delta y]$ fetched from the keyness map $M$ indicates the more key direction. Therefore, the primary role of keyness map $M$ is to guide the keypoint coordinates, rather than identifying the corresponding keypoint for each pixel.

Q2: For the base classes, how is the visibilities $V$ supervised?

A2: In our model, the visibilities $V$ are the max values fetched from heatmaps (Line 179-180), and thus are indirectly supervised by the loss on heatmaps (i.e., $\mathcal{L}_{BP}$ at Eq. (3)). For the base classes, the matched heatmaps are supervised to approaching the GT keypoint Gaussian maps (max=visibility=1), while the unmatched heatmaps are supervised to approaching all-zero maps (max=visibility=0). Considering that the supervision on unmatched estimations is trivial and similar to [6,11,9], we omit it for brevity. Thanks for the considerate comments, and we will add more explanations in our revision.

Q3: The description of the offset-guiding loss (Lines 179–184) is somewhat unclear. ① How is the visibilites $V$ obtained via the argmax operation? ② Is there any thresholding involved? ③How exactly is Eq. (5) used to supervise visibility? ④Does $M_i^*[P_i[k]]$ refer to the keyness at pixel $P_i[k]$? ⑤What is the precise operation of dt()? ⑥How does the gradient propagate through $P$ given the use of argmax operations?

A3: The answers to ①②④⑥: Actually, we employ "soft" argmax operation to obtain coordinates and employ max operation to obtain visibilities, as stated in Line 179-180.

Specifically, given one keypoint heatmap $H$, we compute the keypoint coordinates $\tilde{\mathbf{p}}\in \mathbb{R} ^{2}$ by following "soft" argmax: $\tilde{\mathbf{p}}= \sum_{\mathbf{p}} w(\mathbf{p}) \mathbf{p}$, where $w(\mathbf{p}) = \frac{\exp(H[\mathbf{p}])}{\sum_{\mathbf{p}} \exp(H[\mathbf{p}])}.$ Here, the index $\mathbf{p}$ denotes the normalized pixel coordinates (all spatial grid points), and $H[\mathbf{p}]$ denotes fetching the value at $\mathbf{p}$ from $H$. Similarly, $M_i^*[P_i[k]]$ refers to the keyness at pixel $P_i[k]$.

For keypoint visibility, we compute by $v_n = \underset{\mathbf{p}}{\mathrm{max}} H[\mathbf{p}]$. We do not involve thresholding for computing visibility, because soft values are compatible for supervision and evaluation. The above "soft" argmax operation is differentiable, and identical operations could be found in [18,57].

The answers to ③: As also stated in our Abstract and Introduction, the primary role of Eq. (5) is providing direct guidance for keypoint coordinates. We sincerely apologize for the rigor of expression of "and visibilities" at Line 181, and we will remove it in the final version.

The answer to ⑤: The precise operation of dt() is torch.detach().

Q4: What's the meaning of $K$ In Eq. (3) and Eq. (5)? Why not $M$? Is $K$ the exact number of keypoints?

A4: As described in Line 118-120, $K$ is the exact number of keypoints (including padded and invisible keypoints). Besides, we do not employ $M$ to represent a certain number, but employ $N$ to represent the number of keypoint prompts and thus estimate $N$ keypoint heatmaps for each image.

The loss term in Eq. (3) supervises on $K$ matched heatmaps, because there are $K$ GT keypoints for matching. As for Eq. (5), the loss term should average over $N$, i.e., $\frac{1}{N}\sum_k^N$, because the guidance is applicable for all estimated coordinates, which can also be inferred from the dimensional notation $N$ at Line 180. We sincerely apologize for this symbol mistake, and will revise in the final version.

Q5: No illustrations of $S$ and $w_{l,s}$ in Line141.

A5: Specifically, $S$ denotes the number of subsampled timesteps selected from the diffusion timesteps, and $w_{l,s}$ denotes the mixing weights for the $l$-th layer at the $s$-th timestep. We will add the illustrations in our final revision.

Q6: In Table 5, does the “Basic” result in the first row refer to unsupervised setting? It still outperforms other methods under the weak-shot setting by a large margin. If so, what advantages does the proposed framework have over existing approaches?

A6: In the ablation study presented in Table 5, the default learning paradigm is weak-shot learning, with various loss terms activated. Thus, our basic model (the first row in Table 5) is not in unsupervised setting. Specifically, our basic model disables the loss terms for novel classes (i.e., $\mathcal{L}\_{NU}$,$\mathcal{L}\_{NC}$,$\mathcal{L}\_{NG}$), and learns from base classes and directly estimates for novel classes. Existing unsupervised approaches are designed for single-class scenario, and thus cannot: (1) well tackle different keypoint numbers of multiple classes; (2) well transfer across classes. Our basic model is trained with matching-based loss ( $\mathcal{L}\_{BP}$ at Eq. (3)). In this way, different number of keypoints could adaptively and dynamically match to keypoint prompts, and thus our basic model can well tackle multiple classes. Besides, the keypoints of novel classes and base classes may share the same keypoint prompts, and thus our prompts learned from base classes can also estimate precise keypoints for novel classes. Other loss terms (i.e., $\mathcal{L}\_{NU}$,$\mathcal{L}\_{NC}$,$\mathcal{L}\_{NG}$) can further improve.

Q7: Typo: Line133 "\Phi_^c_l () and \Phi_^c_l () $ as the c-th ..."; Line183 "detach and block" -> "detaches and blocks"

A7: Sincerely thanks for the considerate comments, and we will correct the typos and revise the paper more carefully.