Pose Prior Learner: Unsupervised Categorical Prior Learning for Pose Estimation

We thank the reviewer for the feedback and have addressed each question below.

Q.1: Clarity of the Motivation for the Iterative Inference Strategy.

R.1: The memory in PPL stores learned prototypical poses, making it a natural choice for correcting inaccurate pose estimations, especially in occluded scenes, through an autoregressive process. The key motivation behind iterative inference is to repeatedly apply the memory bank’s corrections alongside the model’s image reconstruction in an autoregressive manner. This approach often outperforms one-time correction by progressively refining predictions, leading to greater accuracy and robustness. Here’s why autoregressive iterative inference is superior to a single correction step:

• First, progressive Error Reduction: A single correction may not fully capture complex dependencies and can introduce new errors. Autoregressive inference refines predictions step by step, improving accuracy and consistency.

• Second, handling uncertainty: In tasks like pose estimation, initial predictions can be ambiguous. Iterative updates allow the model to incorporate more contextual information at each step, resolving uncertainties more effectively.

• Third, improved generalization and robustness: Iterative inference helps the model learn structured dependencies across spatial and temporal dimensions, making it more resilient to variations, occlusions, and noise.

• Forth, adaptive refinement: Autoregressive models retain intermediate information, allowing outputs to be continuously refined based on prior iterations. In contrast, one-time correction lacks this flexibility and may struggle in tasks requiring sequential reasoning. As requested by the reviewer, we have provided qualitative results in Figure. 4a of the paper, demonstrating how each iteration progressively enhances the clarity of occluded regions, leading to more accurate pose estimations.

Additionally, we have included additional qualitative samples on occlusions in Figure. R3, showing that PPL consistently outperforms AutoLink by reconstructing plausible poses even with partial visual information. These results will be incorporated into the revised version.

Q.2: Concerns about Aggregating Wrong Predictions.

R.2: Generally, incorrect predictions are corrected by the memory bank, particularly under small-area occlusions, as shown in Appendix A4. However, severe occlusions can lead to plausible yet incorrect reconstructions, potentially compounding errors. An example is provided in Figure. R2 where our PPL fails to reconstruct the ground truth pose when there is a large occlusion. Future work may explore confidence-based updates or early stopping via temporal consistency checks. This will be further discussed in the revised version

Q.3: Results on Pose Estimation Under Occlusion.

R.3: Additional qualitative results comparing PPL and AutoLink under occlusion are shown in Figure R3, demonstrating PPL's consistently superior performance.

Quantitative comparisons on Human3.6m between AutoLink and PPL are presented in Table R5. Results indicate AutoLink significantly underperforms compared to PPL due to lacking pose priors and correction mechanisms. These findings will be incorporated in the revision.

Q.4: Discussion of Essential References Not Discussed.

R.4: Discussions of the suggested references are provided below and will be included in the revision:

-MagicPony (CVPR 2023) & Learning the 3D Fauna (CVPR 2024): Both papers focus on learning animal poses through image-level and feature-level reconstruction. However, they rely on ground truth object masks as supervision, which provide essential shape and pose information. In contrast, PPL employs an unsupervised learning approach that does not require object masks, allowing for pose priors to be learned directly from unannotated images.

-Robust Category-Level 6D Pose Estimation (ECCV 2022) & Unsupervised Category-Level 3D Pose (CVPR 2024): These two works share a similar idea of learning category-level pose for pose estimation with PPL. However, they represent poses with dense point clouds, emphasizing scene reconstruction. PPL differentiates itself by representing poses as sparse key points and their connections, which offers a semantically richer and more invariant representation of poses. Thus, while all these methods contribute to pose estimation, the focus of PPL aligns more closely with scene understanding compared to the dense representations used in scene rendering.

We thank the reviewer for the feedback and have addressed each question below.

Q.1: Lack of comparisons with Unsupervised Keypoints from Pretrained Diffusion Models [1]

R.1: We thank the reviewer for pointing us to the recent method [1]. We will include it in the related work and discuss its differences from ours in the revised paper. We highlight the following differences between our method and theirs:

• First, [1] requires large-scale multi-modal data (image and language) to train a Conditional Stable Diffusion Model to start with. Compared to our approach, this method utilizes an extra modality (language) and extra training data.

• Second, [1] and our method has different objectives. While [1] aims to utilize prior knowledge in pre-trained Conditional Stable Diffusion Models for pose estimation, our focus is to learn the prior information from the data itself.

Due to these differences, a direct comparison is not fully fair. Still, as suggested, we compare results in Table. R1 & R2:

• Human Pose Estimation: Our method outperforms [1] on Human3.6m and is comparable on Tai-Chi.
• Bird Pose Estimation: We outperform [1] on the aligned CUB dataset and underperform on the non-aligned ones.

These results suggest that our method performs competitively well with multi-modal methods. This demonstrates that our method is capable of effectively capturing meaningful pose priors and accurately estimating poses despite having zero prior knowledge of extra modalities.

Q.2: Fewer evaluated datasets compared to AutoLink.

R.2: We tested on six diverse categories (humans, birds, flowers, hands, horses, dogs), as detailed in Appendix A7. Here’s why we excluded the datasets used by AutoLink:

• DeepFashion: Lacks background complexity and contains similar human postures—making it less informative than Human3.6m or Tai-Chi.

• CelebA: Focuses on faces, which lack structural variation. Our method is better suited for objects with articulated parts.

• Zebra: Initially omitted due to similarity with horses, but we will add it in the revised version.

Q.3: Lack of visualization comparisons with existing methods, particularly AutoLink.

R.3: We include visualization comparisons between our PPL and AutoLink in the Figure. R1. Under scenes without a background of complex texture, as in Row 1, both AutoLink and PPL can predict correct poses. Generally, PPL outperforms AutoLink by utilizing learned pose priors to guide pose estimation, ensuring that the transformed poses remain within the intended category. In contrast, AutoLink tends to be influenced by intricate background textures, which can result in unrealistic pose estimations. For example, in Row 2 and Row 4 of Figure. R1, AutoLink detected some keypoints on the background and predicted unrealistic poses, while PPL consistently focuses on human body. In Row3, compared with AutoLink, PPL does not predict the lines on the wall as arms, because we have the learned pose prior to constraining the arm from being too long.

Q.4: Missing baseline results without pose priors.

R.4: Our core contribution is learning pose priors without human interventions or prior knowledge. As such, we are unable to establish ablated methods for PPL without the pose prior component. Removing this component would reduce the model to a direct keypoint regressor—essentially similar to AutoLink. We have included AutoLink as a baseline throughout our experiments to address this.

Q.5: Absence of BKind'22 results in Table 2.

R.5: We now provide BKind’22 results on Human3.6m in Table. R3. Our method significantly outperforms BKind’22, and we’ll include this in the revised version.

Q.6: No results for the Memory Bank (1 vector) comparison with PCT'23 in Fig.A1.

R.6: PCT and PPL both use compositional tokens and memory for occlusion, but differ fundamentally:

• PCT: Supervised, using ground truth keypoints and connectivity.
• PPL: Fully unsupervised, without annotations.

To address the reviewer’s point, we created a variant called PPL-1MemBank, which encodes all keypoints into a single vector and uses one memory bank (512 vectors). This matches PPL’s memory capacity (34 banks × 16 vectors).

In Table R4, results show:

• PPL-1MemBank: Normalized L2 error = 2.72
• PPL-full: Normalized L2 error = 2.56

This confirms our hierarchical memory improves performance and efficiency.

Q.7: Table and Figure Positioning.

R.7: Thanks for pointing this out! We will fix these presentation issues in the revised version.

We thank the reviewer for the positive feedback.