Unified 2D-3D Discrete Priors for Noise-Robust and Calibration-Free Multiview 3D Human Pose Estimation

Q1: Limited methodological novelty: This work proposes an incremental enhancement to existing methods by adding quantized 2D pose features. The DCSA module, while effective for feature fusion, primarily reuses self-attention and cross-attention mechanisms, limiting its conceptual novelty.

A1: We are the first to introduce a data-drive discrete pose prior into continous 3D pose regression for noise resilience, and to propose to fuse discrete and continuous features with complementary strengths. The use of attention for fusion is not very novel but intuitive and effective. However, the novelty stems from the insights of combining data-drive robust discrete priors with precise continuous representations to improve noise robustness and regression precsion, not from a single component. As shown in Figure 3 of the Supplementary Material, our model outperforms the pure-continuous baseline by up to 5 mm under increasing noise levels. Across all noise intensities, our method consistently achieves lower MPJPE than the baseline, demonstrating its strong noise resilience.

Q2: Insufficient motivation and justification: The motivation for introducing quantized features remains unclear. First, can the quantized features maintain consistency with the 2D poses from different viewpoints? Second, the input settings differ between the first-stage learning (single action input) and the second-stage learning (multi-view and multi-sequence input)—could this discrepancy amplify feature misalignment and introduce additional noise?

A2: (1) Viewpoint consistency is neither required nor expected for quantized features.
The quantized features are not designed to maintain consistency across viewpoints. In fact, enforcing such consistency is neither feasible nor necessary. For example, 2D poses from different views (e.g., front vs. side) can differ significantly in visible joints and geometric patterns, and forcing consistency may lead to loss of view-specific information during Stage I. Our goal is not to align cross-view 2D features, but to learn a discrete pose prior that captures robust and generalizable patterns within individual views, which is sufficient for supporting downstream regression.

(2) Stage I and Stage II use different inputs without causing feature misalignment.
Stage I uses single-view single-frame inputs to fully exploit large-scale unpaired 2D pose data, while Stage II leverages multi-view multi-sequence data for final 3D regression. This difference is intentional and serves complementary purposes: Stage I learns a noise-robust discrete prior, while Stage II captures cross-view and temporal dependencies through continuous features.
There is no misalignment, as the discrete prior is not required to encode multi-view or temporal cues—it only needs to offer robust structural guidance. Multi-view and temporal modeling are deferred to the continuous branch, which is optimized jointly with the discrete tokens during Stage II, ensuring effective integration without conflict.

Q3: The experimental setup has several questionable aspects. First, the paper fails to provide an evaluation of the first-stage model's learning performance.

A3: We report the performance of Stage I on AMASS in Table 6 in Supplementary Material. For single-frame, single-view evaluation on Human3.6M GT, Stage I achieves 40.2 MPJPE. It is important to note that the entire VQ-VAE contains only 5.54M parameters, which is quite small. And as expected, the discrete representation yields lower regression accuracy compared to continuous features. We will include those results in the main paper.

Q4: Moreover, since the performance improvement is marginal, the authors should further validate—through visualization or other means—whether the proposed method genuinely addresses the challenges in camera parameter-free settings.

A4: (1) Non-Marginal Performance Gains on Saturated Benchmarks.

(1.1) Saturation of Human3.6M and Stronger Results on 3DHP

We would like to clarify that Human3.6M is already saturated in terms of performance, and Human3.6m contains some inaccurate annotations[4], making further improvement increasingly difficult. Unlike Human3.6M, which is a lab dataset, 3DHP includes both indoor and outdoor scenes, offering higher diversity and better reflecting real-world scenarios. Our method shows stronger performance on 3DHP without extra training data, with a 2.03 mm MPJPE reduction and a 4.24 AUC increase over the second-best method, FusionFormer and SGraFormer.

(1.2) Addressing the Overlooked Real-World Noise: A Novel Use of Data-Driven Discrete Priors for Robustness.

Our core motivation is as follows: continuous representations are well-suited for regression tasks but are vulnerable to noise, while discrete representations tend to be noise-resilient but suffer in precision. Our method aims to leverage both to improve robustness without sacrificing accuracy. As shown in Figure 3 of the supplementary material, using DCSA and the discrete codebook improves noise robustness significantly, achieving up to 6% (5 mm MPJPE) improvement under large noise, which is far from marginal.

(1.3) Steady Performance Gains Through Progressive Discrete Prior Quality

Table 4 (a)(c) shows consistent and stage-wise improvements when increasing the quality of the VQ-VAE prior (see L304–307, L322–324). These improvements are achieved without adding any model parameters, solely by enhancing the quality of the UniCodebook, which directly correlates the quality of discrete prior with model robustness and validating the effectiveness of our method, especially under noisy inputs.

(2) Enhancing Noise Robustness in Camera Parameter-Free Settings Which Is Important for Real-World Deployment

The concern you raised is exactly at the core of our work! In real-world camera parameter-free settings, noise is a big challenge. The core problem our method aims to address is noise sensitivity under uncalibrated conditions. Without camera extrinsics to provide geometric cues, such methods rely more heavily on semantic information from the data to perform multi-view fusion. This makes continuous-feature-based models, especially transformers, more vulnerable to noisy input data. To tackle this, we introduce discrete, data-driven human pose priors to support continuous regression. This improves robustness without sacrificing accuracy. As shown in Figure 3 of the supplementary material, our model consistently outperforms the continuous-only baseline under varying noise strengths and view counts, with up to a 5 mm (6.25%) improvement under the highest noise level. This confirms that our method effectively mitigates the noise issue in camera-free settings.

Q5: What are the differences in experimental settings between the first row of Table 4(a) and the baseline in Table 4(b) that lead to the performance discrepancy?

A5: Table 4 (a) T=1, Table 4 (b) T=27. Unless otherwise specified, all results in the paper default to T = 27. We apologize for not making this clear in the manuscript.

Q6: The title format of Reference 32 is incorrect.

A6: We have corrected it.

Q7: The authors should discuss how the inference efficiency compares with FusionFormer.

A7: We report the results in the table below, which shows that the computational cost of our model grows approximately linearly with the number of views and frames, consistent with Supplementary Table 4. Note that the reported parameter count is smaller than that in Figure 4 as we mistakenly included redundant components such as the 2D decoder, 3D encoder, and 3D decoder of VQVAE.

Dear Reviewer 63UD,

The Reviewer–Author Discussion period will end soon.

Please review the authors’ rebuttal and share any comments or feedback so that the authors can provide additional responses if necessary.

AC

Q1: You claim that your model is not affected by "predefined geometric relationships" (L283), yet you include view-dependent positional encoding  (L199). This appears to entangle positional relationship into the estimation process. I would like to see if your method is truly independent from camera geometric relationships, for example, by swapping or reducing input views only during evaluation.

A1: (1) We acknowledge that our model does exhibit some sensitivity to view biases present in the data. However, this issue is not unique to our method and can also be observed in other SOTA approaches such as FusionFormer. Notably, our method demonstrates stronger generalization—even when trained using only 4-view data or a mixture of 3- and 4-view configurations—outperforming FusionFormer under the same conditions.

We attribute this robustness to the use of discrete codebook features. Continuous representations are typically learned from multi-view data, where the same features must simultaneously encode both pose semantics and inter-view interactions. This entanglement can make them more sensitive to changing of view numbers. In contrast, our discrete features are learned from single-view data, and their integration into the model provides additional information that is less entangled with camera-specific biases.

(2) As attention mechanism is theoretically permutation-invariant, we suspect that this view sensitivity primarily stems from the use of positional encodings (PE). In ablation experiments where we removed PE from the model, we observed consistent performance improvements when shuffled views only when testing, suggesting that PE may inadvertently introduce view-dependent artifacts. But it still can not generalize to less views which is not observed in training. We believe that using reduce views when training would increate the performance.

Q2: You simply add the output of spatial MHSA and DCSA (Fig. 2, Eq. 9). Why wasn't a weighted sum used, as done in L236 for two-path integration? The model benefits from adaptively considering continuous and discrete features depending on noise levels. Employing a weighted sum to allow the model to prioritize specific feature could lead to improved estimation.

A2: Good suggestions! We apply weighted sum but obtained results similar to the original method when T=1. One possible explanation is that the model implicitly learns to balance the contributions from the two branches. Therefore, a straightforward addition already captures most of the benefits that a more complex weighted sum could provide, without adding extra parameters or complexity.

Q3: Currently, since $head_i$ are the sum of $V_i$ with the weights derived from $softmax(QK)$ (which sums to one) for both of spatial MHSA and DCSA, this may hinder effective feature focusing.

A3: Excellent observation! In the original design, continuous queries interacted only with continuous keys/values, while a separate set of continuous queries interacted with discrete keys/values. However, this setup enforces that a fixed number of continuous and discrete tokens are always involved in the output, regardless of their actual feature quality or relevance in the current context. This can be limiting, since the relative importance of continuous and discrete features may vary depending on the context. To address this, we redesign the attention mechanism by unifying the two attention branches into a single one: continuous queries now attend to the concatenation of both continuous and discrete keys/values. This not only simplifies the architecture but also allows the model to dynamically and adaptively focus on the most relevant combination of continuous and discrete tokens, without being constrained by fixed ratios.

However, preliminary experiments show that this unified attention approach does not improve performance—in fact, it slightly degrades it (MPJPE increases from 26.34 to 26.51). This might because that the two-path design inherently reduces coupling between branches, allowing each to specialize: one branch focuses on modeling continuous spatial pose relationships, while the other learns affinity to discrete tokens. This separation facilitates learning complementary features independently. In contrast, the single-path design requires the model to simultaneously capture both continuous spatial relationships and discrete token affinities within the same attention mechanism. This increased learning burden can cause feature interference and make it harder for the model to effectively disentangle and utilize the distinct modalities, leading to degraded performance.

Q1: Missing baselines, especially for the monocular setting, only baselines are listed; it should also evaluate this method's monocular performance on different datasets.

A1: (1) Regarding monocular performance, we are currently training our temporal model on sequences of 243 frames, which requires approximately 3 days to converge. Once training is complete, we will report monocular results and analyze the effect of the discrete codebook on noise robustness in the temporal monocular setting within the rebuttal period.

(2) Our method is primarily designed for the multi-view and temporal setting (v=4, f=27), which is a meaningful and practical setup. Multi-view configurations are as important as monocular ones, as they better reflect real-world applications such as motion capture studios and multi-camera systems, where complementary viewpoints enhance robustness and accuracy.

(3) Notably, recent state-of-the-art multi-view, uncalibrated methods like SGraFormer and SVTFormer also do not report monocular performance, as their architectures and training are specialized for multi-view input.

Q2:The fixed-size UniCodebook may have difficulty with highly diverse or unusual poses that were not included in the training. This creates a representational bottleneck that might limit generalization. Is the codebook generalizable to unseen poses?

A2: (1) We agree that a fixed-size UniCodebook may not fully cover all highly diverse or unusual poses. However, continuous regression models face the same limitation.

(2) Noise in real-world scenarios can also hinder generalization, so improving noise robustness directly contributes to better generalization. As shown in Figure 3 of the supplementary material, when noise intensity increases, the poses become more unseen for the UniCodebook (as traing pose is high quality from AMASS). In this case, our model are more robust than the continuous baseline.

(3) Better data utilization also improves real-world generalization. The UniCodebook is trained on high-quality poses from AMASS and supports both paired and unpaired inputs, allowing broader data utilization than Stage II . This means that Stage I, based on the UniCodebook, is less likely to become a bottleneck—especially considering that the continuous model in Stage II is both more sensitive to noise and trained on less data.

(4) To address generalization concerns, we propose DCSA to effectively combine the complementary strengths of discrete and continuous representations. This hybrid design ensures that our model does not suffer from more domain generalization issues than fully continuous alternatives.

Q3: The paper does not explain how the codebook would adjust to new pose domains. Needing complete retraining could be impractical in real-world situations.

A3: (1) As shown in Section F of the supplementary material, our model encounters challenges primarily when transferring across different pose definitions, rather than across different pose distribution domains.

In our work, a pose is defined by a set of semantically consistent keypoints representing major human joints, their 3D coordinates, and the skeletal connectivity between them. For example, in the Human3.6M dataset, each pose comprises 17 keypoints—including the head, neck, shoulders, elbows, wrists, hips, knees, and ankles—structured as a kinematic chain.

It is important to distinguish this from domain shifts in pose distributions, which are naturally involved in our main experiments—for example, transferring from AMASS to Human3.6M and other datasets. We highlight this distinction to clarify a potential misunderstanding: the reviewer’s concern appears to conflate variation in pose definitions (i.e., joint types and connections) with shifts in pose distribution (i.e., statistical variation within a fixed definition).

(2) As shown in Section F, we propose a potential solution by training a format-agnostic discrete prior. However, we currently do not have a concrete approach for fine-grained adaptation to new pose definitions, as this is not the primary focus of our work. This limitation is common to all models that represent poses as 1D sequences, and it is not specific to the use of VQ-VAE.

(3) Moreover, many joint formats can be easily converted, especially when migrating from models with more joints to those with fewer joints. Thus, retraining is often unnecessary unless the joint sets are truly incompatible. In real applications, models typically rely on mainstream 2D detectors, which use fixed and widely adopted pose definitions. For example, OpenPose, Detectron2 and MMPose all support COCO format keypoint.

Q4: No qualitative evaluation and comparison with baselines are shown in the main paper, which makes it very difficult to show the improvement convincingly.

A4: We include qualitative evaluations and comparisons with baselines in Figure 7 and the accompanying videos in the Supplementary Materials. In the final version, we will move part of these qualitative results into the main paper and provide more detailed analysis to better highlight the improvements achieved by our method.

Q5: Assumptions of a multiview setting limit the application of this method. How to apply it on real scenarios, such as moving cameras or video streams?

A5: (1) As noted in our response to Q1, our model can be directly applied to monocular settings by setting views=1, while still maintaining robustness to noisy 2D pose inputs (see Figure 3 in Supplementary Materials). Although our main experiments use a multiview setup, we share the same concern as the reviewer regarding real-world deployment, where noisy 2D poses are common, especially in moving camera scenarios.

(2) Uncalibrated multiview scenarios are susceptible to noisy 2D poses and thus present a realistic and challenging setting. Without camera extrinsics to provide geometric cues, such methods rely more heavily on semantic information for multi-view fusion. This makes continuous-feature-based models more vulnerable to input noise. Our study on robustness under uncalibrated multiview conditions is therefore also helpful to handle temporal jitter caused by moving cameras, offering practical insights for deployment in dynamic real-world environments.

Q1: My biggest concern for this work is that it seems all its proposed modules only have marginal influence on the model performance. From the results in Table 4 (a), (b), (c), I do not think there any significant improvement resulted from the "new ideas", e.g., the MPJPE improvements are all less then 3% compared to the baseline. For me, they could just be random noises.

A1: (1) Saturation of Human3.6M and Stronger Results on 3DHP

We would like to clarify that Human3.6M is already saturated in terms of performance, and Human3.6m contains some inaccurate annotations[4], making further improvement increasingly difficult. Unlike Human3.6M, which is a lab dataset, 3DHP includes both indoor and outdoor scenes, offering higher diversity and better reflecting real-world scenarios. Our method shows stronger performance on 3DHP without extra training data, with a 2.03 mm MPJPE reduction and a 4.24 AUC increase over the second-best method, FusionFormer and SGraFormer.

(2) Addressing the Overlooked Real-World Noise: A Novel Use of Data-Driven Discrete Priors for Robustness

Our core motivation is as follows: continuous representations are well-suited for regression tasks but are vulnerable to noise, while discrete representations tend to be noise-resilient but suffer in precision. Our method aims to leverage both to improve robustness without sacrificing accuracy. As shown in Figure 3 of the supplementary material, using DCSA and the discrete codebook improves noise robustness significantly, achieving up to 6% (5 mm MPJPE) improvement under large noise, which is far from marginal.

(3) Steady Performance Gains Through Progressive Discrete Prior Quality

Table 4 (a)(c) shows consistent and stage-wise improvements when increasing the quality of the VQ-VAE prior (see L304–307, L322–324). These improvements are achieved without adding any model parameters, solely by enhancing the quality of the UniCodebook, which directly correlates the quality of discrete prior with model robustness and validating the effectiveness of our method, especially under noisy inputs.

Q2: When comparing with state-of-the-art, e.g., Table 1, the author should compare against latest methods instead of out-of-date ones. For example, for Human3.6M, I recommend the authors to compare methods listed in paperwithcode.

A2: (1) Thank you for the suggestion. We have compared our method against all latest open-source available models under the uncalibrated setting (e.g., SVTFormer (AAAI 2025), SGraFormer (AAAI 2024), FusionFormer(AAAI 2024)).

(2) We followed your suggestion and compared our method with the top-3 approaches listed on Papers With Code. Methods ranked lower show performance significantly inferior to ours.

(2.1) In that leaderboad, the top-ranked method LMT R152[1] operates under a different experimental setup by generating 3D poses directly from images and leveraging SMPL prior, which is not comparable to our setting that uses noisy 2D pose inputs. As noted in the main paper (Lines 247–249), the 2D poses predicted by CPN contain significant noise, making it challenging to infer correct 3D poses from erroneous 2D joints. In contrast, image-based methods inherently have access to more accurate 2D information.

(2.2) The second-ranked method, Geometry-Biased Transformer[2], relies on calibrated setups and uses HRNet for 2D pose sequences. Since HRNet’s 2D pose estimation is significantly more accurate than CPN’s, it benefits from better inputs. Despite using noisier inputs and lacking camera calibration, our method achieves comparable results (MPJPE: 26.0 mm VS 26.0 mm), highlighting the robustness and strength of our approach. This also suggests the dataset performance is approaching saturation.

(2.3) Moreover, our method substantially outperforms the third-ranked Epipolar Transformer[3] (26.0 mm vs. 26.9 mm). We will include these methods in the final version and clarify the differences in experiment settings between our method and theirs.

Q3: Some technical details are not clear. For example, in L177, how are the $N$ embeddings computed from the $J$ joints? Why choose $N$ to be 63? It looks like a magic number to me.

A3: We apologize for not explaining the origin of this number. Human3.6M contains 17 joints, each with 3D coordinates, totaling 51 dimensions. To increase representation redundancy, we use 63 tokens, roughly corresponding to 21 joints × 3 coordinates. 63 is not a magic number as similar token counts (e.g., 60 or 65) yield similar performance.

Q4: For the VQ-VAE, why not adding temporal modelling? My gut tells me maybe adding temporal modelling will give some performance boost.

A4: (1) Thank you for the insightful suggestion. As have discussed in Section F in Supplementary Material, we fully agree that incorporating temporal modeling can potentially enhance performance. However, we intentionally avoid relying on pose sequences in order to maximize the use of single-pose data, which is significantly more abundant than sequential data.

(2) We also conducted experiments incorporating a temporal prior during codebook training by introducing a loss term that encourages the predicted joint velocity across time to match that of the ground truth if two joints are from one sequence. This indeed led to improvements in VQ-VAE reconstruction across all four codebook strategies. When applied to the full backbone, the final MPJPE remained almost unchanged (26.09 mm).

Q5: For L209-L230, the authors spend a lot of spaces to introduce the basic attention formulation, for which I do not think it's necessary. TBH, although it does make the paper look more "mathematical" at the first glance, it does not bring any real values to the readers given the attention and transformers are widely used today. Please do not expect reviewers are fool enough to give accept just because there are some "nice looking" formulations for well-known basic stuffs

A5: Thanks! We will simplify those redundant explanation of the basic attention mechanism and focus directly on the core ideas and contributions of our method.

[1] Learnable Human Mesh Triangulation for 3D Human Pose and Shape Estimation. WACV 2023

[2] Geometry-Biased Transformer for Robust Multi-View 3D Human Pose Reconstruction. FG 2024

[3] Epipolar Transformers. CVPR 2020

[4] Learnable Triangulation of Human Pose. ICCV 2019

Dear Reviewer imRL,

The Reviewer–Author Discussion period will end soon.

Please review the authors’ rebuttal and share any comments or feedback so that the authors can provide additional responses if necessary.

AC