RAPTR: Radar-based 3D Pose Estimation using Transformer

Summary & Strengths

Response
We thank the reviewer for recognizing our motivation to use weak supervision for 3D radar perception—an unexplored direction in this domain. We also appreciate your positive feedback on our loss function ablation (Fig. 7) and experiments (Table 1). As noted, our work has implications for future large-scale dataset development. We will release full training/evaluation code and pretrained checkpoints to support such efforts.

Weaknesses

• ...However, the effectiveness of the proposed architecture remains unclear. A close look at the ablation study suggests that the performance gains are primarily due to the two-stage structure of the pose decoder and joint decoder.

Response
Thanks for raising this concern. While we do not claim the two-stage decoder as a core contribution, we deliberately adopt PETR's two-stage design as a strong, empirically validated baseline for our purpose - 3D radar perception under weak supervision.

With additional ablation studay, we summarize our contribution in two folds (as you pointed out): (1) the structural design of loss functions under weak supervision; and (2) the architectural modifications including cross-view (CV) radar encoder and pseudo-3D deformable attention.

First, our structured design of the loss functions—particularly the use of 3D Gravity Loss and 3D Template Loss across the two decoder stages—is key to the substantial performance gains. Along with the suggestion from Reviewer vQxv, Table 1 below provides a complete picture of the significance of each loss function and the decoder stage where it is applied.

• The 3D template prior at the first-stage decoder helps localize initial pose estimates near the correct spatial region (e.g., close to the 3D bounding box center) and promotes human-like keypoint configurations, avoiding degenerate or implausible shapes.
• The second-stage joint decoder refines initial pose estimates by projecting 3D keypoints to the 2D image plane and minimizing 2D keypoint loss (K2D), which leverages the fine granularity of 2D annotations. However, 2D keypoint loss alone cannot constrain absolute scale or depth due to depth ambiguity—where a distant large object may appear identical in 2D to a closer small object. To resolve this, we combine 2D keypoint loss with 3D Gravity Loss, allowing the model to learn depth- and scale-aware corrections during pose refinement.

Table 1: Effect of our structural loss function on RAPTR

Results with ± values are newly added and report mean ± std from at least 3 runs with different random seeds.

symbol are defined below Table 1 of the response to Review bgMt.

Our structural loss design is also compatible with PETR. As shown in Table 2 below, The original PETR with 2D keypoint loss alone completely fails to generalize to 3D radar perception, with MPJPE over 370 cm due to depth ambiguity. Adding gravity and template losses reduces the error to under 20 cm—a 10× improvement.

Table 2: Effect of our structural loss function on PETR

Second, pseudo-3D attention works better with stronger, view-consistent radar features—such as those provided by the cross-view (CV) encoder. Thanks to Reviewer bgMt's suggestion to evaluate +(1)+(3), we include the result in the third row of Table 3 below, alongside Table 3 of the main paper (we re-run +(1) and +(1)+(2)+(3) with 3+ random seeds and report mean ± std for more rigorous evaluation). It is observed that using either the cross-view encoder (CV, 3rd row) or pseudo-3D attention (3D, 2nd row) alone results in slightly worse performance than the base two-stage setup (first row). However, when pseudo-3D attention is paired with the cross-view encoder (last row), it achieves the best performance. This suggests that the effectiveness of pseudo-3D attention depends heavily on the quality and consistency of the underlying radar features, which are better provided by the cross-view encoder than the separate-view encoder. We provide the following justification by looking into the pseudo-3D deformable attention in Fig. 4 of the main paper.

Table 3: Ablation Study on RAPTR architectural modules (Encoder, Decoder, Attention)

Weaknesses

• As shown in Figure 9 of the supplementary material, ...This architectural change appears to be incremental...a comparison regarding how much the switch from QRFPose to RAPTR improves model size (memory usage) or inference time is insufficient.

Response
As you pointed out, Figure 9 in the supplementary material illustrates key differences between the QRFPose and RAPTR:

• decoupled 2D offsets in QRFPose versus 3D offsets in RAPTR. While conceptually simple, our 3D offsets better align with radar physics—ensuring that the same human keypoint projects to consistent depths across multiple radar views. In contrast, the 2D offsets in QRFPose may select unrelated features at different depths, limiting cross-view consistency.
• decoupled attention weights in QRFPose versus integrated attention weights in RAPTR to aggregate the view-vise features or cross-view radar features.

In terms of memory footprint and inference time, we don't expect big differences between the 2D and pseudo-3D deformable attention as pseudo-3D saves one parameter for one offset prediction.

• ...it would be necessary to apply the {Pose + Joint} decoder structure to QRFPose and evaluate the resulting performance.

Response
Suggestion has been taken. As shown in Table 4, the second row reports the MPJPE for the {Pose + Joint} decoder structure added to QRFPose with 2D keypoint and gravity losses, showing significant gains over the base one-stage QRFPose (first row).

This {Pose + Joint} structure is further improved by applying our structured loss design (e.g., 3D template loss at the first-stage decoder), as shown in the third row.

With additional architectural enhancements—such as the cross-view (CV) encoder and pseudo-3D deformable attention (3D)—the last row achieves the best performance across all architecture and loss configurations on the HIBER (MULTI) dataset.

Table 4: QRFPose versus RAPTR (with 3D template loss)

• (Minor) ...the paper does not describe how the template pose is calculated for each subject...

Response
Thanks for pointing out the calculation of 3D template. In implementing our 3D Template Loss, we used a standard standing pose of a 1.6 m-tall person (see Fig. 3 in the main paper). More specifically, we used the ground-truth 3D keypoint labels at one frame to construct the template (minus its center position).

Related to your comment, we also conducted an ablation study evaluating the impact of different 3D templates at the first-stage pose decoder (in response to a comment from Reviewer WQeC,). Please refer to Table 4 in our response to Reviewer WQeC.

Summary & Strengths

Response
We thank the reviewer for recognizing our core motivation—advancing 3D radar perception under weak supervision, which we believe is the first to explore this direction. We appreciate the positive feedback on our 3D loss design and experiments. To support reproducibility, we will release full code and pretrained checkpoints.

Weaknesses

• Limited Methodological Novelty:... While practical, this reduces its originality, especially since most contributions are engineering extensions.

Response
We thank the reviewer for raising this concern, which allows us to clarify our core contributions more clearly below. With additional ablation studay, we summarize our contribution in two folds: (1) the structural design of loss functions under weak supervision; and (2) the architectural modifications including cross-view (CV) radar encoder and pseudo-3D deformable attention.

First, our structured design of the loss functions—particularly the use of 3D Gravity Loss and 3D Template Loss across the two decoder stages—is key to the substantial performance gains. Along with the suggestion from Reviewer vQxv, Table 1 below provides a complete picture of the significance of each loss function and the decoder stage where it is applied.

• The 3D template prior at the first-stage decoder helps localize initial pose estimates near the correct spatial region (e.g., close to the 3D bounding box center) and promotes human-like keypoint configurations, avoiding degenerate or implausible shapes.
• The second-stage joint decoder refines initial pose estimates by projecting 3D keypoints to the 2D image plane and minimizing 2D keypoint loss (K2D), which leverages the fine granularity of 2D annotations. However, 2D keypoint loss alone cannot constrain absolute scale or depth due to depth ambiguity—where a distant large object may appear identical in 2D to a closer small object. To resolve this, we combine 2D keypoint loss with 3D Gravity Loss, allowing the model to learn depth- and scale-aware corrections during pose refinement.

Table 1: Effect of our structural loss function on RAPTR

Results with ± values are newly added and report mean ± std from at least 3 runs with different random seeds.

symbol are defined below Table 1 of the response to Review bgMt.

Second, our structural loss design is also compatible with PETR. As shown in Table 2 (new results), PETR with only 2D keypoint loss completely fails to generalize to 3D radar, with MPJPE over 370 cm due to depth ambiguity. Adding gravity and template losses reduces the error to under 20 cm—a 10× improvement.

Table 2: Effect of our structural loss function on PETR

Third, pseudo-3D attention works better with stronger, view-consistent radar features—such as those provided by the cross-view (CV) encoder. Thanks to Reviewer bgMt's suggestion to evaluate +(1)+(3), we include the result in the third row of Table 3 below, alongside Table 3 from the main paper (we re-run +(1) and +(1)+(2)+(3) with 3+ random seeds and report mean ± std for more rigorous evaluation). It is observed that using either the cross-view encoder (CV, 3rd row) or pseudo-3D attention (3D, 2nd row) alone results in slightly worse performance than the base two-stage setup (first row). However, when pseudo-3D attention is paired with the cross-view encoder (last row), it achieves the best performance. This suggests that the effectiveness of pseudo-3D attention depends heavily on the quality and consistency of the underlying radar features, which are better provided by the cross-view encoder than the separate-view encoder. We provide the following justification by looking into the pseudo-3D deformable attention in Fig. 4 of the main paper.

Table 3: Ablation Study on RAPTR architectural modules (Encoder, Decoder, Attention)

Weaknesses

• Lack of effective modeling for occlusion and multi-person interference:...

Response
For occlusion, our preliminary investigation shows that RAPTR with weak supervision has some capability in recovering 3D poses even when some body parts are occluded (e.g., by furniture or other people). This is particularly observed on the MMVR dataset where the subjects are often partially occluded by furniture. We observed that:

• The 2D keypoint labels—generated by pre-trained vision models—often extrapolate occluded keypoints accurately in the image pixel domain.
• Our weakly supervised framework lifts these 2D keypoints to 3D using the combination of 2D keypoint loss, 3D template loss, and 3D gravity loss.

These results appear to suggest that RAPTR can handle small-to-moderate occlusion via strong image-based 2D supervision loss and structured 3D regularization (template + gravity loss). However, due to the absence of ground-truth 3D keypoints in MMVR, we cannot provide a quantitative analysis under varying occlusion levels.

For multi-person interference, this is a good point. In both the HIBER and MMVR datasets, we did not observe strong interference effects, potentially due to the absorption of radar waveforms at 77 GHz at human body and clothing, and multiple-bouncing attenuated signals, resulting in weak ghost targets. One possible direction is to incorporate temporal dynamics of human motion. We leave this as a future topic.

Weaknesses

• Rigid prior assumptions in the 3D template: The reliance on a fixed pose template may reduce generalization to out-of-distribution poses or body shapes.

Response
Thanks for the insightful comment on the 3D template prior. In implementing our 3D Template Loss, we used a standard standing pose of a 1.6 m-tall person (see Fig. 3 in the main paper). To address this comment, we conducted an ablation study evaluating the impact of different 3D templates at the first-stage pose decoder. Results are shown in Table 4 below. Specifically, we experimented with:

• A standing pose scaled by two factors: 0.5× and 1×
• A sitting pose of a 1.6 m-tall person
• A learned scaling factor applied to the standard standing pose

Table 4 suggests that the choice of 3D template has some minor impact on final MPJPE, likely due to the refinement capability of the second-stage decoder, as long as the first-stage decoder generates a reasonable, human-like initial pose.

Table 4: The effect of 3D Template

Questions

• ...Can ablations with more diverse attention mechanisms help validate this choice?

Response
We refer the reviewer to Table 3 above, which evaluates the effect of RAPTR's architectural modules—including the attention mechanisms (2D vs. pseudo-3D)—in response to your concern. We appreciate your suggestion and will consider more attention mechnisms in the update paper.

Summary & Strengths

• The core innovation of the proposed method is that it uses only low-cost, easily accessible 3D BBoxes and 2D keypoints for supervision, instead of relying on expensive and difficult-to-collect 3D keypoint labels. The whole framework is built on PETR to output initial 3D pose and refined 3D pose, which is supervised by the proposed 3D template loss, and 2D keypoint and 3D gravity losses, respectively...
• The proposed method significantly reduces data annotation costs...allows the model to be trained on larger-scale and more diverse scenarios and enhances the scalability of the proposed method.

Response
We sincerely thank the reviewer for the positive feedback and recognition of our core motivation—advancing 3D radar perception under weak supervision—which, to the best of our knowledge, is the first work to explore this direction in the radar domain. We appreciate your acknowledgment of the broader impact of our approach, particularly our effort to enable scalable training and deployment using inexpensive labels, which hopefully encourages more extensive radar data collection without the need for costly 3D annotations. We will release the full training and evaluation code, along with the pretrained model checkpoints, to support reproducibility and future open research in 3D radar perception.

Weakness
*The methodological novelty of the proposed method is somewhat limited. The overall framework of RAPTR is mainly based on PETR. Its primary modification is to "lift" the 2D deformable attention mechanism into a "pseudo-3D" space. However, as shown in Table 3, pseudo-3D deformable attention is harmful to the model performance, which cannot show the effectiveness of the proposed component. What is the performance of +(1)+(3)?

Response
Thanks for raising this concern, which allows us to clarify our core contributions more clearly. We also conducted additional ablation studies on the HIBER dataset, evaluating a comprehensive set of configurations involving architectural modules and loss functions with random seeds, including your suggested combination of +(1)+(3).

First, while we do not claim the two-stage decoder as a core contribution, we deliberately adopt PETR's two-stage design as a strong, empirically validated baseline for our purpose. Rather than reinventing the wheel for the sake of novelty, we plug-and-play PETR's backbone and enable us to focus on advancing 3D radar perception under weak supervision.

Second, PETR with only 2D keypoint loss completely fails to generalize to 3D radar perception, as evidenced by the first row of Table 1 below, which shows drastically worse performance—with an error exceeding 3 meters. The core issue is depth or depth-scale ambiguity, a limitation well illustrated by the behavior of a pinhole camera or flashlight projection: A larger object that is farther away can appear identical in 2D to a smaller object that is closer. In such cases, 2D keypoint supervision alone provides no constraint on absolute scale or depth, making 3D recovery fundamentally under-constrained—especially in weakly supervised settings.

Table 1: Effect of our structural loss function on PETR

Results with ± values are newly added and report mean ± std from at least 3 runs with different random seeds.

Symbol Definitions

• Decoder:
  1S = one-stage, 2S = two-stage

• Encoder:
  SV = separate-view encoder, CV = cross-view encoder

• Attn (Attention):
  2D = 2D deformable attention, 3D = pseudo-3D deformable attention

• Loss Terms:
  T3D = 3D Template loss (weak-supervsion label)
  G3D = 3D Gravity loss (weak-supervsion label)
  K2D = 2D Keypoint loss (weak-supervsion label)

Third, it is our structured design of the loss functions—particularly the use of 3D Gravity Loss and 3D Template Loss across the two decoder stages—that leads to substantial performance gains. As shown in the second and third rows of Table 1, these additions reduce the MPJPE of PETR from over 370 cm to under 20 cm—a more than 10× improvement. We also observe in Table 2 that our structured loss function significantly outperforms other loss combinations, further validating its effectiveness on RAPTR.

Table 2: Effect of our structural loss function on RAPTR

Fourth, Pseudo-3D attention works better with stronger, view-consistent radar features—such as those provided by the cross-view (CV) encoder. Thanks to your suggestion to evaluate +(1)+(3), we include the result in the third row of Table 3 below, alongside the original Table 3 from the main paper (we re-run +(1) and +(1)+(2)+(3) with 3+ random seeds and report mean ± std for more rigorous evaluation). It is observed that using either the cross-view encoder (CV, 3rd row) or pseudo-3D attention (3D, 2nd row) alone results in slightly worse performance than the base two-stage setup (first row). However, when pseudo-3D attention is paired with the cross-view encoder (last row), it achieves the best performance at the two HIBER data splits (WALK, MULTI). This suggests that the effectiveness of pseudo-3D attention depends heavily on the quality and consistency of the underlying radar features, which are better provided by the cross-view encoder than the separate-view encoder. We provide the following justification by looking into the pseudo-3D deformable attention in Fig. 4 of the main paper.

• Pseudo-3D attention predicts 3D spatial offsets—unlike the decoupled 2D offsets in QRFPose—to align and extract spatially consistent features across radar views.
• These 3D offsets are projected into each radar view to determine where features should be sampled.
• The sampled features from different views are then aggregated using normalized attention weights to form the pseudo-3D deformable attention output.
• For this mechanism to function effectively, the underlying radar features need to be spatially coherent and consistent across views—a property better provided by the cross-view (CV) encoder.
• In contrast, when pseudo-3D attention is combined with the separate-view (SV) encoder, the lack of shared multi-view context may limit its ability to extract reliable 3D features, leading to less powerful aggregated multi-view radar feature at the pseudo-3D deformable attention output.

Table 3: Ablation Study on RAPTR architectural modules (Encoder, Decoder, Attention)

Summary

We thank Reviewer bgMt for the thoughtful and constructive feedback. Your positive evaluation of our work's quality, clarity, significance, and originality (all rated 4) affirms our core contributions to weakly supervised 3D radar perception. We hope that the added clarification on pseudo-3D deformable attention, additional ablations (including your suggested +(1)+(3)), and new results with random seeds further strengthen the impact and completeness of our submission.

Summary & Strengths

Response
We thank the reviewer for recognizing our core motivation—advancing 3D radar perception with weak supervision—which, to our knowledge, is the first to explore this direction. We appreciate your summary of our contributions, including pseudo-3D attention, structured loss design, and evaluations on two large indoor radar datasets. We will release full code and pretrained checkpoints to support future research.

Weaknesses & Questions

• Elaborate on the intuition of using gravity loss and conduct the ablation...with and without gravity loss.

Response
First, we highlight that the motivation of using a hierarchical and structured loss function:

• (a) a 3D Template Loss (T3D) at the first-stage pose decoder, using coarse 3D bounding boxes and a human keypoint template to guide initial 3D pose estimates, and
• (b) a combination of 2D Keypoint Loss (K2D) and 3D Gravity Loss (G3D) at the second-stage joint decoder to refine the pose predictions.

Second, to clarify the role of gravity loss, we follow your suggestion and include an ablation (last row of Table 1). The second-stage joint decoder refines poses using 2D keypoint loss (K2D), but K2D alone cannot constrain depth due to depth ambiguity (a limitation illustrated by the behavior of a pinhole camera or flashlight projection: A larger object that is farther away can appear identical in 2D to a smaller object that is closer). Adding 3D gravity loss (G3D) regularizes depth and scale. Comparing the 3rd and last rows of Table 1 shows that removing G3D leads to over 50% performance drop (e.g., 18.99 cm → 30.06 cm on MULTI), confirming the necessity of using G3D in the second-stage decoder.

Third, we justify the template loss at the first-stage decoder using Table 1. Comparing rows 1–3 shows that adding 3D shape priors reduces MPJPE from 50.96 cm and 27.36 cm to 18.99 cm. The template not only localizes poses near the 3D box center but also encourages human-like keypoint structures, avoiding degenerated or implausible shapes produced without this human shape prior.

Table 1: RAPTR with Weak Supervision (2D Keypoint Loss (K2D), 3D Gravity Loss (G3D), and 3D Template Loss (T3D))

Results with ± values are newly added and report mean ± std from at least 3 runs with different random seeds.

symbol are defined below Table 1 of the response to Review bgMt.

Weaknesses

• Conduct the experiment of training the method with full supervision (3D keypoints) to show the upper bound performance for comparison.

Response
Suggestion has been taken. We have included the evaluation of RAPTR under full supervision (using 3D keypoint loss, K3D) in Table 2 below. We agree that this setting provides a meaningful upper bound on achievable performance, and allows us to quantify the performance gap between weak and full supervision.

Under full supervision, RAPTR achieves an MPJPE of 8.93 cm, whereas our best weakly supervised variant—leveraging 3D Template Loss (T3D) and 3D Gravity Loss (G3D)—achieves 18.99 cm. The weakly supervised model yields about a loss of ~10 cm in MPJPE, compared with the full supervision result. For completeness, we also include additional weakly supervised variants from Table 1 to highlight how each loss term contributes to final performance.

Table 2: RAPTR with Weak and Full Supervision

Symbol Definitions

• Loss Terms:
  T3D = 3D Template loss (weak-supervsion label)
  G3D = 3D Gravity loss (weak-supervsion label)
  K2D = 2D Keypoint loss (weak-supervsion label)
  K3D = 3D Keypoint loss (Full-supervsion label)

Weaknesses

• ...The authors should explain why pseudo-3D deformable attention mechanism becomes a contribution regarding performance and complexity...

Response
Thanks for this insighful comment. Pseudo-3D attention works better with stronger, view-consistent radar features—such as those provided by the cross-view (CV) encoder. Thanks to your suggestion to evaluate +(1)+(3), we include the result in the third row of Table 3 below, alongside the original Table 3 from the main paper (we re-run +(1) and +(1)+(2)+(3) with 3+ random seeds and report mean ± std for more rigorous evaluation). It is observed that using either the cross-view encoder (CV, 3rd row) or pseudo-3D attention (3D, 2nd row) alone results in slightly worse performance than the base two-stage setup (first row). However, when pseudo-3D attention is paired with the cross-view encoder (last row), it achieves the best performance. This suggests that the effectiveness of pseudo-3D attention depends heavily on the quality and consistency of the underlying radar features, which are better provided by the cross-view encoder than the separate-view encoder. We provide the following justification by looking into the pseudo-3D deformable attention in Fig. 4 of the main paper.

• Pseudo-3D attention predicts 3D spatial offsets—unlike the decoupled 2D offsets in QRFPose—to align and extract spatially consistent features across radar views.
• These 3D offsets are projected into each radar view to determine where features should be sampled.
• The sampled features from different views are then aggregated using normalized attention weights to form the pseudo-3D deformable attention output.
• For this mechanism to function effectively, the underlying radar features need to be spatially coherent and consistent across views—a property better provided by the cross-view (CV) encoder.
• In contrast, when pseudo-3D attention is combined with the separate-view (SV) encoder, the lack of shared multi-view context may limit its ability to extract reliable 3D features, leading to less powerful aggregated multi-view radar feature at the pseudo-3D deformable attention output.

In terms of complexity, we provide such an analysis of RAPTER in Appendix A. As shown in Fig. 9 of Supplemental, we don't expect big differences between the 2D and pseudo-3D deformable attention as pseudo-3D saves one parameter for one offset prediction.

Table 3: Ablation Study on RAPTR architectural modules (Encoder, Decoder, Attention)

Minor Weaknesses

• What is SNR in line 91?

Response
SNR denotes Signal-to-Noise Ratio. Raw radar waveforms are often affected by random thermal noise from the hardware and incoherent multi-path reflections. Integrating radar waveforms over the Doppler domain typically improves SNR.

Minor Weaknesses

• In Figure 3, the arrow from Pose Decoder to Joint Decoder is missing, as Joint Decoder takes the output from Pose Decoder as inputs (line 135 - 136).

Response
We will revise the figure in the updated manuscript.

Summary

We appreciate the reviewer’s constructive feedback, which has helped us clarify our technical contributions and strengthen the manuscript. With the added ablation studies, full-supervision upper bound, and improved explanation of the loss design and pseudo-3D attention, we hope that our paper now has clearly demonstrates its quality, clarity, significance, and originality.