Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data

We appreciate the valuable comments aimed at improving our paper. We will revise the draft according to the reviewer's suggestions.

Q1: Limited novelty

Our novelty lies in our approach to solving the body pose estimation problem given doubly sparse data, specifically in how we address the under-constrained problem by measuring and exploiting uncertainty. Previous methods rely heavily on dense hand signals, requiring hand controllers for ego-body pose estimation. Another approach that only uses head poses to estimate the whole body does not utilize hand pose information. Our proposed solution strikes a novel balance between these two approaches, eliminating the need for hand controllers while achieving better results by incorporating a few constraints from detected hand poses.

Q2: The effectiveness of Uncertainty Aware MAE

As the reviewer noted, numerous works have addressed trajectory imputation. However, imputing the hand trajectory itself is not our primary task or the focus of our novelty. Our design of MAE aims to capture uncertainty while imputing the hand trajectory, which differs from other imputation methods such as mask token, in-betweening, and interpolation. Additionally, we introduced a couple of ways to utilize this uncertainty —sampling, dropout, and distribution embedding— in a diffusion model while spatially completing the full body. Given that our newly introduced task of estimating ego-body pose from doubly sparse video data is an under-constrained problem, one of our key motivations is to leverage the "uncertainty" that arises from this under-constrained data.

Q3: Hand pose estimation module and its effectivness

\begin{array}{l|cc} \hline Methods & MPJPE & MPJVE \\ \hline \hline FrankMocap & 16.84 \pm 0.04 & 39.86 \pm 0.05 \\ ACR & 16.69 \pm 0.05 & 40.12 \pm 0.06 \\ Hand Ground Truth & 16.43 \pm 0.02 & 37.49 \pm 0.04 \\ \hline \end{array}

We agree that recently introduced hand models can be used instead of FrankMocap. Before we submitted the paper, we compared the results of FrankMocap with the ground truth 3D hand joint location of Ego-Exo4D and concluded that the effect of hand detectors is not significant. We believe this is because we utilized only the wrist 3D location from the detected hand. Therefore, even though FrankMocap was our initial choice only to prove the concept, we decided not to replace it. We visulized a table of comparison for the different inputs of the hand detector, and it shows that the performance differnece is not significant from ground truth and other hand detector models.

Q4: Training on AMASS benefits Ego-Exo4D performance?

We greatly appreciate the suggestion of applying transfer learning to the Ego-Exo4D dataset. The Ego-Exo4D paper's baseline implementation of EgoEgo [1] took the second-place in the Ego Body Pose Estimation Challenge. This implementation has demonstrated the benefits of training on the AMASS dataset for improving performance on Ego-Exo4D. This method employs a conditional diffusion model, cross-attention for conditioning, and rotary positional embeddings with SLAM pose input. We recognize the potential of this approach and intend to explore its application in our future work on this task.

[1] Li, Jiaman, Karen Liu, and Jiajun Wu. "Ego-body pose estimation via ego-head pose estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Q5: Hand visualization on Figure 3-(c)

We appreciate the reviewer's feedback. While our primary focus is on body pose estimation, rather than hand pose estimation, we understand the importance of comprehensive visualization. Therefore, we can provide visualizations of the canonical hand poses, similar to the approach we used for mesh visualization in Fig 3-(c), to enhance clarity and completeness.

Q6: Writing clarity regarding our constraint in Section 3.3.

Thank you for the valuable feedback. We acknowledge the oversight in Section 3.3, where the title mentions both "imputed hand trajectories and head tracking signal," but the text only references "imputed hand trajectories." We will clarify our method by including a head tracking signal in this section to ensure consistency and completeness.

Q7: Suggestions for the clarify.

Thank you for pointing out the oversights. Following the reviewer's comments, we will make the following revisions to enhance clarity:

1. Add a brief explanation of $x$ in Table 1.

2. Clarify the dimension of the RGB video data.

3. Move $T_w$​ to its first appearance in the text.

These changes will help ensure the information is clear and easy to understand.

We sincerely appreciate the reviewers' insightful comments for our paper.

Q1: FoV modeling result of EgoPoser (ECCV 2024).

\begin{array}{c|cc|cc|cc}\hline Strategies&MPJPE (180°)&MPJVE (180°)&MPJPE (120°)&MPJVE (120°)&MPJPE (90°)&MPJVE (90°) \\\hline \text{EgoPoser}&5.31&39.69&6.07&46.01&6.60&48.25 \\ \text{DSPoser (Ours)}&4.80&22.58&5.28&23.13&5.51&24.19 \\\hline \end{array}

Thank you for bringing EgoPoser (ECCV 2024) to our attention. We were unaware of this work at the time of our submission, and it is encouraging to see parallel interest in ego body pose estimation from intermittent observations. We will ensure to cite this work in the next version of our paper.

Regarding contributions, we understand that EgoPoser's main focus is on preparing training data based on field-of-view (FoV) modeling rather than random masking. While we share an interest in FoV considerations, our work offers distinct algorithmic contributions, including:

1. A multi-stage approach to pose estimation.

2. An uncertainty-aware masked auto-encoder (MAE).

These aspects of our work were recognized by reviewers as innovative contributions to the field. We believe our approach complements the ideas presented in EgoPoser.

Moreover, the experimental data presented in the preceding table demonstrates that our approach surpasses EgoPoser in performance, with notably superior results in scenarios involving a narrow field-of-view (FoV).

Q2: Generalization on Ego-Body Pose Estimation given dense data.

Please see the table in Q3 of the Author Rebuttal.

The quantitative results of our method are influenced by the performance of the VQ-VAE component. Since Table 3 in the main paper is intended to show the versatility of our framework rather than argue our main contribution, we did not conduct an extensive hyper-parameter search to achieve state-of-the-art performance initially. To address this, we performed hyperparameter tuning on the VQ-VAE to mitigate performance loss attributed to this component. Following these adjustments, we observed improved results compared to other methods.

The results demonstrate that while our method may not exhibit the best performance across all metrics, it consistently shows at least the second-best performance for all metrics. A one-to-one comparison with other baselines reveals that our method outperforms them in at least 3 out of 4 metrics. Specifically, we included the Jitter metric to provide a comprehensive analysis of our method, as suggested by Reviewer VMPC. Our method significantly achieves better smoothness, indicating that our approach generates smoother motion close to the ground truth in terms of Jitter.

Q3: Metrics related to computational complexity.

Please see tables in Q2 of Author rebuttal, or the Table B and C in the pdf.

We recognize the importance of including metrics related to computational complexity. While we have roughly provided the inference time in the supplementary material, we agree that a more detailed comparison would be beneficial. Including metrics such as the number of parameters, MACs, and inference time will offer a more comprehensive comparison with previous methods and better illustrate the potential of our approach for real-world applications. We will incorporate these details in the revised version of our paper.

Q4: Consistency between tables

Please refer to table in Q1 of the Author Rebuttal, or Table A in the pdf file.

Thank you for highlighting this inconsistency. Following the reviewer's comment, we plan to update the Tables 1 and 2 to include comparisons with AvatarPoser and AvatarJLM with various imputation method, ensuring that relevant state-of-the-art methods are consistently evaluated throughout the paper. However, due to limited time and resources, we are currently only able to provide baseline results on the AMASS dataset. We are working on training baseline methods on the Ego-Exo4D dataset and will update the results as soon as they are completed.

Q5: Missing details

Thank you for bringing this to our attention. We realize that some technical details were not clearly outlined in our submission. As described in Supplementary Section A., the window size is set to 40 frames, so the lengths of the input and output frames can be inferred as 40 frames from the explanation in the Preliminary section. However, we acknowledge that this is not clear and easy to catch, so we will explicitly state these details in the main paper. Additionally, we will specify the sliding window size, which is set to 20 frames. Upon a thorough review of the paper, we also noticed that some details of the hand detectors, such as how we handled visibility for the Ego-Exo4D dataset, were not clearly stated. We will ensure that these details, along with the step size of the sliding window used during evaluation, are clearly presented in the main text to provide a complete understanding of our methodology.

Q6: Qualitative comparisons against SoTA

Please refer to the Q4 of the Author Rebuttal.

Q7: the motivation for using diffusion models

Please refer to the Q2 of the Author Rebuttal.

Q8: How we used IMU information.

The Ego-Exo4D dataset, collected by Meta using the Aria device, includes head trajectory data processed from IMU measurements. In the paper, we refer to this head trajectory as the tracking signal from the IMU, which is integral to our analysis and experiments. For the AMASS dataset, while IMU data is not used, we follow the approach of previous works such as AvatarPoser, where aggregated data of joint pose, joint velocity, 6D rotation, and angular velocity mimic the tracking signals from IMU. Both AvatarPoser and Avatar JLM have demonstrated that models trained with these signals are applicable to real-world data from AR devices.

Q9: Lack of publicly available code.

Please refer to Q5 of the Author Rebuttal.

We appreciate the reviewers insightful feedback on our paper. We will improve the draft based on the comments.

Q1: Stronger baselines & Reorganization of Table 1 and 2

Due to limited time and resources, we are currently only able to provide baseline results on the AMASS dataset. We are currently working on training basline methods on Ego-Exo4D datasets, and we will update as soon as they are completed. Please refer to Q1 in the Author Rebuttal to see our response on stronger baselines.

Thank you for your valuable feedback. We appreciate the suggestions for improving the structure and fairness of our evaluations. Since we now have more baselines as shown in the Table A in the pdf, it will be better to re-organize the table for a fair comparison and easier understanding.

Q2: The runtime performance analysis, and possible direction.

Please refer to the Q2 of the Author Rebuttal.

Thank you for your feedback regarding the runtime performance analysis. We understand the importance of computational efficiency, especially for real-time applications.

In Table B in the pdf and Q2 of the Author Rebuttal, we present a detailed comparison of the computational complexity of VQ-VAE, MAE, and VQ-Diffusion modules, highlighting the number of parameters, multiply-accumulate operations (MACs), and inference time. Our results show that the overhead introduced by the MAE, including uncertainty computations, is minimal compared to the significant overhead from the diffusion process. Specifically, MAE adds only 3 ms to the inference time, which is negligible compared to the 955 ms required by the VQ-Diffusion module. Therefore, the primary computational burden arises from the diffusion process rather than the MAE with uncertainty computations.

When we chose the diffusion model, we recognized the heavy computational cost but concluded it was more appropriate for solving the under-constrained problem of ego-body pose estimation. To mitigate this heavy computation issue, we selected the VQ-Diffusion method, which denoises in discretized latent spaces and is considered more computationally efficient. Additionally, as shown in Table C in the pdf, our approach allows skipping denoising steps using the reparameterization trick, following methods from [1] and [2]. The results show that our method provides four times faster options, effectively balancing the trade-off between MPJPE and inference time.

Recent research, such as [3] and [4], has focused on improving the speed of diffusion inference. We expect that as these advancements continue, the diffusion model's versatility and extendability will become even more beneficial, reducing the cost of the diffusion sampling process.

[1] Austin, Jacob, et al. "Structured denoising diffusion models in discrete state-spaces." Advances in Neural Information Processing Systems 34 (2021): 17981-17993.

[2] Gu, Shuyang, et al. "Vector quantized diffusion model for text-to-image synthesis." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Zheng, Hongkai, et al. "Fast sampling of diffusion models via operator learning." International conference on machine learning. PMLR, 2023.

[4] Yin, Tianwei, et al. "One-step diffusion with distribution matching distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q3: Clarification on Uncertainty

Yes, aleatoric uncertainty with a sampling strategy is used in our Table 1 and Table 2. To avoid confusion, we will clarify the strategy and the type of uncertainty at the beginning of Section 3.3.

Q4: Qualitative comparisons against state-of-the-art approaches

We appreciate your comment on the importance of qualitative comparisons. While we have visualized our qualitative results in a video format, we found that it is not permitted to include links during the rebuttal phase. As an alternative, we have included detailed comparative qualitative results in the PDF file attached to this rebuttal. We hope this additional information provides clarity and supports the quantitative improvements we reported. We also ensure that we will add video comparisons against state-of-the-art methods in the final version.

Q5: Suggestion of concurrent related works

Thank you for the suggestion. We will include suggested references in the related work section to ensure comprehensive coverage of concurrent research.

Q6: Mixed Density Network as Aleatoric Uncertainty

\begin{array}{l|ccc} \hline Methods & MPJPE & MPJVE & MPJRE \\ \hline \hline \text{w/o Uncertinaty}&6.05\pm0.01&30.12 \pm 0.04&4.36 \pm 0.00 \\ \text{DSPoser w/ MDN}&5.84\pm0.05&28.34 \pm 0.15&4.72 \pm 0.04 \\ DSPoser (Ours)&5.51\pm0.02&24.19 \pm 0.10&4.09 \pm 0.02 \\\hline \end{array}

Thank you for the interesting idea for uncertainty measurement. We reported the result of the whole pipeline after substituting the head of our MAE with a Mixed Density Network (MDN), setting the number of mixtures $M$ to 4 for fair comparison. Similar to the calculation of aleatoric uncertainty in our paper, we measure the aleatoric uncertainty of MDN by $\mathcal{U}_{ale}(\mathbf{x}) \approx M^{-1} \sum_i \pi_i\sigma_i^2 (\mathbf{x})$, where $\pi$ is mixture weight. The results show that MDN improves performance compared to our method without uncertainty; however, it shows worse performance compared to the MAE approaches.

\begin{array}{cc} \hline Methods & MPJPE \\\hline MDN & 13.45 \\ MAE (Ours) & 10.85 \\\hline \end{array}

To investigate the performance difference, we also analyzed the results of temporal completion of hand trajectories. Different from the MAE, the loss of MDN often diverged, so we early stopped the MDN training at 800 epochs, leading to worse MPJPE for hand trajectories constrast to the MAE model trained fro 4000 epochs.

Q7: Typos.

We will revise the sentence as suggested.

Clarification on inference setting in the evaluations.

Thank you for your feedback on our evaluation protocol. We would like to clarify that our method follows the BoDiffusion protocol, which uses a sliding step of 20. This approach averages the overlapping frames, producing a 40-frame output for each sliding step of 20. In contrast, AvatarPoser and AvatarJLM use a 1-step sliding window, focusing only on the final frame for real-time applications.

Is the inference time per frame or total time for 40 frames?

Our inference time is measured based on generating 40 frames. However, unlike AvatarPoser and AvatarJLM, our method's input and output are fixed (40 frames). Therefore, we can answer your question with a 'yes,' but considering the question may be assumed in an online-inference setup, we need to clarify that the inference time listed in the table B also represents the per-frame time in an online setup.

Additional experiments under real-time inference setup.

Recognizing that our initial comparison in previous rebuttal was not conducted under fair conditions, we performed additional experiments to ensure a fair comparison with the baselines.

\begin{array}{l|c|c|c|cc} \hline Methods & \text{imputation} & sliding step & averaging & MPJPE & MPJVE & MPJRE & Jitter \\\hline \text{AvatarPoser [10]} & \text{MAE} & \text{1} & & 9.88 & 62.31 & 5.98 & 37.89 \\ \text{BoDiffusion [3]} & \text{MAE} & \text{20} & \text{temporal avg} & 7.35 & 31.33 & 5.47 & 1254.84 \\ \text{AvatarJLM [34]} & \text{MAE} & \text{1} & & 7.12 & 37.60 & 5.24 & 16.95 \\\hline DSPoser (Paper) & \text{MAE} & \text{20} & \text{temporal avg} & 5.51 & 24.19 & 4.09 & 4.27 \\ \text{DSPoser (\#1) } & \text{MAE} & \text{1} & & 5.87 & 52.38 & 4.31 & 34.12 \\ \text{DSPoser (\#2) } & \text{MAE} & \text{1} & \text{temporal avg} & 5.23 & 21.73 & 3.83 & 5.94 \\ \text{DSPoser (\#3) } & \text{MAE} & \text{1} & \text{4 samples} & 5.68 & 29.48 & 4.23 & 12.98 \\\hline \end{array}

Before discussing the results, we want to clarify that our primary focus is on introducing two key aspects: (1) the underexplored problem of doubly sparse video data, and (2) a generic multi-stage framework to address such problems. The specific choices within our framework (e.g. VQ-Diffusion, MAE, and the sliding window) were deliberately kept simple to demonstrate the efficacy of the intermittent tracking signal and our multi-stage framework. As reviewers jBAY and 2EAc noted, the computational demands of diffusion models pose concerns for real-time applications. While we provided baseline results to justify our design choices, we want to emphasize that alternatives such as AvatarJLM and AvatarPoser (instead of VQ-Diffusion style pose estimation algorithms used in the submission) are also viable within our multi-stage framework. These alternatives can balance time complexity and accuracy, making them suitable for real-time applications.

Exp. #1, fair comparison.

We first tested our method using the evaluation protocol of AvatarPoser and AvatarJLM, with a sliding window step of 1 where only the final output frame (current frame) is used for each step. Our method performed worse in Jitter and MPJVE compared to AvatarJLM but showed better performance in MPJPE and MPJRE. We believe this drop is due to the probabilistic nature of our method, unlike the deterministic approach of AvatarJLM and AvatarPoser.

Exp. #2, temporal averaging

Next, we modified Exp #1 to better utilize the diversity from our uncertainty modeling by averaging overlapped frames while advancing the sliding window. This significantly reduced errors across all metrics. However, when it comes to real-time infernece, the improvement doesn't affect to current frames but only applies to historical frames, which are not useful for real-time inference.

Exp. #3, multiple sampling

Finally, we implemented a multiple sampling approach, averaging four samples for each step using the end frames (current frames). This method outperformed AvatarJLM across all metrics and can be efficiently implemented with parallel processing, resulting in minimal time overhead from #1.

In conclusion, we agree with the reviewer 2EAc that the better performance in MPJVE and Jitter of our method is due to the difference in sliding steps. However, based on Exp #2 and Exp #3, the performance drop appears to stem from our framework's ability to generate diverse motions from the same input. This issue can be mitigated by sampling multiple times using parallel processing and averaging the results.

We appreicate the acknowledgement of our motivation and the novelty of our method. We will improve the draft based on the valuable comments of the reviewer.

Q1: Qualitative comparisons against state-of-the-art approaches and video visualizations of the results.

We appreciate your comment on the importance of qualitative comparisons. While we have visualized our qualitative results in a video format, we found that it is not permitted to include links during the rebuttal phase. As an alternative, we have included detailed comparative qualitative results in the PDF file attached to this rebuttal. We hope this additional information provides clarity and supports the quantitative improvements we reported. We also ensure that we will add video comparisons against state-of-the-art methods in the final version.

Q2: VQ-VAE ablation studies.

\begin{array}{l|c|cccc} \hline Methods & \text{Pipeline} & MPJPE & MPJVE & MPJRE & Jitter \\ \hline \hline \text{BoDiffusion [3]} & \text{MAE}  + \text{Skeleton Space Diffusion} & 7.35 & 31.33 & 5.47 & 1254.84 \\ \text{AvatarJLM [34]} & \text{MAE} + \text{Transformer} & 7.12 & 37.60 & 5.24 & 16.95 \\ DSPoser (ours) & \text{MAE} + \text{VQ-Diffusion} + \text{VQ-Decoder} &5.51 & 24.19 & 4.09 & 4.27 \\ \hline \end{array}

To evaluate the effectiveness of our pipeline, we implemented baseline models to tackle the pose estimation problem given doubly sparse data. For these baseline models, we introduced MAE at the initial stage of their methods to complete the temporally sparse data, then fed the imputed trajectory into their respective models. The implementation of BoDiffusion as a baseline can serve as an ablation study for our VQ-VAE, as it applies the diffusion process directly to the skeleton space, contrasting with our approach of applying the diffusion process on the Vector-Quantized latent space. Additionally, AvatarJLM can be considered another ablation study, as it utilizes a Transformer instead of a diffusion model to complete the sparse data. These results demonstrate that VQ-VAE outperforms the other architectural options.

Q3: Overfitting issue.

Thank you for your observation and insightful feedback. While it is possible that overfitting could explain this observation, there are other plausible explanations. To the best of my knowledge, when a human moves, the motion of each joint influences the others. Previous research, such as [1], [2], has demonstrated that joint movements are not only connected to adjacent joints through bones (explicit relationships) but also highly related to distant joints that are not directly connected (implicit relationships) in a certain motion context. Additionally, in [3], even though only head position is utilized for whole body estimation, the results often show very accurate lower body movements. This indicates that even if there appears to be no direct information for reconstructing the kicking motion in our visualized example, the intermittent hand observations and dense head trajectory data may provide sufficient information to reconstruct the kicking motion. Therefore, what may seem like overfitting could actually be the model leveraging these implicit relationships to generate a plausible motion sequence.

[1] Chi, Hyung-gun, et al. "Infogcn: Representation learning for human skeleton-based action recognition." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[2] Wu, Zhize, et al. "SelfGCN: Graph Convolution Network with Self-Attention for Skeleton-based Action Recognition." IEEE Transactions on Image Processing (2024).

[3] Li, Jiaman, Karen Liu, and Jiajun Wu. "Ego-body pose estimation via ego-head pose estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Q4: Evaluation on the velocity/acceleration metric.

\begin{array}{l|cccc} \hline Methods&\text{MPJPE}&\text{MPJVE}&\text{MPJRE}&\text{Jitter} \\ \hline \text{AvatarPoser [10]}&40.42&64.07&16.37&27.89 \\ \text{Bodiffusion [3]}&46.45&75.33&17.99&2793.32 \\ \text{AvatarJLM [34]}&25.02&68.42&14.14&32.18 \\ \text{AvatarPoser [10]}&9.88&62.31&5.98&37.89 \\ \text{Bodiffusion [3]}&7.35&31.33&5.47&1254.84 \\ \text{AvatarJLM [34]}&7.12&37.60&5.24&16.95 \\\hline DSPoser (Ours)&5.51\pm0.02&24.19\pm0.10&4.09\pm0.02&4.27\pm0.03 \\\hline \end{array}

\begin{array}{l|c|cccc} \hline \text{Methods} & \mathbf{y} & \text{MPJPE} & \text{MPJVE} & \text{MPJRE} & \text{Jitter} \\ \hline \hline \text{GT} & - & - & - & - & 4.01 \\ \text{VQ-VAE (Paper)} & \text{Full body} & 1.26 & 11.37 & 1.81 & 3.93 \\ \text{VQ-VAE (Opt'ed)} & \text{Full body} & 1.15 & 10.59 & 1.67 & 3.89 \\ \hline \text{AvatarPoser [10]} & \text{Dense traj.} & 4.18 & 29.40 & 3.21 & 13.63 \\ \text{Bodiffusion [3]} & \text{Dense traj.} & 3.63 & \mathbf{14.39} & \mathbf{\textcolor{blue}{2.70}} & 493.78 \\ \text{AvatarJLM [34]} & \text{Dense traj.} & \mathbf{3.35} & 20.79 & 2.90 & \mathbf{\textcolor{blue}{8.39}} \\ \hline \mathbf{DSPoser (Paper)} & \text{Dense traj.} & 3.61 \pm 0.01 & 18.36 \pm 0.03 & 2.81 \pm 0.02 & 4.08 \pm 0.02 \\ \mathbf{DSPoser (Opt'ed)} & \text{Dense traj.} & \mathbf{\textcolor{blue}{3.48 \pm 0.01}} & \mathbf{\textcolor{blue}{17.86 \pm 0.03}} & \mathbf{2.68 \pm 0.02} & \mathbf{4.03 \pm 0.02} \\ \hline \end{array}

We deeply appreciate your valuable comments on our paper. Thanks to your suggestion, we found out that our method shows significantly better performance on the Jitter metric, which is often used to measure the smoothness of motion. Jitter is a measure of jerk calculated by the derivative of acceleration, and MPJVE, already reported in our paper, indicates the velocity error. As seen in the table above and Q2 of the VQ-VAE ablation studies, it is demonstrated that our method produces smoother results compared to other methods.

We appreciate your acknowledgment of our paper's novelty and your valuable suggestions for improving our method. We fully recognize the importance of open-source code in advancing research, as this paper is also built upon the publicly available code of other researchers.

Q1: Key contributions can be consolidated into one & no experiments involving real-life AR devices.

We appreciate the feedback regarding the similarity between our first and second contributions. Our primary aim was to highlight different aspects of our approach, but we understand the need for clarity and consolidation. Additionally, while the dataset we used in our experiments, EgoExo4D, is collected using Meta's Aria Devices (an HMD/AR device) and demonstrates the potential for AR applications, we recognize that explicit experiments involving real-life AR devices would strengthen our claims. We will consider this in future work.

Q2: Qualitative comparisons against state-of-the-art approaches and video visualizations of the results.

We appreciate your comment on the importance of qualitative comparisons. While we have visualized our qualitative results in a video format, we found that it is not permitted to include links during the rebuttal phase. As an alternative, we have included detailed comparative qualitative results in the PDF file attached to this rebuttal. We hope this additional information provides clarity and supports the quantitative improvements we reported. We also ensure that we will add video comparisons against state-of-the-art methods in the final version.

Q3: Lack of publicly available code.

We recognize the value of open-source code and plan to release ours upon acceptance. Our organizational policy prioritizes careful review and preparation before any public release. In the meantime, we're committed to providing detailed methodologies in our publications to support reproducibility and further research in the field.

Q4: Misc.

Thank you for pointing out the grammar mistakes and writing issues. Following the reviewer's comments, we will:

1. Conduct a thorough grammar review of the manuscript, including the ones you mentioned.

2. Remove the phrase "using the transformer architecture" in L155.

We believe these changes will improve the clarity and readability of our paper.