Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery

We appreciate that the reviewer acknowledges the significance of our work. We would also like to thank the reviewer for the suggestions on clinical applications, evaluation, and experiments - these have been really crucial for improving our work.

W1 & Q1- Clinical applications:

We sincerely thank the reviewer for their insightful comments regarding the practical applications of our work. As we emphasized in the introduction, accurate three-dimensional reconstruction provides substantially richer information compared to traditional two-dimensional imaging methods. Capturing the precise spatial relationships, depth information, and temporal dynamics of hand-instrument interactions forms the foundation for numerous downstream clinical applications. Our 3D reconstruction framework captures critical information inherently missing from 2D methods: precise positioning of instrument tips in 3D space, accurate hand joint poses under occlusion, authentic spatial relationships in bimanual interactions, and motion trajectories with depth perception. These 3D features are essential prerequisites for the following applications:

(1) Quantitative surgical skill assessment: Unlike 2D video analysis, our 3D reconstruction enables measurement of precise motion metrics including 3D path length, economy of motion, millimeter-level tremor amplitude, and authentic bimanual coordination patterns. Additionally, we have included synchronized ophthalmic microscope videos along with surgical/practice experience duration data from 40 surgeons, facilitating more comprehensive and systematic analysis.

(2) Intelligent surgical training systems: Detailed 3D motion trajectories support the development of AI-driven training systems with real-time feedback on instrument angles, force vectors, and spatial precision—elements that cannot be accurately assessed from 2D projections alone.

(3) Surgical workflow optimization: Our phase-annotated 3D data can be used to analyze motion efficiency, workspace utilization, and ergonomic patterns, thereby providing evidence for improving surgical techniques and reducing operator fatigue.

(4) Next-generation robot-assisted surgery: Millimeter-precision 3D hand-instrument coordination data provides crucial training data for developing semi-autonomous surgical robots capable of replicating expert-level minimally invasive surgical movements.

(5) Risk assessment and safety monitoring: 3D reconstruction can detect potentially hazardous movements, proximity to critical structures, and predict complication risks based on deviations from optimal trajectories.

We recognize these application domains need comprehensive discussion to fully demonstrate our work's clinical impact. In the final version, we will expand the discussion section to include detailed use cases, implementation pathways, and clinical validation strategies, highlighting the transformative potential of 3D surgical scene understanding in advancing ophthalmic surgical practice.

W2 & W3 – Temporal consistency and comparison with HaMeR:

We thank the reviewer for these valuable suggestions, we fully agree that assessing temporal motion quality is important. We would like to answer this from two aspects: 1) data annotation pipeline and 2) baseline method, by comparing both methods to HaMeR [1].

(1) Annotation pipeline vs HaMeR. Since we are constructing the ground truth itself, it is inherently challenging to perform standard quantitative evaluation like MPJPE or acceleration error without a reference. Nevertheless, we have implemented multiple physical constraints and regularisation terms with temporal smooth to guarantee the annotation reliability. We now provide evaluations using Acceleration (not error term), Jerk following Dyn-HaMR [2], and Penetration (between hands and instruments) metrics where each of them designed to quantify motion stability and interaction plausibility (also in response to MyX8, feMU). Although traditional error metrics like MPJPE or acceleration error cannot be computed without a reference, we can still observe the significant improvements over HaMeR, where it presents much lower Acceleration, Jerk and Penetration, indicating the high temporal consistency and the effectiveness of the annotation method. Additionally, the strong performance of downstream models trained on our annotations, and the superior values across the temporal metrics, offer indirect yet compelling evidence of annotation quality.

(2) OH-Net vs HaMeR. We appreciate the reviewer highlighting HaMeR. As noted above, HaMeR is used as initialization in our data annotation pipeline and is also evaluated separately, where we demonstrate significantly improved temporal consistency, stability, and plausibility.

However, it is important to note that HaMeR is a large-scale foundation model trained on 2.7M samples using a huge ViT backbone, and is designed primarily for per-frame local hand pose estimation. In contrast, our baseline methods are specifically developed to reconstruct joint hand–instrument and support interaction-aware reasoning. As HaMeR is a large-scale foundation model designed exclusively for hand pose estimation, without accounting for object interactions, we instead benchmark our approach against state-of-the-art hand-object (instrument) pose estimation baselines, as reported in Tables 2 and 3.

Moreover, both HaMeR and our baseline models are per-frame methods and do not explicitly model temporal consistency. Nevertheless, to provide additional insight, we evaluated HaMeR using its official checkpoint on our dataset under the same protocol. Specifically, we randomly selected 100 clips from the test split and repeated the evaluation 5 times (totalling 500 sequences) to ensure consistency and robustness. Due to the scale of HaMeR and the substantial computational & time cost associated with retraining a foundation model, domain-specific retraining was not feasible during the limited time and the scope. We thank the reviewer for the advice and carefully add this into the final version.

As shown in the table below, our methods consistently achieve lower Acceleration and Jerk, indicating superior temporal stability. This further supports the value of our annotation pipeline and task-specific modeling.

W4 - No surgical quality metrics:

In JIGSAWS, the authors conducted quantitative assessment of technical skills for three fundamental surgical tasks (Suturing, Knot-Tying, and Needle-Passing) using six specific dimensions (Respect for tissue, Suture/needle handling, Time and motion, Flow of operation, Quality of final product, and Overall performance), with each dimension rated on a 1-5 point Likert scale (where 1 point represents the poorest performance, 3 points indicates "competent/adequate" performance, and 5 points denotes the best performance), and the total score calculated as their sum. The annotation was performed by an experienced gynecological surgeon with extensive expertise in robotic and laparoscopic surgery through video review. It is worth noting that JIGSAWS' assessment method still relies on subjective expert scoring, where this direct rating approach based on human observation not only has limited interpretability but also inevitably suffers from raters' subjective biases. In contrast, our work provides a more fundamental and objective analytical framework. As emphasized in the introduction and W1, we establish a new technical foundation for the entire surgical skill assessment field through constructing a large-scale dataset and developing advanced 3D reconstruction techniques. Though space constraints prevent us from incorporating additional annotations or experiments, our core contributions in this work are pioneering.

More importantly, our framework represents a paradigm shift: from 2D video analysis relying on subjective scoring to objective data analysis based on precise 3D reconstruction. This transformation is particularly crucial in ultra-high precision ophthalmic surgery, where millimeter-level operational errors can lead to severe consequences. By integrating 3D hand joint positions, 3D instrument poses, depth information, and visualizations, we not only provide quantifiable, verifiable, and reproducible objective measurements, but more importantly, establish a technical platform that can eliminate human biases and support standardized assessment. This objectivity and precision, unattainable through traditional subjective scoring methods, enables: (1) surgeons to make more scientific decisions based on precise spatial data; (2) training systems to provide more accurate feedback; and (3) skill assessment to be based on objective metrics rather than subjective impressions. We will cite JIGSAWS and include a comparative analysis in the final manuscript. We believe this foundational work will drive a new generation of surgical education, training, and assessment systems, while offering essential tools and data resources for researchers in the field.

References:

[1] Reconstructing Hands in 3D with Transformers

[2] Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera

Dear Reviewer 3Ziv,

We sincerely appreciate the time and effort you have invested in reviewing our submission. Your insightful feedback has been invaluable to us, and we have diligently worked to address all the concerns you raised in our rebuttal. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We are more than happy to provide any further details or explanations. Thank you once again for your thoughtful review and consideration.

Best regards,

The Authors

Additional experiments on HaMeR:

We thank the reviewer for their thorough evaluation and for raising this important point. To fully address this concern, we evaluated HaMeR using its official checkpoint on our test set, following the protocol described above: we randomly selected 100 clips and repeated the evaluation 5 times to ensure stability and fairness.

As shown in the table, our H-Net and OH-Net, trained with domain-specific supervision, outperform HaMeR in terms of positional accuracy, achieving lower MPJPE, MPVPE under our task setting. This result is expected given that HaMeR is a general-purpose, large-scale hand model not fine-tuned for our surgical domain.

Importantly, HaMeR is known to suffer from failure cases such as hallucinated hand configurations (e.g., wrong-handedness, missing detections) in challenging scenarios involving gloves or extreme lighting, which leads to higher MPJPE because of mismatching. These issues are exacerbated by the absence of post-processing steps like non-maximum suppression in its default pipeline, resulting in low MPJPE and MPVPE regardless of reasonable single hand performance. In contrast, our method benefits from domain-specific annotations and a bimanual-restricted initialization, allowing for more robust disambiguation of left and right hands, which is particularly critical in our setting.

On interpenetration between hands:

We thank the reviewer for raising this point. To verify the validity of our annotation in terms of hand–hand interpenetration, we inspected the entire dataset as well as consulted our experienced ophthalmic surgeon. According to expert feedback, interpenetration between the left and right hands during ophthalmic surgery is extremely rare. In standard practice, the surgeon's hands typically operate independently and often hold separate instruments with sufficient spacing. Even when both hands interact with the same instrument, their points of contact are well-separated, often aided by layers of irrigation or adhesive materials that physically separate the hands.

As such, our annotation pipeline does not explicitly impose loss terms penalizing hand–hand intersection to speed up optimization process. However, we did not observe any notable cases of interpenetration during our annotation and visual verification process. We appreciate the reviewer’s suggestion and will consider adding an explicit hand–hand collision term in future work for completeness, although it appears unnecessary in the context of ophthalmic surgical motion.

Thank you for your detailed review. We hope the experiments address your concern.

We appreciate the reviewer’s recognition of the contributions of our work. We are glad that the novelty of modeling bi-manual hand and instrument motion in surgical contexts was well received, and that the dataset were found valuable and significant.

W1 & Q1 - About the annotation pipeline:

We thank the reviewer for raising this point and apologize for any confusion. Our annotation pipeline does not rely on manual corrections during the optimization process after the initialization. Overall, manual correction was applied only to the mask initialization in the first frame of each video to improve hand-instrument tracking. All subsequent masks were automatically inferred by SAM2. We have now clarified this more explicitly in the revision.

W2 & Q2 - More systematic evaluation on dataset:

We appreciate the reviewer’s comment regarding the systematic evaluation of annotation quality. Since we are constructing the ground truth itself rather than evaluating against an existing dataset, it is inherently challenging to perform standard quantitative evaluation. Nevertheless, we have implemented multiple constraints to ensure annotation reliability, including thorough visual inspection across all sequences, cross-view 3D consistency checks, and sanity checks on temporal smoothness. Our method is based on the SotA initialisations and refines the hand instrument motion in the aspects of temporal consistency, contact and the hand pose accuracy in the following stages. To support our claim, we hereby provide a comparison on the test split between our data and SotA model (e.g. HaMeR [1] + instrument initialized as mentioned Section 3.2 and transformed into HaMeR's coordinate system) in terms of motion quality using Acceleration, Jerk and Penetration (between hand and instrument) metrics in the table below following Dyn-HaMR [2], as well as the ablation study provided in the response to [MyX8], which demonstrates the effectiveness of the annotation method and the quality. Note that although other standard error metrics (e.g. MPJPE, MPVPE, Acc Err) for them cannot be computed in this case as we are constructing the ground truth, we can still analyse the Acceleration, Jerk for hands and Penetration rate between hand and instrument.

We can observe significant improvements over the SoTA (e.h. HaMeR), where much better Acceleration and Jerk proves stability of our annotation pipeline and the Penetration shows the effectiveness of our physical constraints and regularisation terms. The results are obtained by randomly selecting 100 clips on test split and repeating for 5 times.

W3 & Q2 - More qualitative result:

We fully agree with the reviewers that more visualizations can offer a better understanding of the dataset. In addition to the operational-frame visualizations in the appendix, our supplementary video presents, beginning at 33s, 8 operation clips along with their reconstruction visualizations (4 shorter and 4 longer clips). While we already included video samples of the predictions in our original submission, we regret that the NeurIPS rebuttal policy does not allow us to share additional materials at this stage. We acknowledge the value of presenting more examples and agree that doing so can help further demonstrate the effectiveness and limitations of our method. We now provided a comprehensive and diverse set of results in our final version. We invite the reviewer to follow our final release, where we have also share extended visualizations on the project page.

References:

[1] G. Pavlakos et al., “Reconstructing Hands in 3D with Transformers,” CVPR ’24.

[2] Z. Yu et al., “Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera,” CVPR ’25.

Dear Reviewer feMU,

We sincerely appreciate the time and effort you have invested in reviewing our submission. Your insightful feedback has been invaluable to us, and we have diligently worked to address all the concerns you raised in our rebuttal. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We are more than happy to provide any further details or explanations. Thank you once again for your thoughtful review and consideration.

Best regards,

The Authors

Thank you for your detailed feedback and positive assessment. We are pleased that our response addressed your concerns.

We kindly request you to consider increasing the score, as we believe the clarifications and additional evidence strengthen our work.

We will include further evaluation results and enhanced visualizations to better highlight the dataset’s quality. We sincerely appreciate your constructive suggestions and the time you dedicated to reviewing our paper.

We thank the reviewer for their thoughtful and constructive feedback. We are pleased that they recognize the impact of our proposed dataset and its significant contribution to the research community and applications such as the analysis of critical operational workflows. We also appreciate the reviewer’s constructive assessment of our benchmark system design, which they found appropriate and supportive of the dataset’s utility as a benchmarking instrument. We are encouraged that the reviewer has a clear understanding of the goals and contributions of our work.

W1 - Challenges in fine-grained motion estimation:

We agree with the reviewer that, as the pioneer in the field to recover 3D bimanual instrument interactions in the challenging context of ophthalmic surgery, capturing fine-grained instrument articulation, particularly during delicate procedures (e.g., capsulorhexis) involving subtle motions and instruments such as toothed forceps and capsulorhexis forceps, under extreme lighting & occlusion conditions may exhibit slightly reduced alignment accuracy compared to other phases. We would especially polish the dataset and keep updating with the newest features.

W2 - Minor corrections & typos:

We thank the reviewer for bringing these up. We have now carefully corrected them and incorporated them into our final revision of the paper.

W3 & Q1 - Ablation study:

We absolutely agree with the reviewer that, due to the novelty of the task and our benchmark, a detailed ablation study could further benefit the community. This is the reason why we ablates the on the essential components of the baseline OH-Net in Table. 3 of the main paper. Moreover, we hereby also provide the ablation study for the loss terms used in the annotation pipeline including $L_{inter}$ and $L_{sdf}$, by analysing the Acceleration, Jerk for hands following Dyn-HaMR [1] and Penetration rate between hand and instrument:

The table shows the impact of different error terms in our joint loss function. Specifically, we found $L_{3D}$ can speed up the convergence and increase the stability. Among them, $L_{inter}$ and $L_{sdf}$ can help with better interaction modelling with less inter-penetration. Other regularization terms (e.g. $L_{prior}$, $L_{bio}$) are included in all experiments to avoid unrealistic hand poses. The results are obtained by randomly selecting 100 clips on test split and repeating for 5 times.

Q2 - Model training clarification:

First of all, we would like to kindly point out that only the proposed annotation pipeline relies on multiple large pre-trained models, while the baseline methods (e.g. OH-Net, H-Net) do not require any pre-training. In terms of the annotation pipeline, the methods used for initialization were using the officially released checkpoints without any additional pre-training. For the comparison with baseline OH-Net, all the methods in the experiments were trained and evaluated on our dataset with the same data split and protocols.

Q3 - Runtime analysis:

For our baseline method OH-Net, it takes about 0.1 ~ 0.3 seconds on a monocular RGB image to predict the poses using a single NVIDIA A100 GPU whilst the SoTA method HaMeR takes around 1.4 seconds. As we mentioned that only our annotation pipeline needs a reasonable initialization from existing pose estimation models, the average runtime for it could vary depending on different initialization methods. In terms of the runtime of annotation pipeline, please refer to Section D.2 of the Appendix.

[1] Z. Yu et al., “Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera,” CVPR ’25.

Dear Reviewer MyX8,

We sincerely appreciate the time and effort you have invested in reviewing our submission. Your insightful feedback has been invaluable to us, and we have diligently worked to address all the concerns you raised in our rebuttal. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We are more than happy to provide any further details or explanations. Thank you once again for your thoughtful review and consideration.

Best regards,

The Authors