GaussianClin: Multimodal Featured Gaussian Splatting for Dynamic Clinical Videos

1. In line 131, the author claims one contribution is accelerating training and inference speed, but the manuscript does not report the training times for the proposed method or comparison methods.
2. In line 727, the author mentions the language-guided editing function can "highlight the tumor area" and "isolate blood vessels near the incision site," but no results for these functions are shown.
3. In line 296, the author discusses the challenge of rapid camera motion, yet the three datasets used for experiments have a fixed camera position.
4. In line 323, the parameters $\lambda_s$ and $\lambda_t$ are not clearly defined or used in the manuscript.
5. The language-guided tool deletion is unnecessary since the datasets already contain segmentation masks of tools, which can be used to filter out regions in the NeRF and GS settings.
6. There is no reference provided for the Feature DS method.
7. The manuscript lacks comparisons to recent methods like EndoGaussian [1], and the reconstruction quality appears lower than these methods.

[1] Liu, Yifan, et al. "Endogaussian: Gaussian splatting for deformable surgical scene reconstruction." arXiv preprint arXiv:2401.12561 (2024).

• The paper is extremely busy and it is hard to assess what the exact methodological contributions are. What parts of the spatio-temporal graph distillation method are really novel, for example? How does the feature distillation method presented here differ methodologically from the other approaches mentioned in 2.3? Is the innovation mainly the application to a medical setup? In my opinion, it would have been better to substantially shorten the first two section of the paper to have additional space to make the contributions more explicit in Sec. 3.
• The paper claims that the use of the foundation models equips the model with clinical knowledge. I am not entirely sure if that's true and if it is not more an incorporation of general object detection etc. skills that help to better segmentation structures. I am especially surprised about this claim as the majority of foundation models included are general purpose model such as SAM.
• I find the chosen baselines in parts of the experiments quite unconvincing. For example, in Sec. 2.2 several recent 4D approaches are discussed (e.g, EndoGaussian), but none of them is actually used in the experiments. While I recognize that they may not be usable for all downstream tasks, wouldn't it have been possible to use them at least for the novel view rendering part? On the other hand, the chosen baseline FeatureDS does not seem to be described at all. In general, a better justification of the baselines for each experiment is critically needed.
• I am also struggling with fully understanding the training process in the way the paper is written. Are all feature maps derived from all possible foundation models used simultaneously? Is everything trained end-to-end?
• A major contribution is the spatio-temporal regularization approach, but I fail to see a specific experiment that evaluate this aspect. Where can I find results that show its effectiveness?

Some weaknesses/areas of improvement:

1. Incremental Novelty: While integrating multimodal features into Gaussian Splatting for dynamic 3D reconstruction is an interesting approach, as I mentioned before, it may be considered an incremental advancement rather than a groundbreaking innovation. The method builds upon existing techniques by combining pre-trained 2D models with 3D reconstruction, which, although useful, might not represent a significant leap in the field.

2. Limited Experimental Evaluation: The experimental results are promising but somewhat limited in scope. The evaluations are conducted on a narrow set of datasets, and comparisons are primarily made against baseline methods without including a broader range of SOTA techniques.

3. Practical Applicability Concerns: I wonder the extent to which this could be implemented practically. No inclusion of experimenting with real-time performance in clinical settings.

4. Insufficient Detail on Key Components: As I told before, the spatiotemporal graph distillation mechanism is a central part of the proposed approach, but the paper provides insufficient detail about its implementation and impact.

5. Overreliance on Pre-trained Models: The approach depends heavily on pre-trained 2D models that may not be specifically tailored to medical imaging. This reliance could limit the method's ability to capture specialized anatomical details or pathological variations critical for medical applications, potentially affecting accuracy and reliability. Maybe a fine-tuned model could be a better fit?

Additional clarity regarding the addition of the feature embeddings into the Gaussian model can be suggested. Similarly, it is recommended to give some details regarding the training/validation/testing setup used during the experiments (See questions)