SegFit: Robust SMPL-X Fitting with Body Part Segmentation on Real-World Point Clouds

1. The paper is greatly lacking in experimental detail which makes it difficult to evaluate. It's not clear even what data the models were trained on. See Questions for elaboration. It is imperative that the authors clarify whether the segmentation model and the baseline methods were trained on the same data. If not, the "robustness" claims appear somewhat dubious.
2. The authors claim "This enhanced segmentation serves as pseudo ground truth to fine-tune the segmentation network in a self-supervised manner, creating a feedback loop where improvements in pose fitting lead to better segmentation and vice versa." (emphasis added) but appear to only evaluate the effect of the finetuning on segmentation performance, leaving the pose-fitting portion of the claim seemingly unsubstantiated. Relatedly, the authors should consider changing the paper title as the focus of the work appears to be segmentation, with point-cloud registration appearing primarily as a means.
3. In the conclusion, the authors write "This self-supervised cycle enhances the robustness and generalization capabilities of the method across diverse datasets, as shown by our experiments on the PosePrior (Zhang et al., 2021), BEHAVE (Bhatnagar et al., 2022), EgoBody (Zhang et al., 2022), and Hi4D (Yin et al., 2023) datasets," highlighting generalization of the segmentation method to Hi4D. However, it must be pointed out that the adaptation method failed on Hi4D, requiring supervised finetuning: "Furthermore, as Human3D struggles to differentiate between touching instances in Hi4D, we finetune on this dataset before evaluating SegFit." The authors seem to forget about this and later attribute high performance on Hi4D to data quality: "Both Hi4D and PosePrior are very clean datasets with high-quality point clouds and barely any occlusions. Thus, Human3D has already achieved high accuracy, which we cannot improve."
4. The authors have demonstrated improved performance through finetuning the segmentation model on pseudo-ground-truth samples, but do not seem to have evaluated whether the nearest-neighbor segmentations obtained through the registration are more accurate: "Use the fitted SMPL-X meshes to reassign part labels to the point cloud via nearest-neighbor matching, resulting in enhanced segmentation."
5. The authors' reference to prior work is sloppy and the papers cited often do not match the sentences in which they appear. This includes all citations in the motivation of the work:

Fitting a parametric human body model to 3D point cloud data (measured by, e.g., Lidar or Kinect) without contextual guidance can be challenging and often yields imprecise results (Bogo et al., 2016; Kolotouros et al., 2019). However, having prior information about body parts can significantly improve this process (Varol et al., 2017; Zanfir et al., 2018; Xu et al., 2020). Thus, this paper focuses on incorporating body part segmentation into the body fitting process.

The first four referenced works, SMPLify, SPIN, SURREAL, and Zanfir et al. 2018, are image-based HPS methods and do not deal with 3D point-cloud data. GHUM involved point-cloud registration, but it's not clear how it relates to the claim. This continues throughout the paper. As a second example, from the following quote it might be understood that the cited works are quantitatively compared against:
> Our experiments demonstrate significant improvements in both pose modeling accuracy and body part segmentation performance compared to leading methods (Kolotouros et al., 2019; Kocabas et al., 2020; Zanfir et al., 2018), especially in the complicated scenes.

However, these are image-based HPS works, none of which are compared against nor even could they be.

I summarize my questions and concerns here:

1. Why does the paper fixate so much on relying only on the point clouds for model fitting? All the datasets used in the paper contain RGBD data, so it would be natural to integrate the RGB information into the model. Also, as far as I know, the current SOTA of SMPL model fitting relies largely on multi-view RGB information.
2. Line 279. How is VPoser used in the optimization? These details seem to be missing from the paper and should be added.
3. Section 3.6: What is the reasoning behind using a direct assignment of the segmentation using the nearest point on the SMPL model? Would a probabilistic/voting scheme that considers more neighbors and the current assignment of the point not be more robust?
4. Line 305: "By training the network on these improved labels, we enable it to generalize better to new, unlabeled data and enhance its segmentation accuracy in diverse scenarios" -> This is untrue. The paper proposes to use data of a specific dataset and fine-tune the segmentation model on this specific data. This does not naturally lead to better generalization performance, as the segmentation model will be adapted to perform well on this one specific dataset. I suggest changing this sentence to reflect this or adding an experiment that confirms that fine-tuning a model on a single dataset leads to better generalization. If this statement were true, we could finetune the model on the BEHAVE dataset and apply it on the EgoBody dataset to achieve better performance and vice versa.
5. Table 1. Why does ArtEq perform better on the PosePrior dataset? On PosePrior, ArtEq also takes much longer to optimize than on the other datasets. Does this mean there exists an error on the other evaluations? In contrast, the proposed methods and PTF take always about the same time on all datasets. It seems suspicious that ArtEq does so much better on one dataset but not on the others. Please explain the difference.
6. How would the proposed approach compare to Multi-view or even single-view methods that use RGB-D data instead of only point clouds?
7. Table 3. The fact that Human3D does better in some instances than when we do reassignment based on simple nearest neighbor seems to indicate that a probabilistic approach that considers multiple factors would be better.

The components are all very simple. I've been working on human motion tracking and pose estimation for more than a decade, either fitting to 2D and/or 3D data. I know very well how brittle a SMPL-based fitting method is and how sensitive it is to the noise in observations. The proposed method consists of only a data term and a few standard prior terms, e.g., Eq. 5, which is very easy to break.  The 3D observations can contain missing data, e.g., Fig. 1(a), and/or outliers, backpacks, and loose apparel, and yet there is even no temporal smoothness term in the proposed method. I am missing some key insight why it works well, so I dropped a few concrete questions below. While I put "marginally below the acceptance threshold," it is really borderline. If the questions get addressed with evidence, I am happy to increase the rating.

The innovation of this paper is somewhat lacking. While the main contribution is in unsupervised body part segmentation, it appears that a good initialization is crucial for subsequent processes. However, this initialization relies on a supervised method called "3D Segmentation of Humans in Point Clouds with Synthetic Data."

Additionally, the paper lacks essential experimental analysis. Table 1 indicates significant performance improvements on the "BEHAVE" and "EgoBody" datasets, but the enhancement on the "PosePrior" dataset is limited.