ContactField: Implicit Field Representation for Multi-Person Interaction Geometry

Thank you for your valuable feedback on the synthetic dataset and our experiments. Below are our responses to weaknesses and questions in your comments.

Weakness 1: details about datasets

We outline the relevant information about the dataset below.

Data Source: Initially, we acquired characters with various ages and interaction motion sequences from Character Creator 4 [1].

Synthesis and Annotation Protocol: Subsequently, we composed scenes featuring multiple characters using Omniverse USD Composer [2]. To facilitate the dataset generation process, we engineered the Kaolin rendering tool[2], incorporating tasks such as multi-person normalization to achieve the desired outputs. This process enabled us to generate multi-view rendered images, mask images, instance masks, and 3D geometry. Additionally, we created normal ground truth maps and joints, although these were not utilized in this paper.

Data Scale: The dataset includes 49 characters (26 female, 23 male) with 50 motion sequences across 43 scene configurations, totaling approximately 25,000 frames. For a detailed breakdown, please refer to Figure A in the attached PDF file. We hope this detailed information will provide a clearer understanding of our synthetic dataset and its role in evaluating our proposed method.

Weakness 2: detailed hyper-parameters

The hyper-parameters are as follows: $\tau_{c}=0.25$ in L236, $\omega_{s} = 1, \omega_{\text{contra}} = 0.1 \omega_{\text{group}}=0.1$ in equation (9). We will ensure these details are included in the revised version of the paper. Thank you for your careful review and pointing out the missing hyper-parameters.

Question 1: aspect ratio

Thank you for your observation regarding the aspect ratio of the data sample videos. The aspect ratio of figures in the paper is aligned with benchmark datasets such as Hi4D. However, it appears that the sample video had the contact field rendered with a different aspect ratio compared to the other results. We have re-adjusted the aspect ratio and created a new video. Refer to adjusted frame examples in Figure E. of the attached PDF.

Question 2

Here is a detailed explanation of how different people are distinguished using predicted ID values:

1. Region Identification: During inference, the algorithm identifies and marks regions of interest based on the occupancy field, excluding background regions. This step generates both an ID field and a contact field.

2. Clustering: The normalized ID values within these regions are processed using a clustering algorithm, specifically K-Means. This algorithm groups the data into unique clusters, each representing a different person, and includes the contact regions for all individuals.

3. Segmentation: After clustering, each cluster is isolated with a binary mask. A binary mask is then created and smoothed with a Gaussian filter to form a blending mask, ensuring gradual transitions at boundaries. This blending mask is applied to the occupancy field to enhance boundary details. Finally, the marching cubes algorithm generates a 3D mesh model for each cluster from the processed occupancy field.

Instance mesh visualization results are presented in Figure B. of the attached pdf file.

Question 3: local neighborhood

The local neighborhood $N(v)$ is defined as the set of points within a specified radius (contact threshold) around the point $v$, excluding $v$ itself, and is determined using a KDTree [4] for efficient querying. The criteria for marking a point as a contact is based on the presence of neighboring points with differing surface identifiers.

If you have any concerns or questions, feel free to leave comments.

References

• [1] Character Creator 4, https://www.reallusion.com/character-creator/

• [2] NVIDIA Omniverse USD Composer, https://docs.omniverse.nvidia.com/composer/latest/index.html

• [3] NVIDIA Kaolin, https://github.com/NVIDIAGameWorks/kaolin

• [4] Bentley, Jon Louis. "Multidimensional binary search trees used for associative searching." Communications of the ACM. 1975.

Thank you for your valuable question. Below are our responses to raised weaknesses and questions.

Weakness 1: multi-person interaction with SMPL

Thank you for your valuable feedback. In the case of DeepMultiCap (DMC) [1], which uses the SMPL human prior, reconstruction is performed one person at a time. When body parts are occluded, they are masked and included in the input. Using SMPL is particularly helpful in handling occlusions when many body parts are not visible, as it follows the human shape. For areas where body parts are not visible, DMC relies more on the SMPL shape itself than on pixel-aligned features. As the below table shows, the reconstruction performance varies for results from each SMPL method.

In extreme poses or close interaction scenarios, estimating SMPL becomes challenging and requires an optimizing process for accurate estimation, which is time-consuming. Additionally, SMPL has limitations in broadly representing the human shape, especially when children are present in our synthetic datasets. SMPL struggles to accurately represent children, making training difficult. We also considered expanding to interactions with entire scene objects, which raised concerns about using SMPL. Therefore, we aimed to handle occlusions by reconstructing the entire geometry from multi-view inputs and predicting the ID field in 3D space using SRT features. By predicting contact based on ID, our method aims to understand multiple people in close interaction scenarios. Additionally, we created and trained our model on synthetic data with numerous instances of self-occlusion and multi-person interaction using benchmark datasets to ensure robust SRT feature extraction.

• [1] Yang Zheng, Ruizhi Shao, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, and Yebin Liu. DeepMultiCap: Performance capture of multiple characters using sparse multiview cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

Weakness 3: contact information from the output meshes

Thank you for your insightful comments regarding the generation of pseudo ground-truth contacts from output meshes and the proposed variance estimation method for contact detection. Contact label generation relies on geometric proximity and surface identifiers to infer contacts directly from the mesh, while Estimated Contact Fields use statistical variance to detect contacts within the 3D space. To address your comment, we have inferred contacts from the output meshes using the contact label generation approach and measured the precision of our contact predictions. Examples of instance meshes are presented in Figure B in the attached PDF file.

Question 1: the performance of different number of input views

We will report the performance of our model with smaller views before the end of discussion period.

Weakness 2 and Question 2: generalization ability on novel object or different extrinsic camera parameter

To thoroughly evaluate this aspect, we conducted two experiments using our pre-trained model. These experiments included zoom-in and zoom-out tests with the Hi4D dataset, as well as tests on synthetic data featuring four individuals performing extreme poses, such as breakdancing, introducing novel postures and configurations unseen during training with new camera settings. Qualitative results are provided in Figures C and D of the attached PDF. During the zoom-in and zoom-out tests on Hi4D, we adjusted the images and camera parameters accordingly. In the zoom-out scenario, the person's size in the image decreased, leading to a loss of detail in feature extraction. However, our implicit fields prevent significant performance degradation by taking the 3D space bounding box around the person's center, thereby preserving the geometrical results. The qualitative results, shown in Figure C. indicate that the model can handle previously unseen zoom-in and zoom-out scenarios. Notably, the model performed better during zoom-out than zoom-in for ID field prediction, as it relies on extracting features from the entire scene.

Question 3: robustness on occlusion

Our approach addresses occlusions by presenting interaction geometry. We estimate complete geometries from multiple views and use SRT features to predict IDs and contact regions in 3D space, even in occluded areas. By predicting contact regions based on IDs, our model handles occlusions in close interaction scenarios as shown in the first row of Figure B in the attached PDF file. Additionally, we created synthetic data with numerous multi-person interactions and trained our model using this data alongside benchmark datasets to enhance robustness in extracting SRT features.

Question 4: Why not using contact information of Hi4D

To the best of our knowledge, while the Hi4D dataset provides SMPL contact ground truth (G.T.) information, we could not find any contact G.T. information specifically for the 3D scans in the Hi4D dataset. As a result, we generated a pseudo ground truth using the 3D instance mesh provided by Hi4D and used it for our measurements, as detailed in the paper. We appreciate your question and will include a clarification on this point in the revised manuscript to ensure transparency. There may be misunderstandings in our responses, so it would be great if we could continue the discussion regarding any unclear or intriguing parts of our answers or any unresolved questions.

If you have any concerns or questions, feel free to leave comments.

We greatly appreciate your insight and agree that including an ablation experiment to support this statement is crucial for the integrity and quality of our paper.

Weakness 1: missing ablation on grouping loss function

In response to your suggestion, we are currently conducting an ablation study on each term in the grouping loss function in Eq (12). We will report the performance before the end of the discussion period.

Weaknesses 2: more information about the dataset

We present the statistics of the dataset in Figure A. in the attached pdf file. Initially, we acquired characters with various ages and interaction motion sequences such as from Character Creator 4 [1]. Subsequently, we composed scenes featuring multiple characters using Omniverse USD [2] Composer. To facilitate the dataset generation process, we engineered the Kaolin rendering tool [3], incorporating tasks such as multi-person normalization to achieve the desired outputs. As a result, we generated multi-view rendered images, mask images, instance masks, and 3D geometry. Furthermore, we created Normal ground truth maps and joints, although these were not utilized in this paper.

Weakness 3: SMPL close interactions representation

However, I don’t understand why having individual instances of SMPL would complicate the representation of close interactions? I think this is not necessarily true.

In frameworks like DMC that utilize SMPL, the initial SMPL parameters for individual humans are obtained and then optimized to determine the entire scene. In contrast, our approach derives the human pose and shape for the entire scene in a one-shot manner. We intended to mention that using individual instances of SMPL could complicate the representation of close interactions due to the need for separating parameter optimization for each person, which might require complex coordination. In the paper, we will revise to distinguish between SMPL and the models that use it.

I agree that it does not naturally include a representation for interactions, but this could be added to the SMPL model. Now saying that it complicates the representation may be too much?

We understand your point about the potential benefits of integrating the SMPL model for multi-person interaction geometry. Applying multi-person interaction geometry to the SMPL model is a reasonable approach that can enhance interaction modeling. While existing methods [4,5,6] optimize SMPL pose estimation for multi-person scenarios, a potential limitation is the need for multiple optimization steps. We believe that leveraging a representation for interactions can effectively optimize the SMPL parameters and address these configuration and optimization challenges. This approach can provide a more accurate and computationally efficient solution for modeling close interactions between multiple individuals.

Weakness 4: baseline models

For the baseline models discussed in Section 4.3, we employed different SMPL acquisition methods tailored to each dataset. Specifically, for the HI4D dataset, we used MVPose, aligning with the experimental setup described in the HI4D paper. In their experiments, DMC was run on the HI4D dataset using MVPose for SMPL acquisition, and we adopted this approach to ensure consistency and comparability. However, we encountered challenges when using MVPose for the synthetic dataset, as its modules were trained on real data, resulting in less accurate SMPL estimations for synthetic data. To address this, we utilized Learned Vertex Descent (LVD) for the synthetic dataset. LVD is designed to fit SMPL to a 3D human model (3D scan) and has proven to provide more accurate results in this context. While LVD is originally intended for single-person scenarios and may be sensitive to occlusion, it was chosen for the synthetic dataset due to its superior performance in fitting SMPL to 3D scans, which is crucial for achieving accurate results in our study. To ensure the robustness of our data and fairness in our experiments, we will report the results trained with MVPose on the synthetic data.

We will revise our paper to include these detailed results and analyses during the rebuttal for providing comprehensive evidence on the proposed framework.

Weaknesses 5: minor details

Thank you for your detailed feedback on our manuscript. We acknowledge the points you raised regarding notation, phrasing, and citation formats. We will address them meticulously in our revisions.

If you have any concerns or questions, feel free to leave comments.

References

• [1] Character Creator 4, https://www.reallusion.com/character-creator/

• [2] NVIDIA Omniverse USD Composer, https://docs.omniverse.nvidia.com/composer/latest/index.html

• [3] NVIDIA Kaolin, https://github.com/NVIDIAGameWorks/kaolin

• [4] Dong, Junting, et al. "Fast and robust multi-person 3d pose estimation from multiple views." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

• [5] Zijian Dong, Jie Song, Xu Chen, Chen Guo, and Otmar Hilliges. Shape-aware multi-person pose estimation from multi-view images. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

• [6] Yuxiang Zhang, Zhe Li, Liang An, Mengcheng Li, Tao Yu, and Yebin Liu. Lightweight multi-person total motion capture using sparse multi-view cameras. In Proceedings of the 17026 IEEE/CVF International Conference on Computer Vision. 2021.

Thank you for your valuable feedback and for recognizing the contributions of our work.

First, we will revise our paper to discuss related works including single-view and multi-view settings mentioned in [1, 2, 3]. We emphasize that the mentioned methods [1, 2] primarily focus on single-view reconstruction not multi-view reconstruction. We designed our overall architecture with multi-view settings in mind from the very beginning to handle occlusion challenges in multi-person interaction cases. A multi-view approach fundamentally incorporates triangulation, making it more robust to unseen scenarios and significantly improving frame-to-frame consistency compared to single-view settings.

Additionally, we understand the importance of providing information on computational costs. Here are the details of our computational resources:

• Training Time: Our model was trained for approximately 2 days with 2 NVIDIA A100 GPUs.
• Testing Time: Each test instance takes around 60 seconds on the same GPU. We will include these computational cost details in the revised manuscript to provide a comprehensive understanding of our approach.

We appreciate your thorough research on relevant works and your interest in comparative analysis. If you have any concerns or questions, feel free to leave comments.

References

• [1] Cha, J., Lee, H., Kim, J., Truong, N. N. B., Yoon, J., & Baek, S. (2024). 3D Reconstruction of Interacting Multi-Person in Clothing from a Single Image. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 5303-5312).

• [2] Mustafa, A., Caliskan, A., Agapito, L., & Hilton, A. (2021). Multi-person implicit reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14474-14483).

We sincerely appreciate the valuable feedback and discussion throughout the rebuttal and discussion period. Based on comments and discussions, we will revise the current manuscript to strengthen and clarify our framework and explanations. Below, we present additional discussion points addressed during this period.

Additional experiments during discussion period

As previously mentioned, we have conducted additional experiments with fewer views to address, reviewer Vhig's question about generalization ability and perform an ablation study in response to reviewer Ar19's feedback.

Firstly, our ablation study has yielded promising results, showing that integrating both squared distance and exponential penalty terms into our grouping loss function enhances accuracy and consistency. This outcome not only validates our approach but also directly responds to the feedback provided by reviewer Ar19.

Additionally, we have addressed reviewer Vhig’s concerns about generalization ability by performing experiments with fewer views. Our findings indicate that our method consistently outperforms DMC across all metrics—Chamfer Distance (CD), Point-to-Surface (P2S), and Normal Consistency (NC)—in both 4-view and 8-view settings.

Explanation on Figures in Attachment

We missed providing detailed explanations for the meaning of each figure in the attachment during the rebuttal. We would like to address that before the end of the discussion period.

• Figure A illustrates the diversity within our proposed dataset, highlighting variations in age, gender, and scene composition. To further enrich this diversity beyond what is seen in Hi4D, we generated synthetic data that includes individuals with varying ages, heights, weights, and garments.
• Figure B displays the instance mesh visualization results produced by our interaction geometry.
• Figure C presents qualitative results from zoom-in and zoom-out experiments, demonstrating that our implicit fields generalize effectively to unseen camera parameters. Our model handles these scenarios well by utilizing a 3D bounding box centered around the individual, preserving the geometry. It particularly excels in zoom-out scenarios for ID field prediction, as it can draw on features from the entire scene.
• Figure D highlights our model’s generalization to new subjects and configurations. The experiments shown in Figure D of the attached document indicate that our model performs robustly with entirely new scene configurations, including new subjects, extreme poses from breakdancing sequences, and new camera settings with varying resolutions and parameters. These results confirm that our model generalizes effectively beyond the synthetic dataset it was trained on.
• Figure E shows adjusted frame examples from our data sample video, ensuring consistent visualization.