CoMotion: Concurrent Multi-person 3D Motion

Qualitative videos: We appreciate the feedback about the qualitative videos. We had hoped the accompanying HTML page would help compare results side-by-side, but before any release we will take that into account and prepare a clearer demonstration of our method’s results!

Benchmarking on 3DPW and MuPoTS-3D: Thanks for the suggestion. We focused our efforts on PoseTrack since although it does not provide ground truth 3D annotations we can still evaluate 2D pose estimation quality and PoseTrack offers extremely challenging dancing and sports scenes with large crowds that test the limits of our pose estimation and tracking model. In contrast, MuPoTS, for example, generally consists of scenes of only 2-3 people typically walking around with a fixed camera. In our controlled experiments we made sure to also include quantitative evaluations on EgoHumans, which offers similar benefits to 3DPW/MuPoTS with a fair number of serious occlusions in its scenes.

Skeleton modalities: How we handle skeleton modalities is a great question, and one that took up a considerable amount of time and attention when iterating on this project. We have added a section to the appendix providing more details. But to briefly summarize:

The 2D keypoints in the COCO format are placed at different positions than the corresponding key points in the SMPL skeleton. This is noticeable in the hips where COCO-style annotations place the hips near the outer edges of the body higher along the waist line while SMPL puts the hips roughly at their anatomical joint position attached at the pelvis. Other subtle differences exist for ankle and shoulder joints. And a particularly challenging case is the handling of face keypoints, where it can be difficult to have the SMPL mesh fit the head such that it matches 2D-annotated eye, ear, and nose keypoints.

We explored several approaches to handling the discrepancies. A common way of dealing with this discrepancy is to define linear regressors from SMPL mesh vertices out to joint locations. We attempted to use a predefined regressor from the SMPL mesh out to COCO-style keypoints as well as attempted to learn a custom regressor during training. We also investigated a simpler choice which was to modify the loss. Specifically we down-weighted the 2D reprojection losses of keypoints with a greater mismatch between annotation styles (e.g. face + hips). This was inspired by the benefits TokenHMR demonstrated with their threshold loss. In the end, we ended up not using any of these approaches, and instead manually narrowed the hips on the COCO-style annotations, and treated everything else the same as this worked best. One consequence is that on the pseudo-labeled datasets the 3D losses are applied on the pseudo-SMPL keypoints while the 2D reprojection loss is on the provided 2D keypoints which don’t necessarily match. Interestingly, 3DPW seems to have followed a similar strategy in their annotations where the SMPL hip keypoints match up with the detected 2D keypoints narrowed to half their original width.

“Ignoring” people: In general we do not find that the model treats tracks instantiated later any differently from tracks instantiated at the beginning. As long as we provide a box to initialize a particular person, the model will predict the pose for that person and follow them. That said, one weak point in our tracking heuristics is instantiating a new track for someone who appears from behind an existing track. In this case, it is not until the person is sufficiently separated that they are likely to get instantiated.

Mesh shape: Thank you, we revised this statement in the paper. We rely on supervision from pseudo-GT SMPL annotations which were not built with the intention of correctly matching body shape/size. An example for this is SMPL’s modeling of infants/children, which are typically predicted with the height and body proportions of an adult body. A good investigation into this is provided in the concurrent CameraHMR (https://arxiv.org/abs/2411.08128).

We nevertheless predict the SMPL shape parameters as they determine bone lengths, which affects the model’s 3D pose accuracy. We do however not expect remaining shape properties like girth or relative proportions to be modeled correctly by the SMPL shape parameters and consequently do not expect to be competitive with any method trained for accurately estimating body shape.

Intrinsics: Inference requires an input intrinsics matrix. If no ground truth is provided, we fall back to a reasonable default. As this default may not match the actual (missing) ground truth, we augment input matrices during training to increase robustness. We perform an analysis on PoseTrack PCK accuracy as a function of the assumed focal length and find that the network maintains good 2D accuracy across the full range of focal lengths seen during training. This can be found in Appendix A.2.5. We clarified in the revised paper.

Typos/missing details: Thanks for the attention to detail! We have updated the draft to address these comments.

CHI3D: Thanks for the suggestion! Unfortunately, we were not able to integrate and benchmark on this dataset within the rebuttal time frame.

Off-the-shelf MAE: This is the masked-autoencoder pretrained version of the model, we have updated the writing to clarify the unexplained acronym.

Pose prior: Thanks for the suggestion! We previously experimented with TokenHMR as a stronger pose prior, but did not observe a noticeable improvement. We find the integration of a generative model an interesting topic that deserves a more thorough investigation. As this goes beyond the scope of the paper, we leave it for future work.

Unclear architecture details: We have added a more thorough description of the architecture in the appendix. We also plan to release inference code for the model and pretrained weights for reproducibility.

Hidden state: We conducted an experiment on the effect of the hidden state on performance (Sec. A.1.2). By removing the hidden state we observe noticeably worse performance on challenging PoseTrack scenes and an increase in ID switches by 25%. The effect is smaller on 3DPW validation scenes which may be explained by the fact that most scenes only involve one or two people, where one can assume that most transitions from adjacent frames can be handled trivially without the more sophisticated modeling that the hidden state might provide.

Multiple datasets: One of the key motivations of our efforts was to design and build an approach that could be trained on video. Other approaches avoid this by breaking down the problem into sub-problems and training individual modules. We demonstrate the benefits of building a single, learnable stack that can be unrolled across video frames performing the core problems of pose estimation and tracking together. As such, we make use of training data not leveraged by prior work. That said, we would like to highlight that there is substantial overlap in datasets, particularly large scale data like InstaVariety which consists of 3 million images. One of our key video datasets is PoseTrack, and 4D Humans trains on MPII which consists of individual images from the same data source. The main dataset that differs is BEDLAM for which we provide an ablation showing that it is not as critical for maintaining good 2D pose tracking performance on PoseTrack but is important for preserving good 3D estimates unrolled across time (Appendix, Sec A.2.2).

Unstable pose estimates: We hypothesize that this behavior could be due to noisy 2D supervision in our video training data. As 2D annotations can be explained by multiple 3D poses, there is limited supervision on 3D pose consistency across frames.

Strict threshold: This is the threshold on detection scores above which we accept detections from the bounding box model. A higher threshold reduces the chances of including background figures and smaller or more unusual poses. The detections that are left are generally clearer and easier to follow, hence the improved precision. By default we use a threshold of 0.3. For the strict setting we use 0.7.

Performance versus CLIFF: The performance on 3DPW is curious. As a point of comparison, we would like to draw the reviewer’s attention to the respective numbers for CLIFF, HMR 2.0a, and HMR 2.0b on 3DPW. The authors of 4D Humans observed they could train a model that was competitive on 3DPW (HMR 2.0a) but not as good in-the-wild, and with longer training they had a more robust model (HMR 2.0b) that did worse on 3DPW. One possibility to explain this is annotation discrepancies. In-the-wild 2D annotations are fairly different from SMPL annotations, and it is possible that as a result of fitting both during training, we perform worse on 3DPW. In our response to Reviewer VXUb we discuss some investigations we undertook to deal with annotation discrepancy. Occasionally this resulted in models with better 3DPW performance, but we did not pursue this further and instead focused our efforts on in-the-wild numbers. We suspect a large-scale consistently-annotated dataset would go a long way to resolving this issue.

CoTracker: Dense point tracking methods like CoTracker are a fascinating line of work, and we are excited about the possibilities they enable. We find however setting up a fair comparison nontrivial. For instance, dense tracking approaches are typically instantiated from pixel positions, whereas many of the body joints that are tracked don’t necessarily correspond to visible surface positions (e.g. the hips). For a fair comparison, all keypoints of the pose would have to be visible in the first frame for the dense tracker, which strongly limits the type of videos for evaluation. One advantage of tracking a pre-defined discrete representation is that many downstream use cases are made easier working directly with an output such as kinematic joint angles which would be difficult to parse from a set of point trajectories.

Contributions & recent methods: We appreciate the feedback! We believe we are one of the first methods to train an online 3D multi-person pose system that handles crowded in-the-wild scenes with diverse and challenging poses.

• TRACE (Sun et al.) also predicts multi-person 3D poses in videos, but has not been demonstrated to work on scenes with more than 2-3 people.
• The only approaches that benchmark on the full target setting of our method are PHALP and its follow up 4D Humans. Both methods take a fundamentally different approach - they predict individual 3D poses on cropped bounding boxes and link them together into tracks. In contrast, CoMotion consists of a unified model that explicitly is trained on video to reason about all people in the scene simultaneously and model their motion through time. In order to achieve this, we push tracking-by-attention into a much more difficult setting modeling a richer signal (3D pose) in camera coordinate frame, training on longer sequences rather than pairs or small handfuls of frames, and modeling through occlusion. This is only possible with a nontrivial combination of datasets and insights around how best to leverage single image and video data with 2D and 3D annotations. The scale and scope of this work unifies several fields (tracking, 3D pose estimation, human motion modeling) and we believe this undertaking and the results are a significant contribution in their own right. We elaborate below on some of the differences to the bounding-box tracking literature.

Contrast to other tracking-by-attention methods: CoMotion targets multi-person pose estimation in 3D and tracking jointly. Tackling this task poses several additional challenges which are typically not handled by existing bounding-box tracking-by-attention approaches.

• CoMotion differs from prior tracking-by-attention approaches in its handling of state. MeMOTR in particular does not make predictions for unconfident boxes and leaves occluded queries idle in case they are relevant again. CoMotion in contrast is trained to keep making pose predictions even if the person is occluded or out-of-bounds.
• A key contribution of our approach is the multi-stage curriculum across a suite of datasets, which starts on single images and ramps up training on longer video sequences. This also allows us to take advantage of the strengths of the various training datasets at our disposal, and considerably improves performance with each stage of the curriculum as demonstrated by the ablation experiment in Sec. A.2.3. We are not aware of any prior work supervising bounding-box trackers that would use a similar curriculum.
• Making predictions in a 3D coordinate frame requires explicitly accounting for camera intrinsics such that results are projected correctly to the image. This is in contrast to bounding-box trackers that only operate in the image plane.
• Ground truth data for multi-person 3D pose tracking is scarce. Most datasets only provide a limited set of annotations, which do not provide full supervision for the task or not at the required level of diversity to generalize to in-the-wild scenes. To nevertheless build a system that can operate on complex in-the-wild videos and predict 3D poses of multiple people, we harvest several datasets that contain mutually inconsistent annotations. We added further details in Sec. A.4 of the Appendix.

Subtle pixel cues: The qualitative results included in our introductory figure serve as examples of this behavior. Unfortunately, heavily occluded figures are seldomly annotated in benchmark datasets, which prevents adequately quantifying the phenomenon. Metrics and ablation discussion: We have updated the paper with more details, thanks! We have added brief explanations of the metrics as well as an additional section in the main paper summarizing the results of our controlled experiments.