Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation

We thank Reviewer R9xH (R4) for the assessment and questions. R4 considers the idea "interesting and novel" and finds that we do "a good job in explaining" the motivation and technical details. R4 further sees the performance as "impressive" on "extensive benchmarks and experiments".

Lack of discussion on inference speed. [...] whether this method is suitable for real-time inference.

The method is suitable for real-time inference. NLF-S has a batched throughput of 410 fps and unbatched throughput of 79 fps on an Nvidia RTX 3090 GPU. For NLF-L these are 109 fps and 41 fps respectively. (Bounding box detection needs to be performed on top of this, but fast off-the-shelf detectors are readily available.)

Note that NLF’s inference-time overhead (for predicting weights through the field MLP) can be eliminated by precomputing the weights once for a chosen set of canonical points. (Typically one wants to localize the same points, i.e. same skeleton formats, for many images.) For reference, in case of NLF-S with no image batching, and about 8000 points to be predicted (mesh vertices and skeletons), forwarding the field MLP to obtain convolution weights takes 7.7 ms, while the rest of the network including the backbone takes 12.7 ms. For NLF-L with batch size 64 the latter takes 587 ms, making the MLP cost negligible in comparison even if we do not precompute it. We will add this information to the paper.

How does the model do on small details such as fingers?

For this, we refer to our AGORA results on hand keypoints (Table 5, RHan and LHan columns), where we achieve second-best results. However, given our focus on body pose and shape, most of our training datasets do not contain detailed finger annotations, and hence we can consider the second-best results obtained for this subset of keypoints still a strong result (see also our answer to R3.)

We thank Reviewer YzeS (R3) for the review and questions. R3 considers that our "insight is simple, clearly explained and well motivated", and finds the quantitative metrics "impressive" and "convincing".

Its not clear if the performance is derived from just a few data sources or all of them i.e. how each dataset affects results.

While an extensive ablation for assessing each individual dataset's contribution is computationally not feasible, we provide ablations for using only synthetic or only real 3D data. Please refer to the global answer regarding this.

Extent of generalizability: Usual suspects for human pose&shape estimation failures/limitations are loose clothing, occluded views, unique poses. It would be nice to see how the method works on such cases and if the method generalizes well to such cases.

We include further qualitative examples in the attached PDF that cover such cases. (Such challenging examples are rare in quantitative benchmarks, hence the qualitative examples.)

lower accuracy for face and hands in Table 5

Although our focus throughout the paper is mainly on body pose and body shape, we still achieve second-best scores for hands and faces in Table 5 (AGORA benchmark). It is true that many of the training datasets do not provide detailed annotations for hands and faces, and we indeed attribute our lack of SOTA results on hands and faces to this property of the training data.

different dataset scales which might be widely off

We adjust for different dataset scales (i.e. different number of training examples in each) by sampling more training examples from larger datasets. We did not tune these sampling-proportion hyperparameters, as this can lead to combinatorial explosion and would be very resource-intensive.

Its not clear what the canonical space representation is.

The canonical space is defined in reference to a T-like pose of the default SMPL mesh. All other keypoints (as e.g. shown in the referenced Fig. 4 column 1) are represented in this same coordinate system, as points within this canonical human volume. Again, no explicit conversion between formats is necessary and the exact locations of points in the canonical volume are tuned automatically during training. We will include this important point in the paper, and we thank R3 for calling attention to this omission.

We are glad that R1 sees our idea as "novel", further acknowledging the novel aspect of putting together such a "super-dataset". R1 further praises the model quality as "strong across the board" and foresees significant community impact.

Importance of the hypernetwork and datasets: see the global answer.

Complicated explanation of localizer field, simply a hypernetwork: our description emphasizes the connection to neural fields, given the similar 3D-spatial input and the use of positional encodings. However, we agree that the hypernetwork view is also important. While we do mention this connection and cite the HyperNetworks paper (Ha et al., 2017), we will extend the manuscript with further connections to other relevant uses of hypernetworks.

how datasets with 2d and 3d annotations are used for training? How does the "approximate initialization" work? Is this done by fitting SMPL to the 3d points[...]?

We will make our existing explanation around L197-214 clearer. Importantly, our formulation allows us to sidestep the generally ill-posed problem of converting annotations to a single format, such as SMPL. Each dataset with 3D joint annotations can have different skeleton formats, i.e., the joints may designate different anatomical locations, e.g. Human3.6M's shoulder point is not the same as the shoulder point of the MPI-INF-3DHP dataset. We directly train the network with points that are annotated for a training example, i.e. we query those points for which we have annotations, so we can compute and minimize the average loss for those points. For this, we need to know where to query the field, e.g. where the Human3.6M shoulder point is in the canonical space. (For SMPL this is given, since the canonical space is derived from a SMPL mesh of mean shape and a T-like pose). Training can be started with an approximate placement of e.g. the Human3.6M shoulder point in the canonical human volume in the general shoulder area, and we let the gradient-based optimization update all parameters jointly, including the backbone parameters, the neural field parameters and the 3D query location of each skeletal format in the canonical space. Datasets with 2D point annotations are treated similarly, except that the prediction is projected before computing the loss in 2D.

bold and counterintuitive claim: that by posing the task of 3d human pose estimation as 3d registration (a more complicated task), it is possible to achieve better 3d poses than SOTA. Furthermore, this is achieved by exploiting data that is not annotated for 3d registration

We do not make such a “bold claim” in the paper, instead our claims are as follows. We are able to jointly train on heterogeneous data sources, some annotated for 3D/2D skeleton pose estimation with different formats as well as some for 3D human mesh recovery (also different ones, SMPL/SMPLX/SMPLH male/female/neutral), obtaining a single model that achieves SOTA on both kinds of tasks. Our work enables this common treatment by designing a generalist model that can estimate any arbitrarily chosen points, and then by casting each task as a point localization task (with different point sets) that are simple to tackle with the proposed generic point localizer. The end goal is to obtain a strong model for body pose and shape estimation, that is test-time configurable for user-chosen skeleton formats and mesh formats. Our model is designed to make spatially smooth predictions w.r.t. the selected points, resulting in consistent predictions for the different output formats.

The supplementary material discusses the creation of a large synthetic dataset using SMPL fittings of the DFAUST dataset, which is not mentioned in the abstract or the paper -- how important is this for the overall results?

Individually ablating the effect of each dataset is computationally infeasible. However, we provide experimental results with using only synthetic or only real 3D-annotated training examples, see the global answer.

dimensionality of the volumetric heatmap? Is this depth defined over the entire scene or only over the depth of the human body? Why does the architecture predict a 2d and a 3d heatmap? Is it possible for the 2d heatmap to disagree with the projection of the 3d prediction?

The heatmap does not cover the entire scene but a cube of side length 2.2 meters around the human. The 2D and 3D heatmaps are used in order to estimate the human scale and distance from the camera. They could theoretically disagree, but we did not observe such a problem in practice - training them together results in compatible predictions.

The last two layers of Fig 6 show FC layers going from 1024 to 384 dimensions, and later going from 1024 to 384 again. Is this a typo? If so, what does the actual architecture look like?

The architecture was designed like this to express the Global Point Signature-based positional encoding, which is inspired from [72] in 2D surface modeling. The 1024-dimensional vector is hence initially trained to output the Global Point Signature. As explained in the paper from L268, we found it best to finetune the full MLP after this initialization. Without this special pretraining, the two linear layers could indeed be replaced by a single layer without losing expressivity.

The paper uses the number 384 several times in seemingly unrelated areas

There is no special connection there, the reason is simply that 384 is the sum of two high powers of 2 (256+128) and hence a "round number" in binary. Tensor sizes divisible by high powers of two are often more convenient and hardware-efficient in practice.

What is the time it takes the official SMPL fitting code to achieve an error comparable to the one achieved by the proposed method?

Please refer to Table 2 in the PDF. The official code runs for about 7 minutes (for all 33 samples included with the code) achieving an average error of 8.0 mm, while our method achieves 7.8 mm in just 13 milliseconds.

We thank Reviewer M7cf (R2) for the suggestions. R2 notes that we performed "extensive ablation studies", which are seen as "helpful" for future model design decisions, and further praises our strong qualitative results with good pixel alignment.

If we use the same training dataset but remove the point-based representation, will the results be similar? To what extent does this new point-based representation help?

Please refer to the global response regarding the role of the neural-field-based point querying.