Incorporating Test-Time Optimization into Training with Dual Networks for Human Mesh Recovery

Question 1: The biggest concern is that the performance improvement is not significant when compared with the existing method. The comparison of 3DPW is not fair, since the results of the last two lines use 3DPW for training, while the reported PLIKS does not use 3DPW for training. PLIKS trained on 3DPW can achieve 40.1 PA-MPJPE and 73.3 PVE, which greatly outperforms the results in this paper.

Our Response: Thanks very much for the comments. We have checked the results of PLIKS again. After the checking, we confirm that our comparison is fair, and our results outperform those of PLIKS.

For convenience, we copy the following content from Table 1 of the paper of PLIKS (see https://openaccess.thecvf.com/content/CVPR2023/papers/Shetty_PLIKS_A_Pseudo-Linear_Inverse_Kinematic_Solver_for_3D_Human_Body_CVPR_2023_paper.pdf).

From caption of Table 1 of PLIKS: PLIKS+ means that the network was additionally trained with 3DPW. PLKIS++ means that the network was additionally trained with 3DPW and AGORA.

In our paper, we compare with PLIKS+ which is trained with 3DPW too. So, from the aspect of training dataset, our comparison is fair.

As for the results of 40.1 (PA-MPJPE) and 73.3 (PVE) of PLIKS++, they are obtained by further finetuning PLIKS on 3DPW and Agora. Since our method is not finetuned with Agora, we are thus more appropriate to compared with PLIKS+ instead of PLIKS++.

However, we find there is a typo in our paper. That is, we write that PLIKS adopts the backbone of HRNet-W48, but actually it uses HRNet-W32. This makes it somewhat unfair between PLIKS and our method. Therefore, we make the following fair comparison, which also shows our method outperforms PLIKS.

Question 1: In Fig. 3, Why EFT gets worse and which design of the proposed approach alleviates this issue?

Our Response: This is probably because EFT is only finetuned with 2D reprojection loss and is more sensitive to the errors in the 2D joints. At the first several optimization steps, the estimated 3D SMPL approaches the 2D joints from a relatively distant initialization, therefore the result gets better gradually. However, with more optimization steps, the SMPL may overfit the 2D joints whose annotation-errors then distort the SMPLs, thus yielding worse evaluation metrics.

In contrast, our method is guided by both 3D and 2D supervisions. The 3D pseudo SMPLs plays the role of regularization that mitigates the influences of errors in 2D joints.

Question 2: Fig. 1 is a bit difficult to follow. Reorganize and simplify Fig. 1.

Our Response: Thanks for the suggestion. We have revised Fig. 1. Please see the new figure in the attached PDF document.

Overall, we think the difficulty in reading Fig. 1 comes from the fact that the illustrations of SMPL meshes and 2D joints in the testing and training losses are free of text descriptions. In the new figure, we increase space between losses, and add descriptions for the inputs of the loss functions.

Question 3: Provide statistical significance for experiments as suggested by the NeurIPS Paper Checklist 7.

Our Response:

Thanks very much for the reminding. During this rebuttal period, we have conducted experiments to include the standard errors for the main results. Please see the above tables. The above results are obtained by running the test-time optimization 5 times using different initializations. More statistical experimental results of both training and testing for the main and ablation experiments will be added in the revised paper finally.

Question 4: Although two failure cases are provided, more are encouraged to give a comprehensive overview.

Our Response: Thanks very much for the suggestion. We have added more failure cases. Please see the attached PDF document for more information.

Question 1: Rephasing “key observation”, “rethinking”, “proposes”, etc., as Kim et al. 2022 have used meta learning already.

Our Response: Thanks very much for the suggestion. Yes, Kim et al. 2022 also adopted meta learning, as referenced in our paper. The difference is that their method is a direct extension to EFT, while we take a different perspective from the Human Mesh Regression model. Please see the difference between Kim et al. 2022 and our method in the response to Question 3.

Following your suggestion, we will rephase the above presentation to credit the contribution of Kim et al. 2022 more appropriately.

1. “The key observation in this paper is that the pretrained model may not provide an ideal optimization starting point for the test-time optimization” will be changed to:

“However, the pretrained model may not provide an ideal optimization starting point for the test-time optimization”.

2. “The above analysis motivates us to re-think the test-time optimization problem in the framework of learning to learn, i.e., meta learning. Inspired by optimization-based meta-learning [12], we incorporate test-time optimization into the training process” will be changed to:

“Based on the above analysis and inspired by Kim et al. [22], we incorporate the test-time optimization into the training process, formulating the test-time optimization in the framework of learning to learn, i.e., meta learning [12].”

Besides the above modifications, we will add analysis about the difference between Kim et al. 2022 and our method, and revise relevant places of the paper accordingly.

Question 2: Does Algorithm 1 compute second-order derivatives like MAML? Explain why “simulating” this meta learning approach in Line 153.

Our Response: We do use the MAML algorithm. However, the original MAML needs to compute second-order derivatives which is too much time-consuming and memory-cost. We thus adopt FOMAML instead, a simplified version of MAML suggested by the Finn et al. [12] which discards second-order derivatives.

To clarify, there is a typo in Line 7 of Algorithm 1, where the gradient should be computed with respect to w instead of w’, as indicated by Reviewer PPKd.

Therefore, we confirm that the statement of our algorithm “simulates” MAML in Line 153 is correct.

Question 3: Quantitative comparison with Kim et al. (2022). Highlight the difference.

Our Response:

Please see the above quantitative comparisons. Due to time limit, we collect the result from Kim et al. 2022 for the comparison.

1. Backbone: the backbone of Kim et al. 2022 is SPIN, which is a stronger backbone than ours HMR.
2. Training dataset: the training datasets are nearly the same except that Kim et al. 2022 additionally uses LSP while we do not.

As can be seen, the PA-MPJPE of Kim et al. is 57.88 which is worse than ours 44.3, indicating that our method performs better than Kim et al. 2022.

Now, we elaborate the key difference between Kim et al. 2022 and our method in detail.

The key difference is that that Kim et al. 2022 use the 2D reprojection loss only (please see Eq. 1 in their paper) in both inner and outer loops of meta learning, while our method uses 2D and 3D losses. This is a small difference in formulation, but a large difference in contextualization.

EFT is performed under the guidance of 2D reprojection loss, and Kim et al. (2022) extend EFT to both the inner and outer loops of meta learning. In this sense, the method of Kim et al. (2022) is a direct extension of EFT to meta learning.

In contrast, our method considers meta learning from the perspective of the complete HMR model trained with 2D and 3D losses. In other words, we use the complete HMR model in both inner and outer loops of meta learning. This is more reasonable, as our aim is to personalize the HMR model on each test sample, rather than a model trained with only 2D reprojection loss.

The above difference enables us to design a dual-network architecture that is very different from the network used in Kim et al. 2022. Besides, the introducing of 3D losses improves the HMR quality significantly, as shown by the above quantitative comparisons and those in the paper.

Question 4: Line 251, add quantitative results to support the OOD claims.

Our Response: Thanks very much for the suggestion. We have added quantitative comparisons. Due to space limit, please refer to our answer to Question 3 of Reviewer PPKd (i.e., Reviewer 1) for more information.

Question 5: Place the backwards sum notation in Figure 1 on the left side of the summands.

Our Response: Thanks for the suggestion. We have revised Figure 1. Please see the attached PDF document for the modified figure.

Question 6: Move discussion of limitations in Appendix to the main text.

Our Response: Thanks a lot for the suggestion. We will move them to the main text as suggested.

Question 1: There remains a discrepancy between the training loss (using ground truth labels) and the testing-time loss (using pseudo labels).

Our Response: Thanks a lot for the insightful question which deserves further interpretations.

The design of dual networks is one of the core contributions of our method. We use the dual networks to unify the formulations of training and testing objectives, but the content of them cannot be completely unified, as one uses GT labels and the other uses pseudo labels.

We interpret this mismatch and provide analysis as follows.

1. There are 3D losses in the training objectives, while there are not at test time. We find this yields a gap between the training and testing optimizations, which motivates us to design the auxiliary network to generate pseudo 3D labels to mitigate the gap. Although there still exists discrepancy between GT and pseudo labels, the gap is greatly reduced compared to using only 2D loss without introducing the pseudo labels. The experiments in Table 3 validates this key design of our method.

2. In our system, the auxiliary network in combination with the main network is trained simultaneously using our proposed training strategy. Through this process, the auxiliary network learns to estimate pseudo 3D meshes that are effective for optimizing the network during test-time optimization, which guarantees to a certain extent the alignment between the gradients from ground truth labels and pseudo labels.

Question 2: Eliminate ensembling effects: include results of EFT_CLIFF with two networks (averaged results).

Our Response: Thanks a lot for the valuable comment. Adding this experiment can further validate the meta-architecture of our method.

Following your suggestion, we use two CLIFFs to generate SMPLs and compute the averaged SMPL of them, based on which we conduct EFT optimization which in turn optimizes the two CLIFFs. This process is iterated for 20 rounds. The finally obtained CLIFFs and their averaged result is used for comparison.

The comparison results are shown in the following table.

The row of EFT_2CLIFFs shows results of using two CLIFFs in EFT. It shows that this really works, indicating there indeed exists ensembling effects. However, although the two CLIFFs improve EFT, the improved results are still much worse than ours, proving that the ensembling effect by meta-learning outperforms that by simply using two CLIFFs.

We will add this new comparison into Table 3. Thanks again for the great suggestion.

Question 3: A quantitative comparison with EFT on the LSP-Extended dataset would provide a more comprehensive validation of the method's effectiveness in OOD scenarios.

Our Response: Thanks a lot for the suggestion. We perform the following quantitative experiments to validate the OOD effectiveness of our method compared with EFT.

First, we perform a quantitative comparison with EFT on the LSP-Extended dataset. Since LSP provides GT 2D joints but not GT SMPLs, we compare the 2D loss (measured w.r.t. the GT joints) with EFT.

Training dataset: COCO, MPII, MPI-INF-3DHP, Human3.6M, 3DPW
Backbone: HR-W48

Our method approaches the GT joints more nearby than EFT, i.e., 6.1e-3 (ours) vs. 8.3e-3 (EFT).

The calculation method for the 2D loss is as follows: Both the predicted values and GT joints are transformed into the range [-1,1] using the formula 2*y/crop_img_height-1 (similar for the width). Then, the mean squared error (MSE) loss is calculated.

Second, to further evaluate the OOD effectiveness of our method, we train our method and EFT on COCO (an outdoor dataset), and test them on Human3.6M (an indoor dataset).

Training dataset: COCO
Backbone: Res-50

Since there are GT SMPLs, we report MPJPE and PA-MPJPE. As can be seen, our results are also better than EFT.

Finally, The above quantitative experiments will be edited into the main text of the revised paper, to validate the OOD effectiveness of our method with quantitative evidences.

Question 4: In line 7 of Algorithm 1, why use the gradient w.r.t. w′ instead w?

Our Response: Thanks for the careful checking. We are sorry that this is a typo. We do use the MAML algorithm. Specifically, we use FOMAML that discards the second-order derivatives. So, w’ should be corrected to w. We will correct this typo in the revised paper.