ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization

Thanks for recognizing our SOTA performance, and the potential for pseudo-label generation. We truly appreciate your constructive comments and address them below.

Q1. Clarification of claims

Thank you for pointing this out. We would like to clarify that: end-to-end diffusion models predicting from noise typically avoid reprojection loss, as early denoising steps yield unrealistic poses, making such loss ineffective. CloseInt also notes: “Early iterations are meaningless for human motion.”

In contrast to ScoreHMR and CloseInt, which are two-stage (using external pose estimators before diffusion), our method is purely end-to-end, sampling from random noise. This avoids dependency on potentially biased external estimators and improves the robustness of in-the-wild (ITW) data. Visual comparison with ScoreHMR:

https://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/mE1y_Q1.pdf

We will revise the phrasing in the final version.

Consistency with the input image. Our method directly learns 3D human pose distribution conditioned on images without needing an explicit camera model or reprojection loss in training. We follow ScoreHypo to predict UVD joints used for mesh reconstruction and scoring. At inference, we recover 3D joints from UVD using a full-perspective projection with estimated intrinsics.

As TokenHMR notes, minimizing 2D error can harm 3D accuracy due to imprecise camera parameters, even with full-perspective cameras. Instead, we ensure consistency implicitly via HMR-Scorer, which learns alignment using pixel-aligned image features. On clean data, scoring by 3D error naturally favors well-aligned results; on ITW data, the scorer transfers 2D alignment knowledge. Additional ITW results are provided at:

https://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/8ugF_Q1.pdf

Q2. Differences with HypoNet

We introduce a preference optimization framework to improve image consistency and ITW robustness. Rather than proposing a new diffusion model, our method focuses on finetuning existing diffusion-based HMR models like HypoNet.

Q3. How to ensure accurate in-the-wild scoring

To address limited and noisy outdoor data, we train HMR-Scorer on large-scale synthetic datasets (e.g., BEDLAM, GTA-Human) for effective knowledge transfer. As noted in BEDLAM, HMR models trained solely on synthetic data can achieve SOTA accuracy. Table 1 in the paper shows strong results on GTA-Human (with ITW scenes). HMR-Scorer visualization is at:

https://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/mE1y_Q3.pdf

Q4. About experiment comparison

We updated Table 2 to indicate which methods are finetuned on the target benchmark:

Updated Table 2: https://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/mE1y_Q4.png

We added results showing:

• Our ADHMR outperforms finetuned HMR 2.0a.
• Prior works (e.g., HMDiff, Zolly) also finetune on 3DPW. To compare fairly, we introduce ADHMR (ITW)†, trained without Human3.6M, which still outperforms others. Table 3 and Table 4 in the paper show that finetuning on target or extra datasets offers only marginal gains, highlighting the effectiveness of our framework.

Q5. More details

UVD in L176 denotes 2D coordinates (u, v) in the image and their corresponding depth (d) of human keypoints, which aligns 3D joint with image space.

We will include the formulas for PLCC and SRCC in the revision.

The datasets used for data cleaning are listed in L401–403 (right column).

Q6. Comparison with ScoreHMR and CloseInt

• ScoreHMR is a two-stage method that refines pretrained pose estimates using a diffusion model, applying reprojection loss only during sampling while trained with standard diffusion loss.
• CloseInt is also a two-stage method and focuses on multi-person interaction. It refines plausible initial poses with a proxemics and physics-guided diffusion model.
• Our ADHMR is one-stage and learns to synthesize poses from random noise, guided solely by image cues and preference alignment, without handcrafted priors.

The Updated Table 2 (see Q4) includes results of ScoreHMR and CloseInt, where our ADHMR outperforms both.

Q7. Method generalization analysis

We provide more ITW results at:

https://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/8ugF_Q1.pdf

ADHMR shows generalization under extreme poses (a–c), occlusions (a, b, d, e), and complex backgrounds (f).

However, extreme camera angles remain challenging, likely due to limited training coverage, as shown here:

https://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/mE1y_Q7.pdf

Q8. Qualitative examples and failure case of HMR-Scorer

Although HMR-Scorer is robust even for ITW scenes (see Q3), it still fails in some occlusion cases:

https://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/mE1y_Q8.pdf

With limited visual cues, the scorer may prefer predictions aligned with the image but not the 3D GT. We plan to explore temporal scorers to address this.

Thank you for recognizing the effectiveness of our scoring strategy, significant improvements over the base model, and good experiment design. We deeply appreciate your valuable comments and address them below.

Q1. Generalizability to unseen data & Bias caused by DPO

To further support our claim on in-the-wild robustness, we provide more ITW qualitative results here:

https://postimg.cc/jwY2qCgT

These examples demonstrate improved performance under extreme poses, occlusions, and complex backgrounds—for example, (a–b) involve lying or rolling on the floor, where our method shows better 2D alignment. In the paper, Figure 2 (g–h) shows ITW images without ground truth, allowing only qualitative comparison. Results on 3DPW also partially reflect ITW capability with outdoor, unconstrained scenes.

Bias brought by DPO. Our framework improves generalization by shifting the objective from fitting noisy in-the-wild pseudo labels to learning from relative preferences, encouraging the model to capture perceptually aligned and semantically meaningful poses (see L55–59). Besides, HMR-Scorer implicitly learns 2D image alignment by integrating global and pixel-aligned features. This leads to better robustness and transferability in challenging real-world scenarios.

Q2. More relevant references

Thank you for the helpful comment. While our work is the first to apply DPO to HMR, we acknowledge related efforts in motion generation and reward modeling. We also agree that data cleaning has been explored in prior works. For example, SPIN [A] filters training data based on reprojection error, and Unite the People [B] relies on manual selection. We will include these references in the revision.

[A] Kolotouros et al., Learning to reconstruct 3D human pose and shape via model-fitting in the loop, ICCV 2019.

[B] Lassner et al., Unite the people: Closing the loop between 3D and 2D human representations, CVPR 2017.

Q3. About contribution

Thank you for your thoughtful assessment and for acknowledging the strength of our experimental design.

While DDPO is well-established in tasks like image generation, its application to HMR is non-trivial. ADHMR introduces key domain-specific innovations:

• a learned HMR-Scorer capturing perceptual image–mesh alignment, and

• a preference dataset constructed without human labels.

ADHMR is the first to successfully apply DPO to HMR, enabling training from relative preferences rather than noisy ITW pseudo labels. In contrast, conventional methods often overfit these labels, degrading 3D accuracy—as also noted in TokenHMR [C].

Beyond performance gains, HMR-Scorer supports automatic data cleaning, making it broadly helpful in enhancing other HMR pipelines (see Sec. 4.5).

[C] Dwivedi, Sai Kumar, et al. "Tokenhmr: Advancing human mesh recovery with a tokenized pose representation." CVPR. 2024.

Q4. Generalization to other hypothesis-generation approaches

Thank you for the valuable suggestion. We chose HypoNet as our main baseline as it is the latest SOTA diffusion-based HMR model for general-purpose scenarios. Our framework is compatible with other diffusion-based methods [D, E], and we plan to extend experiments in future work.

Deterministic models (e.g., HMR 2.0) can also adopt our preference framework by discretizing the pose space (e.g., via VQ-VAE in TokenHMR [C]) and replacing the regression network with a classification network for DPO training.

Meanwhile, HMR-Scorer and data cleaning are model-agnostic and can enhance training quality across both probabilistic and deterministic pipelines—offering a practical path for pseudo-label refinement in in-the-wild settings where 3D ground truth is scarce.

[D] Cho, Hanbyel, and Junmo Kim. "Generative approach for probabilistic human mesh recovery using diffusion models." ICCV. 2023.

[E] Stathopoulos, Anastasis, Ligong Han, and Dimitris Metaxas. "Score-guided diffusion for 3d human recovery." CVPR. 2024.

Q5. Duplicate citations

We will fix this in the revision.

Q6. HMR-Scorer training data preparation

Yes, we add joint-wise Gaussian noise to the ground truth SMPL pose to simulate rotational errors, with magnitudes empirically determined. We will include the exact noise parameters in the final appendix.

Q7. Training/inference cost

We provide the training/inference time and memory cost below:

• Inference | Model| Time per prediction| Batch Size| Memory| |-|-|-|-| | HMR-Scorer| 56 ms| 32| 3350 MiB| | ADHMR ($M=10$) | 3 ms| 160| 8400 MiB| | ADHMR ($M=100$) | 1.6 ms| 80| 24 GB| | ADHMR ($M=200$) | 1.5 ms| 40| 24 GB| All inference was conducted on a single NVIDIA V100 GPU. $M$ denotes the number of predictions.

• Training | Model| Time| Batch Size | Memory per GPU| |-|-|-|-| | HMR-Scorer | 24 h| 32| 11500 MiB| | ADHMR| 7 h| 40| 18 GB| All training was conducted on 4 NVIDIA V100 GPUs.

Thanks for recognizing the strong performance of our method and the clear presentation of our paper. We deeply appreciate your constructive comments and address them below.

Q1. Equation 5 elucidation

Thank you for your helpful comment. Our formulation follows the approach introduced in Diffusion-DPO [A], which provides a detailed derivation (see Section 4 and Supp. S2 of that paper). To improve readability, we will revise our manuscript to explicitly cite this reference at Line 254 and include a pointer to the key steps in the original paper.

[A] Wallace, Bram, et al. "Diffusion model alignment using direct preference optimization." CVPR. 2024.

Q2. Results in Table 5 not in Table 2

We separate Table 2 and Table 5 as they serve different goals. Table 2 evaluates the ADHMR framework with SMPL-based methods for fair comparison, while Table 5 focuses on data cleaning with more challenging SMPL-X data. Methods in Table 5 are deterministic and not compatible with ADHMR. We will clarify this in the final version.

Q3. About ScoreNet training set

Thank you for the insightful comment. While we understand the concern, it’s important to note that ScoreNet and our HMR-Scorer were originally trained on different datasets. Specifically, ScoreNet was trained on H3.6M, 3DPW, MPI-INF-3DHP, MPII, COCO, and UP-3D, while our HMR-Scorer was trained on a set of synthetic and human-interaction datasets with higher annotation accuracy and scene diversity, as detailed in Sec. 5.1.

To ensure a fair comparison, we re-trained ScoreNet using our HMR-Scorer training sets. Please check the results in the updated Table 1:

ScoreNet$\dagger$ is the ScoreNet that is trained on the same train sets as HMR-Scorer. Results show that our HMR-Scorer still outperforms ScoreNet$\dagger$ using the same set of train sets, validating the design of our scoring strategy rather than dataset bias.

We will clarify this point and explicitly list the datasets used in Table 1 to improve rigor and transparency.

Q4. Seen datasets of methods in Table 2

Thank you for your valuable comment. We will include a clearer description of the datasets used in Table 2 in the revised version. Specifically:

1. ADHMR and HypoNet are trained on the same set of datasets, and ADHMR does not use HMR-Scorer. This comparison demonstrates the effectiveness of our framework even without additional supervision.
2. ADHMR (ITW) is not directly trained on the datasets used for training HMR-Scorer (e.g., DNA-rendering). However, it may indirectly benefit from these extra datasets. As you rightly pointed out, Table 4 shows that our method is significantly more effective than directly fine-tuning with those datasets, which further highlights the strength of our approach.

Q5. About Figure 2

Thank you for your careful analysis and helpful suggestions.

We have updated Figure 2 by adding ground truth and side-view visualizations for better interpretability. In addition, we have replaced the first example with a clearer case. The revised figure is available here:

https://postimg.cc/gx0YG3G8

In the original first example, the man’s right hand is fully occluded, and our model still predicts a plausible pose based solely on visible cues. In contrast, HypoNet’s result is not consistent with the observed image evidence. It is worth noting that our model processes single-frame images only and does not leverage any temporal context from the sequence.

We appreciate your detailed observation and have reflected this clarification in the updated figure.

Thanks for highlighting the novelty of applying preference optimization to HMR, the effectiveness of our framework, and the comprehensive experiments. We truly appreciate your encouraging feedback and respond to your points below.

Q1. Equation 5 elucidation

Thank you for your helpful comment. Our formulation follows the approach introduced in Diffusion-DPO [A], which provides a detailed derivation (see Section 4 and Supp. S2 of that paper). To improve readability, we will revise our manuscript to explicitly cite this reference at Line 254 and include a pointer to the key steps in the original paper.

[A] Wallace, Bram, et al. "Diffusion model alignment using direct preference optimization." CVPR. 2024.

Q2. Preference dataset application

Thank you for the great question. The preference dataset is not directly applicable to standard non-probabilistic HMR models, as the DPO framework assumes the model produces discrete predictions with associated probabilities. Our current implementation is based on a diffusion-based HMR model, which naturally supports such probabilistic outputs, consistent with the assumptions in Diffusion-DPO.

However, we believe adapting deterministic models to fit within this preference framework is both feasible and promising. As demonstrated in TokenHMR [B], one can first train a VQ-VAE to discretize the continuous pose space, and then modify the original regression head (e.g., in HMR2.0) into a classification head over quantized tokens. This adaptation would allow deterministic HMR models (e.g., HMR 2.0) to benefit from our preference datasets and preference optimization framework, unlocking similar gains in robustness and image alignment. We consider this a promising direction for future research.

Importantly, the HMR-Scorer and our data cleaning pipeline are fully model-agnostic, and thus directly applicable to both probabilistic and deterministic models. They provide a scalable way to improve training data quality by filtering out noisy pseudo-labels—particularly useful for in-the-wild datasets where 3D ground truth is often unavailable or unreliable. We view this as a practical step toward broader adoption of preference-guided learning in HMR, and we plan to explore these extensions in future work.

[B] Dwivedi, Sai Kumar, et al. "Tokenhmr: Advancing human mesh recovery with a tokenized pose representation." CVPR. 2024.

Q3. Typos

Thank you for pointing out the duplicated citation. We will fix this in the final version.