Physics-based Skinned Dance Generation with RL Fine-tuning

1. L191-L193 demonstrated the importance of controling the character with SMPL skinned mesh because it is a physically-constrainted model, while L192-L222 claimed that disabling the collision checks between adjacent joints in both training and inference helps better replicating the reference motion under collision constraints. This is a bit confusing to me because physically constraints inherited in SMPL are important for motion imitation/synthesis and they are also (I believe) the motivation of the paper itself by involving the step: physical simulator->simulated motion as shown in Fig1.
2. The proposed method trained its imitation policy on AMASS dataset and then finetuned on AIST++, which is a different setting as EDGE. So the comparisons between the two methods seems not fair to me as AMASS comprises many realistic human motions that can apparently help for imitation policy training without any other explict physical constraints.
3. It seems that an optimization based 'projection' step from the generated motions to physically-appealing AMASS dataset as [1,2] can also induce a motion refinement as post-processing, which is very simple and more straightforward to distill the prior of large human motion datasets. While the proposed Anti-freezing Reward is heuristic-based and the generated motions as physical simulator is not the same as the extracted simulated motions.

[1] Tiwari et al. Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fields. ECCV 2022.
[2] NRDF: Neural Riemannian Distance Fields for Learning Articulated Pose Priors. CVPR 2024.

• In Table 1, the Human Perception Results for the proposed method are marked as ‘/’.
• I believe the results are highly dependent on the performance of the imitation policy and the simulation settings. It would be valuable to see some analysis in this direction.
• In the ‘Other Non-physical Metrics’ section, the authors claim that the low diversity issue is caused by short motion clips. Would it be possible to compare the diversity of different methods and the ground truth by clipping the motions to the same length as the diffusion model?

The proposed work borrows a lot respectively from Yuan et al 2023 and Black et al 2024, both in terms of the underlying ideas and the formulations. There is nothing wrong with this if the result is good. However, the way the proposed method is described is far from clear. Possibly due to the lack of space, a lot is left to be implicitly understood, instead of explicitly explained.

The paper first describes the training of an imitation policy and later the reinforcement learning-based finetuning that includes an imitation reward function. However, the link between the two is unclear. How do you go from the policy in (1) to the reward function in (5)? Is the policy used to produce a sequence, which is later used in the reward function? But since the policy is stochastic, is only the mean used? The description of the imitation reward is also confusing since it talks about generated motion in a physical simulator when the generated motion ought to come from the diffusion model. By the way, what depends on $c$ in (5)?

In the notations, many different things are referred to as states or poses: $S_t, s_t, x_{t}^s, x_{t+1}^A, x_{t+1}^{res}, \bar{x}_t^s, x_t, x_0^s$, and $\bar{x}_0^s$. These are not fully described. For example, what does “residual between the current state of the character and the next frame of the reference motion” mean? Which state is referred to and what “reference motion”? No such reference motion has yet been mentioned. Also confusing is that $t$ is used to both denote time and diffusion step. The $0$ in $x_0^s$ should be understood as diffusion step $0$, but $t$ in $x_t^s$ refers to time $t$. Do $x_0$ and $x_t$ in (2) refer to full sequences, while $x_t^s$ in (1) only includes one time-step? Using $s$ in the notation is also confusing since $s$ is often used to indicate the target diffusion step in papers about diffusion, while here it is something else.

While other parts of the paper are easy to read and understand, the theoretical parts ought to be completely rewritten. The way it is written now seems a bit sloppy with derivations more or less directly following those in Yuan et al 2023 and Black et al 2024, but without updating the notations so that everything can be fully integrated and understood.

• What is the difference between the proposed RL fine-tuning (RLFT) and existing methods like PhysDiff? Specifically, what is the relationship between the imitation policy part (Section 3.1) and PhysDiff? I also think it would be beneficial to include a comparison with PhysDiff in the experiments.

• The details of the on-policy RLFT process may be somewhat unclear. For example, it is stated, "we sample 2,048 motions from the training dataset of AIST++." This is a bit ambiguous. Since this is RL fine-tuning, the samplings are generated by the current policy, while as I understand, the diffusion model (EDGE) is already well-trained. Can samplings from the training set provide a balanced input, including negative aspects (penetration/freezing)?

• In the ablation study, I find "EDGE w proj" somewhat confusing; it is described as "similar to PhysDiff" (Line 451). Why do the authors consider PhysDiff as "post-processing"? As I understand it, PhysDiff embeds the simulator into the diffusion models within the final few iterations, which doesn’t seem all equivalent to "post-processing." Could the authors clarify the implementation of "EDGE w proj"? Also, in the demo, the "post-processing" approach results in floating in the air. This seems odd and confusing, as a physical simulator is expected to prevent this.

• Some related works are missed, e.g., Bailando++ (Siyao et al., 2023).