Diffusion Implicit Policy for Unpaired Scene-aware Motion Synthesis

1. Overall writing of the paper is mostly not self-contained. This means that readers with limited background of diffusion-based motion synthesis may have difficult time comprehending the gist of the paper. Most importantly, it is difficult to understand why the method is free from motion-scene paired data.
2. There is no comparison on quality of generated motions with famous metrics such as FID or action recognition accuracy.
3. The proposed DIP fall short in contact metric as shown in Table 1 and Table 2.

• The effectiveness of the ControlNet structure of the motion diffusion model is questionable as there is no ablation study. The position control should be able to enforce during diffusion sampling. Why bother training a control module?
• The interaction policy consists of well known constraint functions, making the contribution of this part weak. It is also debatable whether these constraint terms can be called reward, which is a taxonomy in the reinforcement learning.
• From the visual results, the generation motions seem to have unreasonable arm movements. Is this problem caused by inference-time optimization?
• Writing issues:

1. Almost all the citation format is wrong. Use \citep{} instead of \cite or \citet, (e.g., AMASS~\citep{amass})
2. Line 197: e.g. => e.g.,
3. Line 328: according => according to

This paper seems to have logical or grammatical or formatting errors throughout, relating to how citations belong in sentences. For example it often calls specific people "works", and names and semicolons are often inside sentences where they should not be part of the prose. E.g., "Some of them Wang et al. (2021; 2022a) utilized..." -- this doesn't make sense. Maybe the authors simply pasted their text into the ICLR template and then did not check for any errors. This makes the paper much more difficult to read than it should be.

The paper says that its main contribution is "We propose a brand-new framework,termed Diffusion Implicit Policy, for scene-aware motion synthesis." To me, the newness of the method, and the name of the method, are not very appealing contributions. I recommend that the authors state, as their primary contribution, something more useful about their work.

The authors state that "The diffusion process will gradually add noise to the original motion x0". This is surprising, because a very useful property of the diffusion formulation is that you can jump directly to a desired noise level, instead of stepping there gradually. Why do this? It seems like the method can be sped up a lot by using the standard method.

"given a sequence of interaction sub-tasks" -- not sure what this means.

"Given historical motion, the synthesized future motion should be in consistent with it" -- Do the authors mean inconsistent, or consistent?

The authors say "To ensure the body meshes and joints of synthesized motion are fully differentiable for Diffusion Implicit Policy, we directly train diffusion model ϕ to predict the original motion ˆxϕ0 for each noised xt following MDM Tevet et al. " -- it's not clear to me what the connection is between this detail and differentiability. Are the authors saying that if you predict the epsilon, it won't be differentiable?

The authors say that by normalizing the data by the zeroth pose, "motion in scenes can be derived via simple translation and horizontal rotation". This does not sound true. In general we seek much more detailed motion, and even the figures show the sort of motion I am imagining. Also, I am skeptical of the "horizontal rotation" -- rotation about any horizontal axis seems far too acrobatic for the task considered here.

In two places, it says that the controlnet-informed diffusion model is trained to predict "the origin motion". What is the origin motion? It is referred to with different symbols in the two locations, x_0^psi and {P_s}.

The "initial state maintenance" reward seems to ask for the body to be perfectly still. Why is this a good thing?

It seems like multiple modules are introduced twice: once in 3.3, and once in 3.5. (This is in addition to the initial introductions in 3.2.) Am I misunderstanding something here -- e.g., could it be that these are actually different modules (but with the names)?

In the multi-task motion synthesis, it is not clear to me how it is determined when a task is "finished".

How is the model optimized? From these "reward" functions it seems like reinforcement learning is involved, but I couldn't find anything concrete on this topic.

consequent -> subsequent

There are also several weaknesses in the proposed. First, it does not model the interaction between the hands and the objects, which restricts the motion synthesis to scenarios where one only emphasizes correct contact of the body shape with coarse surfaces in the scene, which further limits the diversity of the synthesized motion. Second, the dependence on human motion diffusions on one hand provides human motion priors for motion synthesis, but on the other hand, can be a source of bias, so it is critical to establish the solidity of the proposed using multiple scene-unaware motion synthesis modules. More specifically, why was the motion diffusion model picked in the paper, how about other models? Thirdly, The employed reward functions are also observed in the literature, it is more or less like an inclusion of all of them, which may not be bad, but currently, it does not contribute too much to the novelty of the proposed pipeline. Furthermore, the whole system looks very heavy, e.g., too many components, and also the implicit policy optimization is used in each denoising step, which makes it inefficient, so how does the robustness look like, and how it is practically useful?