PhaseFusion: A Diffusion-based Periodic Parameterized Motion Generation Framework

The two main areas of concern are the empirical evaluations and potentially narrow scope.

The empirical results are interesting, however there are concerns related to the setup and evaluation. The data used to train the proposed approach is unclear, with only the evaluation data sets mentioned. For the extended length sequence, evaluation is only performed on the last 196 frames, failing to capture the performance difference in the earlier frames. Additionally reporting an average over strided evaluations would further demonstrate the improvement of the model. Additionally it would be interesting to see the impact of excluding the data augmentation strategy to demonstrate the performance improvement due to the proposed model versus this data augmentation.

With respect to the scope, it appears the main value of the proposed approach is an improvement in performance for extended length sequences for motion generation, as at the standard length the proposed approach does not appear to significantly outperform existing models. This not only limits the potential audience of interest but additionally would be of more interest if corresponding annotated datasets with longer sequence lengths existed in order to better evaluate rather than cropping longer generated sequences to evaluate utilizing shorter annotated data.

1. It's challenging to assess the actual impact of this study. The authors assert their ability to generate diverse and arbitrary-length motion sequences. However, only two samples are displayed in Fig. 3, both of which appear relatively simple. While the supplementary material offers more results, comparisons with other works are absent.

2. The paper lacks a detailed explanation of how the generation of long and complex motion is achieved, especially regarding motion transition. The diversity also seems somewhat limited.

3. There are several issues with the writing:

• Instead of "using the Inverse FFT (3.2)", it should be "using the Inverse FFT (Section 3.2)". Similarly, "the starting pose (3.3)" should be rephrased as "the starting pose (Section 3.3)".
• The images in the figures are somewhat blurry, and the text font size is too small for comfortable reading.
• In Table 1, shouldn't the results under "Real" in "Humanml" be centered?
• The paper does not provide illustrations or explanations on how the evaluation metrics were calculated.

I believe the paper's idea is promising. However, this version seems hastily put together. From the writing to the experiments, the authors should give them a thorough review and polish.

1. The paper claims that existing methods have difficulties in generating motion sequences of arbitrary length with respect to motion diversity and smooth transitions, and I agree that phase diversity might be easy to lead to more diverse motion sequences. However, for smooth transitions generated over long periods of time, there are existing methods that combine interpolation with diffusion modeling that achieve good performance, not limited to [1,2,3,4].

2. in line with the above, I assume that the baselines used for long-term generation (e.g., MDM) take advantage of the fact that transformer architectures can accept different input lengths (and if possible, please confirm this in the revision as well), whereas I do believe that the methods mentioned above, autoregressive generation and interpolation during diffusion is feasible, as this is already a very popular tool [1,2,3,4], which should be compared to as well.

3. The proposed method seems to require an initial pose as input and baselines do not. If this difference is taken into account when performing the quantitative evaluation, then the comparison would not be fair.

4. In the demo video provided, I do not see complex motion in HumanML3D, such as backflips or crawling on the ground, I suspect that the proposed phase-based approach is still ineffective for such less periodic movements with high-frequency details.

[1] Raab et al. Single Motion Diffusion. Arxiv 2023

[2] Shafir et al. Human motion diffusion as a generative prior. Arxiv 2023

[3] Zhang et al. DiffCollage: Parallel Generation of Large Content with Diffusion Models. CVPR 2023

[4] Tseng et al. EDGE: Editable Dance Generation From Music. CVPR 2023

I have two major concerns: (A) the method is not well-presented and (B) the authors make many claims but do not substantiate them in text nor in ablations.

(A) Overall, the method is not well-presented:

• the authors extend on DeepPhase and the paper requires an understanding of this work - the authors should have briefly described DeepPhase in equations at the beginning of Section 3 - this would have been a great opportunity to also differentiate their work from DeepPhase
• A method overview is missing and Figure 2 is insufficient for this. For example, it only becomes clear in Section 3.3. on page 5 that the authors utilize a multiple training stages.
• Figure 2 is unclear: what are the different steps and how are they connected? This could be improved by adding better descriptions in text and in the figure, i.e. linking the Figure with the Equations and text.
• In Section 3.3 diffusion is briefly introduced. However, only the forward step is presented in equations while the backward step- arguably the more relevant part for this work - is not.
• The last Sentence of 3.3. is unclear: “what” loss function is used to train “what” part of the network?

The method has many steps, starting at the pose preprocessing, pre-training of the encoder decoder model and finally the diffusion model. I strongly suggest that the authors first start Section 3 with an overview of the method, outlining the general idea, explaining that the encoder-decoder are retrained and how the pre-processing works. The method is not well-descried in Equations but rather in text. This makes it really difficult to follow. I strongly suggest to rely more on Equations - those can be used not just in text but also for example in Figure 2, linking the overview image with the text.

The authors claim that they utilize a “compact yet expressive … phase space” (Abstract) but in Section 3.2 it seems that latent “phase” representation X has the same amount of phases as pose (?) signal S has time steps - so it seems that no high-frequency signals are actually removed and no compression is happening.

(B) The paper makes many claims which are not substantiated in text or ablation:

• Page 2: “By encoding motion in … periodic phase space we alleviate the need for large amounts of training data” - How? This is not confirmed by any experiments and I don’t see how that would help with missing data. This claim should be explained in more detail and also ablated.
• Page 3: “(2) DeepPhase tends to encode low-frequency components…. leading to limited accuracy” - why?
• Page 5: “it facilitates the synthesis of a broader range of motions that surpass the training motion distribution;” how is this verified? This is a very big claim
• Page 5: “it fosters a continuous motion embedding in the frequency domain” - could the authors expand on what the mean here by “continuous motion embedding”?

Minor:

• Figure 4 is missing x- and y-axis labels.
• The authors sometimes use the term “video” (e.g. Section 3.2) when I believe they mean “human motion sequence”?
• In 3.1 replace “... feature vector $d_i$ that <extracts> temporal …” with “... feature vector $d_i$ that <contains> temporal …”